Portrait action driving method and device, storage medium and computer equipment

Abstract

According to the portrait action driving method and device, the storage medium and the computer equipment provided by the invention, when action driving is performed on the target portrait, the target audio data are firstly acquired, then the audio expression features of the target audio data are extracted by using the target feature extraction model, and the model can simultaneously capture the features of multiple layers in the audio data, so that the accuracy of action driving is improved. Many-to-many mapping among mouth shape action features, expression style features and head posture features in the audio expression features is realized, so that the driving effect coordination is improved; then, a target expression parameter generation model is determined, and the model is obtained through training based on the overlapping frame prediction technology, so that seamless conversion of the motion driving parameters obtained by predicting the audio expression characteristics through the model on the time sequence can be realized, the naturalness of the driving effect is further improved, and therefore, the driving efficiency is improved. And head action driving is performed on the target portrait based on the action driving parameters, so that a portrait action animation with high harmony and high naturalness can be generated.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a portrait motion driving method, device, storage medium and computer equipment. Background Art

[0002] With the widespread application of AIGC (Artificial Intelligence Generated Content) technology in the video field, voice-driven 3D portrait motion generation technology is rapidly emerging and showing great potential in many fields such as education, entertainment and virtual reality. Especially in high-cost video shooting and production scenarios such as film, television, games and animation, this technology drives 3D characters or animals to speak, sing and talk through voice, significantly reducing the cost and time of video production.

[0003] However, most current portrait motion driving technologies rely on a single driving model, and multiple models need to be built separately to accurately control the motion postures of different areas of the 3D portrait; in addition, these models not only require customized training for each portrait, but also usually use classification methods to define the motion style during the training process, resulting in poor adaptability of the model. In summary, the existing methods not only limit the scope of application of the technology, but also lack coordination between the motion and voice of different parts in the generated driving effects, making it difficult to meet the needs of the field of high-end video production. Summary of the invention

[0004] The purpose of the present application is to solve at least one of the above-mentioned technical defects, especially the technical defects that the methods in the prior art not only limit the scope of application of the technology, but also lack coordination between the actions and voices of different parts in the generated driving effects, making it difficult to meet the needs of the field of high-end video production.

[0005] The present application provides a portrait motion driving method, the method comprising:

[0006] Acquire target audio data, and use a preset target feature extraction model to extract audio facial features of the target audio data; the audio facial features include lip movement features, expression style features, and head posture features;

[0007] Determine a target expression parameter generation model; the target expression parameter generation model is based on overlapping frame prediction technology and is trained using a target loss function;

[0008] Inputting the audio expression feature into the target expression parameter generation model to obtain the action driving parameters output by the target expression parameter model;

[0009] A target portrait is acquired, and a head movement of the target portrait is driven based on the movement driving parameters to generate a portrait movement animation.

[0010] Optionally, the target feature extraction model includes a time-domain convolutional feature extractor and a multi-layer transformer encoder;

[0011] The step of extracting the audio state features of the target audio data by using a preset target feature extraction model includes:

[0012] Inputting the target audio data into the time-domain convolutional feature extractor, so as to perform low-level feature extraction on the target audio data through the time-domain convolutional feature extractor to generate audio phoneme features; the audio factor features include rhythm features, emotion features and intonation features;

[0013] The audio phoneme features are encoded with the multi-layer transformer encoder to obtain high-level features, and the high-level features are output as audio mood features.

[0014] Optionally, performing low-level feature extraction on the target audio data by the time-domain convolutional feature extractor to generate audio phoneme features includes:

[0015] The time domain audio features of the target audio data are extracted by the time domain convolution feature extractor, and the time domain audio features are converted into frequency domain audio features, so as to perform multi-dimensional feature extraction on the frequency domain audio features to form audio phoneme features.

[0016] Optionally, the encoding of the audio phoneme features by using the multi-layer transformer encoder and outputting the high-level features as audio expression features comprises:

[0017] The multi-head self-attention mechanism in the multi-layer transformer encoder is used to read the temporal dependency and cross-feature dependency of the audio phoneme features, and based on the temporal dependency and the cross-feature dependency, a layer-by-layer feature interaction weighting mechanism is used to perform advanced encoding on the audio phoneme features to generate audio mood features.

[0018] Optionally, determining a target expression parameter generation model includes:

[0019] A pre-trained model is constructed using the Transformers architecture and the CNN architecture, and an overlapping frame prediction technology is introduced into the pre-trained model to obtain an initial expression parameter generation model;

[0020] Inputting the pre-acquired sample audio data into the initial expression parameter generation model to obtain the predicted action driving parameters output by the initial expression parameter generation model;

[0021] The predicted action driving parameters are close to the real action driving parameters of the sample audio data, and the initial expression parameter generation model is trained using a target loss function;

[0022] When the initial expression parameter generation model meets the preset training conditions, the trained initial expression parameter generation model is used as the target expression parameter generation model.

[0023] Optionally, the calculation formula of the objective loss function includes:

[0024]

[0025] In the formula, represents the target loss function; represents the real action driving parameters; represents the predicted action driving parameter; where, represents the time window from the pth moment to the wth moment.

[0026] Optionally, the driving the target portrait to perform a head movement based on the movement driving parameters to generate a portrait movement animation includes:

[0027] Determine a portrait action driving model; the portrait action driving model is composed of a lip action driving module, an expression style driving module, a head posture driving module and an animation generation module;

[0028] The action driving parameters and the target portrait are used as input data, and the input data are respectively input into the lip action driving module, the expression style driving module and the head posture driving module to obtain a first driving result output by the lip action driving module, a second driving result output by the expression style driving module and a third driving result output by the head posture driving module;

[0029] In the animation generation module, the first driving result, the second driving result and the third driving result are aligned and fused in time frames based on the target audio data to generate a portrait action animation.

[0030] The present application also provides a portrait motion driving device, comprising:

[0031] A feature extraction module is used to obtain target audio data and extract audio expression features of the target audio data using a preset target feature extraction model; the audio expression features include lip movement features, expression style features and head posture features;

[0032] A model determination module is used to determine a target expression parameter generation model; the target expression parameter generation model is obtained by training with a target loss function based on an overlapping frame prediction technology;

[0033] A parameter prediction module, used for inputting the audio expression feature into the target expression parameter generation model to obtain the action driving parameters output by the target expression parameter model;

[0034] The action driving module is used to obtain a target portrait and drive the head action of the target portrait based on the action driving parameters to generate a portrait action animation.

[0035] The present application also provides a storage medium, in which computer-readable instructions are stored. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the portrait motion driving method as described in any one of the above embodiments.

[0036] The present application also provides a computer device, comprising: one or more processors, and a memory;

[0037] The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the portrait motion driving method described in any one of the above embodiments are performed.

[0038] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0039] The portrait action driving method, device, storage medium and computer equipment provided by the present application, when it is necessary to drive the target portrait for action, the target audio data for action driving can be obtained first, and then the audio demeanor features of the target audio data can be extracted by using the pre-trained target feature extraction model, where the target feature extraction model can simultaneously capture the features of multiple levels in the audio data, and realize the many-to-many mapping between the lip action features, expression style features and head posture features in the audio demeanor features, thereby improving the coordination of the driving effect; then the target expression parameter generation model can be determined, and since the model is obtained by training based on the overlapping frame prediction technology, the action driving parameters obtained by predicting the audio demeanor features through the target expression parameter generation model can achieve seamless conversion in the time series, thereby improving the naturalness of the driving effect, therefore, based on the action driving parameters, the head action of the target portrait is driven, and a portrait action animation with high coordination and high naturalness can be generated. Through this method, the present application can directly copy the feature distribution of different forms in the target audio data, realize many-to-many feature mapping, thereby avoiding the limitations of portraits and styles, thereby improving the coordination and naturalness of the driving effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0041] Figure 1 A schematic diagram of a flow chart of a portrait action driving method provided in an embodiment of the present application;

[0042] Figure 2 A flowchart of an audio expression feature extraction process provided in an embodiment of the present application;

[0043] Figure 3 A schematic diagram of a process for determining a target expression parameter generation model provided in an embodiment of the present application;

[0044] Figure 4 A schematic diagram of the structure of a portrait motion driving device provided in an embodiment of the present application;

[0045] Figure 5 A schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] Most current portrait motion driving technologies rely on a single driving model, and multiple models need to be built separately to accurately control the motion postures of different areas of a 3D portrait. In addition, these models not only require customized training for each portrait, but also usually use classification methods to define the motion style during training, resulting in poor adaptability of the model. In summary, the existing methods not only limit the scope of application of the technology, but also lack coordination between the motion and voice of different parts in the generated driving effects, making it difficult to meet the needs of high-end video production.

[0048] Based on this, this application proposes the following technical solutions, see below for details:

[0049] In one embodiment, Figure 1 As shown, Figure 1A schematic diagram of a flow chart of a portrait motion driving method provided in an embodiment of the present application; the present application provides a portrait motion driving method, which specifically includes the following:

[0050] S110: Acquire target audio data, and use a preset target feature extraction model to extract audio facial features of the target audio data.

[0051] In this step, when the user needs to perform an action drive on the portrait, the target audio data for the action drive can be selected on the computer device, so that after the computer device obtains the target audio data, it can use the pre-trained target feature extraction model to extract the audio expression features of the target audio data as the basis for the target action drive of the corresponding portrait.

[0052] Among them, audio expression features refer to the dynamic information related to the character's expression implied in the target audio data, including lip movement features, expression style features and head posture features. To elaborate, lip movement features refer to the shape and movement features of the mouth related to the speech pronunciation process, such as the degree of lip opening or closing or the shape of lip opening and closing; expression style features refer to the expression style features conveyed by the emotion or tone information implied in the audio data, such as happiness, anger or confusion; head posture features refer to the head movement features expressed in the audio data, such as nodding, shaking head or tilting the head.

[0053] It can be understood that the target feature extraction model of the present application refers to a model that extracts features from the input target audio data and obtains multimodal fused audio demeanor features, which is trained using various types of audio data as training samples and the audio demeanor features corresponding to the audio data as training labels. The target feature extraction model trained in this way can simultaneously capture the complex associations between features at multiple levels in the audio data, and realize a many-to-many mapping between lip movement features, expression style features, and head posture features. This mapping can not only show the strong correlation between audio data and driving actions, but also effectively capture potential cross-feature interactions.

[0054] For example, in a target audio data, through the feature extraction results of the target feature extraction model, it can be found that some high pitches or rapid speech rates drive more exaggerated mouth movements, richer expression changes, and more frequent head movements. Therefore, the model can ensure the close synchronization and emotional consistency of movements, expressions, and postures during the portrait action driving process, thereby improving coordination.

[0055] S120: Determine a target expression parameter generation model; the target expression parameter generation model is based on overlapping frame prediction technology and is trained using a target loss function.

[0056] In this step, after extracting the audio expression features of the target audio data in step S110, the computer device can also determine a target expression parameter generation model for converting the audio expression features into corresponding action driving parameters for portrait action driving.

[0057] It should be noted that in order to achieve seamless conversion of the prediction results of the target expression parameter generation model in the time series, the computer device can use the target loss function for training based on the overlapping frame prediction technology when training the target expression parameter generation model.

[0058] Among them, overlapping frame prediction technology refers to a time series prediction method, which aims to achieve temporal smoothness and consistency of prediction results by introducing partial overlapping areas between consecutive time windows; it mainly uses the data of the current window and the last part of the data of the previous window as conditional input, which can not only capture the short-term dependencies of the time series, but also ensure seamless transitions between windows, avoiding breaks or discontinuities caused by independent window predictions.

[0059] Specifically, when applying the target expression parameter generation model, the overlapping frame prediction technology can divide the time series into windows of fixed length, and the end of each window has a certain number of overlapping frames with the beginning of the next window. For example, for a fixed-length window, in addition to the audio data of the current window, the last few frames of the action-driven parameters generated by the previous window are also provided to the model as conditional inputs, so that the model can utilize the contextual information of the time series, fully refer to historical features in the current prediction process, capture long-term and short-term dependencies, and thus achieve temporal continuity and natural transition of the prediction results.

[0060] S130: Input the audio expression features into the target expression parameter generation model to obtain the action driving parameters output by the target expression parameter model.

[0061] In this step, after determining the target expression parameter generation model through step S120, the computer device can input the audio expression features into the target expression parameter generation model, and perform parameter prediction on the audio expression features through the target expression parameter generation model. Since the target expression parameter generation model is based on overlapping frame prediction technology and is trained using a target loss function, the computer device can directly obtain high-quality action driving parameters output by the target expression parameter model based on the audio expression features.

[0062] It is understandable that the action driving parameters are a set of fixed-length numerical vectors extracted and mapped from the audio expression features, which are used to drive the expression and movement changes of the portrait. The action driving parameters may include multiple elements, each of which represents a control signal of the portrait in a specific action dimension, such as controlling eye movements, controlling lip movements, controlling head posture, etc. Through the action driving parameters, the present application can encode multi-dimensional expression features into a compact representation in numerical form, so that each dimension directly corresponds to a specific action driven by the portrait.

[0063] Specifically, the target expression parameter generation model of the present application is obtained by training by adding noise and denoising. In the denoising stage, the target expression parameter model can sample random noise data from the standard Gaussian distribution, and gradually superimpose it on the real action driving parameters to construct a denoising process. In each step of the prediction in the denoising stage, the model can receive the noise data of the current step, the audio expression features, including partial features of the current window and the previous window, and the motion parameters generated by the previous denoising as input, and estimate the noise residual of the current step through network reasoning, and then reverse update the noise removal based on the estimation result, and generate an intermediate result closer to the real parameter; after multiple denoising iterations, the model can gradually optimize the highly noisy initial data to the final result without noise, and output the action driving parameters, which are not only smooth and continuous in time, but also highly matched with the audio data in semantics, and can maintain natural coordination between various action dimensions.

[0064] S140: Acquire a target portrait, and drive the target portrait to perform a head movement based on the movement driving parameters to generate a portrait movement animation.

[0065] In this step, after obtaining the motion driving parameters in step S130, the computer device can also obtain the target portrait, and then drive the head movement of the target portrait based on the motion driving parameters to generate a portrait motion animation with high coordination and high naturalness.

[0066] It is understandable that the action-driven parameters here can capture and restore the rich dynamic features in the target audio data through precise modeling and multi-dimensional feature mapping. It directly copies and maps the different forms of feature distribution in the target audio data, and realizes many-to-many feature mapping including lip movements, expression styles, head postures, etc., so that the generated animation is closer to natural performance. Therefore, the present application can effectively avoid the limitations of portrait structure, style, or expression mode through this parameter generation mechanism independent of specific portrait characteristics, and improve the scope of application.

[0067] In the above embodiment, when the target portrait needs to be action-driven, the target audio data for action-driven can be obtained first, and then the audio demeanor features of the target audio data can be extracted using the pre-trained target feature extraction model. The target feature extraction model here can simultaneously capture the features of multiple levels in the audio data, and realize the many-to-many mapping between the lip action features, expression style features and head posture features in the audio demeanor features, thereby improving the coordination of the driving effect; then the target expression parameter generation model can be determined. Since the model is obtained by training based on the overlapping frame prediction technology, the action driving parameters obtained by predicting the audio demeanor features through the target expression parameter generation model can achieve seamless conversion in the time series, thereby improving the naturalness of the driving effect. Therefore, based on the action driving parameters, the target portrait is driven by the head action, and a portrait action animation with high coordination and high naturalness can be generated. Through this method, the present application can directly copy the feature distribution of different forms in the target audio data, realize many-to-many feature mapping, thereby avoiding the limitations of portraits and styles, and thereby improving the coordination and naturalness of the driving effect.

[0068] In one embodiment, the target feature extraction model in step S110 may include a time domain convolution feature extractor and a multi-layer transformer encoder; Figure 2 As shown, Figure 2 A flowchart of an audio expression feature extraction process provided in an embodiment of the present application; Figure 2 In the process of extracting the audio state feature of the target audio data by using the preset target feature extraction model, the process may include:

[0069] S111: Input the target audio data into the time-domain convolutional feature extractor to perform low-level feature extraction on the target audio data through the time-domain convolutional feature extractor to generate audio phoneme features.

[0070] S112: Encode the audio phoneme features into high-level features using a multi-layer transformer encoder and output them as audio mood features.

[0071] In this embodiment, the target feature extraction model can use the Wav2Vec2 model as the basic architecture, which is mainly composed of two parts: a time domain convolution feature extractor and a multi-layer transformer encoder. Among them, the time domain convolution feature extractor can perform low-level feature extraction on the input target audio data and generate output audio phoneme features; the multi-layer transformer encoder can encode high-level features of the audio phoneme features and output them as audio mood features.

[0072] Among them, audio phoneme features refer to features related to speech phonemes in audio data, which can reflect the basic components of audio data and their changes over time. The audio phoneme features of this application may include rhythm features, emotional features, and intonation features; in detail, rhythm features are related to rhythm and duration changes in audio data, emotional features reflect emotional expressions in audio data, and intonation features describe the change pattern of pitch in audio data.

[0073] It is understandable that the time-domain convolutional feature extractor can efficiently process time series data and retain the details of the audio data, while compressing and abstracting its key information to provide high-quality basic feature representation for subsequent processing. Based on this, the present application can capture the short-term dynamic change characteristics in the target audio data through the time-domain convolutional feature extractor, and generate representative audio phoneme features, including rhythm features, emotional features, and intonation features, which correspond to the time rhythm pattern, emotional expression, and voice intonation changes in the target audio data.

[0074] In addition, based on the audio phoneme features generated by the time domain convolutional feature extractor, the multi-layer transformer encoder can further encode the audio phoneme features into high-level features. Here, the multi-layer transformer encoder can model the complex long-range dependencies in the feature sequence through the self-attention mechanism, and pay attention to the interaction and correlation between features at different levels; after multi-layer encoding processing, the output features are integrated into high-level semantic representations, namely audio expression features. The audio expression features of the present application include lip movement features, expression style features, and head posture features. These features can accurately describe the mapping relationship between audio and visual expression, such as how language rhythm drives changes in lip movement, how emotional intonation affects expression style, and how language intonation is accompanied by head posture adjustment.

[0075] In summary, by combining the time-domain convolutional feature extractor with the multi-layer transformer encoder, the target feature extraction model can efficiently extract multi-level feature information from the target audio data and build a natural association from audio to action. This structural design not only enhances the multi-dimensional expression ability of audio data, but also provides a richer and more accurate feature basis for subsequent action-driven parameter generation and portrait action driving, ensuring the high coordination and naturalness of the driving effect.

[0076] In one embodiment, the process of performing low-level feature extraction on the target audio data by using a time-domain convolutional feature extractor to generate audio phoneme features in step S111 may include:

[0077] S1111: extracting the time domain audio features of the target audio data through a time domain convolution feature extractor, and converting the time domain audio features into frequency domain audio features, so as to perform multi-dimensional feature extraction on the frequency domain audio features to form audio phoneme features.

[0078] In this embodiment, after the computer device inputs the target audio data into the time domain convolution feature extractor, the time domain convolution feature extractor can first extract the time domain audio features of the target audio data, and then convert the time domain audio features into frequency domain audio features, and then perform multi-dimensional feature extraction on the frequency domain audio features to form audio phoneme features.

[0079] Specifically, the time-domain convolution feature extractor can first perform a time-domain analysis on the target audio signal, and extract the time-domain audio features in the target audio data based on the analysis results, so that the local changes of the target audio data in the time series can be reflected through the time-domain audio features. Then, the time-domain convolution feature extractor can use Fourier transform to convert the time-domain audio features from the form of time-domain signals to the form of frequency-domain signals, and then obtain frequency-domain audio features, so that the frequency-domain audio features can reveal the distribution of frequency components in the target audio data, and then capture spectral features, such as energy changes in low-frequency and high-frequency parts. After obtaining the frequency-domain audio features, the time-domain convolution feature extractor can further perform multi-dimensional feature extraction on them, and through multi-layer convolution operations and feature aggregation, extract key features related to phonemes from the frequency-domain audio features, such as rhythm features, emotional features, and intonation features, to form audio phoneme features, which provide a coding basis for the subsequent generation process of audio demeanor features.

[0080] In one embodiment, the process of encoding the audio phoneme features into high-level features using a multi-layer transformer encoder in step S112 and outputting them as audio mood features may include:

[0081] S1121: The multi-head self-attention mechanism in the multi-layer transformer encoder is used to read the temporal dependency and cross-feature dependency of the audio phoneme features, and based on the temporal dependency and cross-feature dependency, a layer-by-layer feature interaction weighting mechanism is used to perform advanced encoding on the audio phoneme features to generate audio mood features.

[0082] In this embodiment, after the computer device inputs the audio phoneme features output by the time domain convolutional feature extractor into the multi-layer transformer encoder, the multi-layer transformer encoder can read the temporal dependency and cross-feature dependency of the audio phoneme features through a multi-head self-attention mechanism, and then based on the temporal dependency and cross-feature dependency, a layer-by-layer feature interaction weighted mechanism is used to perform advanced encoding of the audio phoneme features to generate audio mood features.

[0083] It is understandable that the multi-head self-attention mechanism here can analyze different dimensions of input features simultaneously through multiple attention heads, thereby effectively capturing the temporal dependency and cross-feature dependency between audio phoneme features. Among them, the temporal dependency refers to the temporal variation of audio phoneme features, such as the different dependency patterns formed on the time axis by changes in rhythm, fluctuations in intonation, and fluctuations in emotions; the cross-feature dependency refers to the mutual influence and correlation of features at different levels in audio phoneme features, such as the correlation between rhythm features and emotional features and intonation features, and how these features work together to convey the meaning and emotional color of speech.

[0084] Specifically, after obtaining the temporal dependency and cross-feature dependency of audio phoneme features through the multi-head self-attention mechanism, the multi-layer transformer encoder can adopt a layer-by-layer feature interaction weighting mechanism to ensure that the audio phoneme features can be effectively integrated and strengthened between different layers during the processing of each layer by weighting the features of different layers. Therefore, in the process of high-level encoding of audio phoneme features, the multi-layer transformer encoder not only considers the processing of the features of the current layer, but also can weight them based on the features of the previous layer, gradually refine and enhance the semantic expression of the audio phoneme features, and then generate high-quality audio expression features, providing rich basic data for generating highly coordinated and natural portrait action animations.

[0085] In one embodiment, Figure 3 As shown, Figure 3 A schematic diagram of a process for determining a target expression parameter generation model provided in an embodiment of the present application; Figure 3 In step S120, the process of determining the target expression parameter generation model may include:

[0086] S121: A pre-trained model is constructed using the Transformers architecture and the CNN architecture, and the overlapping frame prediction technology is introduced into the pre-trained model to obtain an initial expression parameter generation model.

[0087] S122: Inputting the pre-acquired sample audio data into the initial expression parameter generation model to obtain the predicted action driving parameters output by the initial expression parameter generation model.

[0088] S123: The goal is to make the predicted action driving parameters close to the real action driving parameters of the sample audio data, and use the target loss function to train the initial expression parameter generation model.

[0089] S124: When the initial expression parameter generation model meets the preset training conditions, the trained initial expression parameter generation model is used as the target expression parameter generation model.

[0090] In this embodiment, when training the target expression parameter generation model, the computer device can use the Transformers architecture and the CNN (Convolutional Neural Networks) architecture to construct a pre-trained model, and then introduce the overlapping frame prediction technology into the pre-trained model to obtain an initial expression parameter generation model. Then, the initial expression parameter generation model can be trained using the pre-acquired sample audio data, and the trained initial expression parameter generation model can be used as the target expression parameter generation model.

[0091] It is understandable that the Transformers architecture is good at processing sequence data and can capture long-term dependencies in the input data, while the CNN architecture performs well in local feature extraction. Therefore, by integrating the advantages of these two architectures, the prediction ability of the pre-trained model is better. Then, the computer device can further introduce overlapping frame prediction technology into the pre-trained model, so as to ensure that the initial expression parameter generation model formed can perceive historical context information when predicting across windows, thereby improving the fluency and accuracy of the prediction results.

[0092] Specifically, when training the initial expression parameter generation model, the computer device can first obtain sample audio data, which is marked with corresponding sample labels, namely, the real action driving parameters of the sample audio data. After inputting the sample audio data into the initial expression parameter generation model, the predicted action driving parameters output by the initial expression parameter generation model can be obtained; then, the computer device can take the predicted action driving parameters approaching the real action driving parameters of the sample audio data as the goal, and train the initial expression parameter generation model using the target loss function. When the initial expression parameter generation model meets the preset training conditions, such as the number of iterations reaches the set value, the training is deemed to be completed, and the trained initial expression parameter generation model is used as the target expression parameter generation model.

[0093] Furthermore, the present application can also store the trained target expression parameter generation model in a computer device. When the subsequent computer device needs to predict action driving parameters, it can directly call the target expression parameter generation model to perform parameter prediction on the input audio expression features, thereby improving the efficiency of portrait action driving.

[0094] In one embodiment, the calculation formula of the target loss function in step S120 may include:

[0095]

[0096] In the formula, represents the target loss function; represents the real action driving parameters; represents the predicted action driving parameter; where, represents the time window from the pth moment to the wth moment.

[0097] In this embodiment, the target loss function The performance of the model can be measured by calculating the square error between the real action driving parameters and the predicted real action driving parameters. In the process of model iteration, the performance of the model is measured by continuously minimizing The model can adjust its internal parameters in multiple rounds of iterations so that the predicted action driving parameters it outputs are closer to the real action driving parameters, so that the model can eventually generate high-quality, coordinated and natural action driving parameters, providing a data basis for portrait action driving.

[0098] It can be understood that in the above formula, the time window Specifies the time range that the model considers when making predictions, including the last The historical action driving parameters of the frame and the current window Therefore, the objective loss function enables the model to learn the temporal continuity and correlation of the action-driven parameters, ensuring the smoothness and naturalness of the prediction results in terms of action and expression.

[0099] In one embodiment, the process of driving the target portrait to perform head movements based on the movement driving parameters to generate the portrait movement animation in step S140 may include:

[0100] S141: Determine a portrait action driving model; the portrait action driving model is composed of a lip action driving module, an expression style driving module, a head posture driving module and an animation generation module.

[0101] S142: Taking the action driving parameters and the target portrait as input data, and inputting the input data into the lip action driving module, the expression style driving module and the head posture driving module respectively, to obtain a first driving result output by the lip action driving module, a second driving result output by the expression style driving module and a third driving result output by the head posture driving module.

[0102] S143: In the animation generation module, the first driving result, the second driving result and the third driving result are aligned and merged in time frames based on the target audio data to generate a portrait action animation.

[0103] In this embodiment, when the computer device performs portrait motion driving, it can first determine the portrait motion driving model. Since the portrait motion driving model refers to a model that performs motion driving on the portrait according to input motion driving parameters, the computer device inputs the motion driving parameters and the target portrait as input data into the portrait motion driving model, and can obtain the portrait motion animation directly output by the portrait motion driving model.

[0104] Specifically, the portrait action driving model can be composed of a lip action driving module, an expression style driving module, a head posture driving module and an animation generation module. The lip action driving module is mainly responsible for driving the lip action of the target portrait, the expression style driving module is mainly responsible for driving the expression style of the target portrait, the head posture driving module is mainly responsible for driving the head posture of the target portrait, and the animation generation module is mainly responsible for aligning and fusing the driving results of each module to generate a portrait action animation. Therefore, when the computer device drives the portrait action, the action driving parameters and the target portrait can be used as input data, and the input data are respectively input into the lip action driving module, the expression style driving module and the head posture driving module to obtain the first driving result output by the lip action driving module, the second driving result output by the expression style driving module and the third driving result output by the head posture driving module. Finally, in the animation generation module, the first driving result, the second driving result and the third driving result are aligned and fused in time frames based on the target audio data to generate a portrait action animation with high coordination and high naturalness.

[0105] It is understandable that through this modular structural design, the portrait action drive model can efficiently convert the action drive parameters corresponding to the target audio data into the performance of the target portrait. Specifically, the portrait action drive model can not only accurately capture the multi-level features in the audio data, but also generate smooth and coordinated dynamic effects through the alignment and fusion of time frames, so that the portrait action animation that is synchronized with the target audio data and highly natural can be achieved through the independent driving and precise control of lip movements, expression style and head posture.

[0106] The portrait motion driving device provided by the embodiment of the present application is described below. The portrait motion driving device described below and the portrait motion driving method described above can be referred to each other.

[0107] In one embodiment, Figure 4 As shown, Figure 4 This is a structural diagram of a portrait action driving device provided in an embodiment of the present application; the present application also provides a portrait action driving device, including a feature extraction module 210, a model determination module 220, a parameter prediction module 230 and an action driving module 240, which specifically includes the following:

[0108] The feature extraction module 210 is used to obtain target audio data and extract audio facial features of the target audio data using a preset target feature extraction model; the audio facial features include lip movement features, expression style features and head posture features.

[0109] The model determination module 220 is used to determine the target expression parameter generation model; the target expression parameter generation model is based on the overlapping frame prediction technology and is trained using the target loss function.

[0110] The parameter prediction module 230 is used to input the audio expression features into the target expression parameter generation model to obtain the action driving parameters output by the target expression parameter model.

[0111] The motion driving module 240 is used to obtain a target portrait and drive the head motion of the target portrait based on the motion driving parameters to generate a portrait motion animation.

[0112] In the above embodiment, when the target portrait needs to be action-driven, the target audio data for action-driven can be obtained first, and then the audio demeanor features of the target audio data can be extracted using the pre-trained target feature extraction model. The target feature extraction model here can simultaneously capture the features of multiple levels in the audio data, and realize the many-to-many mapping between the lip action features, expression style features and head posture features in the audio demeanor features, thereby improving the coordination of the driving effect; then the target expression parameter generation model can be determined. Since the model is obtained by training based on the overlapping frame prediction technology, the action driving parameters obtained by predicting the audio demeanor features through the target expression parameter generation model can achieve seamless conversion in the time series, thereby improving the naturalness of the driving effect. Therefore, based on the action driving parameters, the target portrait is driven by the head action, and a portrait action animation with high coordination and high naturalness can be generated. Through this method, the present application can directly copy the feature distribution of different forms in the target audio data, realize many-to-many feature mapping, thereby avoiding the limitations of portraits and styles, and thereby improving the coordination and naturalness of the driving effect.

[0113] In one embodiment, the target feature extraction model in the feature extraction module 210 may include a time domain convolution feature extractor and a multi-layer transformer encoder; the feature extraction module 210 may also include:

[0114] The phoneme feature extraction submodule is used to input the target audio data into the time domain convolution feature extractor so as to perform low-level feature extraction on the target audio data through the time domain convolution feature extractor to generate audio phoneme features; the audio factor features include rhythm features, emotion features and intonation features.

[0115] The expression feature encoding submodule is used to encode the high-level features of the audio phoneme features using a multi-layer transformer encoder and output them as audio expression features.

[0116] In one embodiment, the phoneme feature extraction submodule may include:

[0117] The feature extraction unit is used to extract the time domain audio features of the target audio data through a time domain convolution feature extractor, and convert the time domain audio features into frequency domain audio features to perform multi-dimensional feature extraction on the frequency domain audio features to form audio phoneme features.

[0118] In one embodiment, the phoneme feature extraction submodule may include:

[0119] The feature encoding unit is used to use the multi-head self-attention mechanism in the multi-layer transformer encoder to read the temporal dependency and cross-feature dependency of the audio phoneme features, and based on the temporal dependency and cross-feature dependency, a layer-by-layer feature interaction weighted mechanism is used to perform advanced encoding on the audio phoneme features to generate audio mood features.

[0120] In one embodiment, the model determination module 220 may include:

[0121] The model building submodule is used to build a pre-trained model using the Transformers architecture and the CNN architecture, and introduce the overlapping frame prediction technology into the pre-trained model to obtain the initial expression parameter generation model.

[0122] The model prediction submodule is used to input the pre-acquired sample audio data into the initial expression parameter generation model to obtain the predicted action driving parameters output by the initial expression parameter generation model.

[0123] The model training submodule is used to train the initial expression parameter generation model with the goal of predicting action driving parameters close to the actual action driving parameters of the sample audio data, and using the target loss function.

[0124] The model generation submodule is used to use the trained initial expression parameter generation model as the target expression parameter generation model when the initial expression parameter generation model meets the preset training conditions.

[0125] In one embodiment, the model determination module 220 may further include:

[0126]

[0127] In the formula, represents the target loss function; represents the real action driving parameters; represents the predicted action driving parameter; where, represents the time window from the pth moment to the wth moment.

[0128] In one embodiment, the action driving module 240 may include:

[0129] The model determination submodule is used to determine the portrait action driving model; the portrait action driving model is composed of a lip action driving module, an expression style driving module, a head posture driving module and an animation generation module.

[0130] The model driving submodule is used to take the action driving parameters and the target portrait as input data, and input the input data into the lip action driving module, the expression style driving module and the head posture driving module respectively, to obtain the first driving result output by the lip action driving module, the second driving result output by the expression style driving module and the third driving result output by the head posture driving module.

[0131] The animation generation submodule is used to align and fuse the first driving result, the second driving result and the third driving result in time frames based on the target audio data in the animation generation module to generate a portrait action animation.

[0132] In one embodiment, the present application further provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the portrait motion driving method as described in any one of the above embodiments.

[0133] In one embodiment, the present application further provides a computer device having computer-readable instructions stored therein. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the portrait motion driving method as described in any one of the above embodiments.

[0134] Indicatively, Figure 5 As shown, Figure 5 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 300 may be provided as a server. Figure 5 The computer device 300 includes a processing component 302, which further includes one or more processors, and a memory resource represented by a memory 301 for storing instructions that can be executed by the processing component 302, such as an application. The application stored in the memory 301 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 302 is configured to execute the instructions to perform the portrait motion driving method of any of the above embodiments.

[0135] The computer device 300 may further include a power supply component 303 configured to perform power management of the computer device 300, a wired or wireless network interface 304 configured to connect the computer device 300 to a network, and an input / output (I / O) interface 305. The computer device 300 may operate based on an operating system stored in the memory 301, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0136] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0137] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0138] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0139] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A portrait action driving method, characterized in that: The method comprises: Acquire target audio data, and use a preset target feature extraction model to extract audio facial features of the target audio data; the audio facial features include lip movement features, expression style features, and head posture features; Determine a target expression parameter generation model; the target expression parameter generation model is based on overlapping frame prediction technology and is trained using a target loss function; Inputting the audio expression feature into the target expression parameter generation model to obtain the action driving parameters output by the target expression parameter model; A target portrait is acquired, and a head movement of the target portrait is driven based on the movement driving parameters to generate a portrait movement animation.

2. The portrait motion driving method according to claim 1, characterized in that: The target feature extraction model includes a time-domain convolutional feature extractor and a multi-layer transformer encoder; The step of extracting the audio state features of the target audio data by using a preset target feature extraction model includes: Inputting the target audio data into the time-domain convolutional feature extractor, so as to perform low-level feature extraction on the target audio data through the time-domain convolutional feature extractor to generate audio phoneme features; the audio factor features include rhythm features, emotion features and intonation features; The audio phoneme features are encoded with the multi-layer transformer encoder to obtain high-level features, and the high-level features are output as audio mood features.

3. The portrait motion driving method according to claim 2, characterized in that: The step of performing low-level feature extraction on the target audio data by the time domain convolution feature extractor to generate audio phoneme features comprises: The time domain audio features of the target audio data are extracted by the time domain convolution feature extractor, and the time domain audio features are converted into frequency domain audio features, so as to perform multi-dimensional feature extraction on the frequency domain audio features to form audio phoneme features.

4. The portrait motion driving method according to claim 2, characterized in that: The step of encoding the audio phoneme features by using the multi-layer transformer encoder and outputting the high-level features as audio expression features includes: The multi-head self-attention mechanism in the multi-layer transformer encoder is used to read the temporal dependency and cross-feature dependency of the audio phoneme features, and based on the temporal dependency and the cross-feature dependency, a layer-by-layer feature interaction weighting mechanism is used to perform advanced encoding on the audio phoneme features to generate audio mood features.

5. The portrait motion driving method according to claim 1, characterized in that: The step of determining a target expression parameter generation model comprises: A pre-trained model is constructed using the Transformers architecture and the CNN architecture, and an overlapping frame prediction technology is introduced into the pre-trained model to obtain an initial expression parameter generation model; Inputting the pre-acquired sample audio data into the initial expression parameter generation model to obtain the predicted action driving parameters output by the initial expression parameter generation model; The predicted action driving parameters are close to the real action driving parameters of the sample audio data, and the initial expression parameter generation model is trained using a target loss function; When the initial expression parameter generation model meets the preset training conditions, the trained initial expression parameter generation model is used as the target expression parameter generation model.

6. The portrait motion driving method according to claim 1 or 5, characterized in that: The calculation formula of the objective loss function includes: In the formula, represents the target loss function; represents the real action driving parameters; represents the predicted action driving parameter; where, represents the time window from the pth moment to the wth moment.

7. The portrait motion driving method according to claim 1, characterized in that: The driving of the target portrait to perform a head movement based on the movement driving parameters to generate a portrait movement animation includes: Determine a portrait action driving model; the portrait action driving model is composed of a lip action driving module, an expression style driving module, a head posture driving module and an animation generation module; The action driving parameters and the target portrait are used as input data, and the input data are respectively input into the lip action driving module, the expression style driving module and the head posture driving module to obtain a first driving result output by the lip action driving module, a second driving result output by the expression style driving module and a third driving result output by the head posture driving module; In the animation generation module, the first driving result, the second driving result and the third driving result are aligned and fused in time frames based on the target audio data to generate a portrait action animation.

8. A portrait motion driving device, characterized in that: include: A feature extraction module is used to obtain target audio data and extract audio expression features of the target audio data using a preset target feature extraction model; the audio expression features include lip movement features, expression style features and head posture features; A model determination module is used to determine a target expression parameter generation model; the target expression parameter generation model is obtained by training with a target loss function based on an overlapping frame prediction technology; A parameter prediction module, used for inputting the audio expression feature into the target expression parameter generation model to obtain the action driving parameters output by the target expression parameter model; The action driving module is used to obtain a target portrait, and drive the head action of the target portrait based on the action driving parameters to generate a portrait action animation.

9. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the portrait motion driving method according to any one of claims 1 to 7.

10. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the portrait motion driving method according to any one of claims 1 to 7 are performed.

Images