Digital human video generation method and device, and computer device

Abstract

This invention relates to the field of digital human technology, providing a method, apparatus, and computer device for digital human video generation. The method includes acquiring text information, audio information, and style control parameters; performing semantic analysis and feature extraction on the text and audio information to obtain text-driven and audio-driven expression features, which are then co-modulated to obtain facial expression parameter sequences and lip-shape parameter sequences; generating a stylized 3D face model of the digital human based on the style control parameters; and using the facial expression parameter sequences and lip-shape parameter sequences to drive the stylized 3D face model to generate the corresponding digital human video. Through the co-modulation of text information, audio information, and style control parameters, the synchronization of facial expressions and lip-shapes is enhanced, improving the naturalness and accuracy of digital human video generation and addressing the problem of incomplete synchronization between lip-shapes and facial expressions when handling complex contexts and emotional expressions.

Description

Technical Field

[0001] This application relates to the field of digital human technology, and in particular to digital human video generation methods, apparatus and computer equipment. Background Technology

[0002] With the rapid development of artificial intelligence and computer graphics technologies, digital human technology has been widely applied in various fields. Digital humans, as representatives of virtual characters, can provide efficient and interactive solutions in industries such as virtual reality, film production, game development, and intelligent customer service. The generation of digital humans involves multiple disciplines, including graphics, computer vision, speech synthesis, and natural language processing. Especially in the film production, advertising, education, and entertainment industries, the application of digital humans is gradually becoming one of the core technologies.

[0003] Among related technologies, digital human video generation technology generally relies on first recording live performances or generating text-based speech, and then combining this data with a 3D facial model using computer graphics technology to match expressions, lip movements, and actions. Common implementation methods include text-based speech generation and facial expression synthesis, generating corresponding facial animations through voice-driven technology, and then synchronizing these facial animations with target audio so that the virtual character can interact with preset dialogue content.

[0004] Regarding the above-mentioned technical solutions, although the digital human video generation technology based on voice and facial expression can effectively achieve synchronization between virtual characters and audio content, existing technologies often exhibit inconsistencies between voice and facial expressions when dealing with complex contexts and emotional expressions. In particular, in scenarios where emotional expression is complex or varied, there is a problem that lip movements and facial expressions cannot be fully synchronized. Summary of the Invention

[0005] To address the issue of lip movements and facial expressions not being fully synchronized when dealing with complex contexts and emotional expressions, this application provides a digital human video generation method, apparatus, and computer equipment.

[0006] This invention provides a method for generating digital human videos, comprising: acquiring text information, audio information, and style control parameters; performing semantic parsing on the text information to obtain text-driven expression features; extracting speech features from the audio information to obtain prosodic feature vectors and pronunciation dynamic vectors; fusing the prosodic feature vectors and pronunciation dynamic vectors to obtain audio-driven expression features; co-modulating the text-driven expression features and the audio-driven expression features to obtain facial expression parameter sequences and lip shape parameter sequences; generating a stylized 3D face model of the digital human based on the style control parameters; using the facial expression parameter sequences to drive the expression of the stylized 3D face model to obtain a facial animation sequence; using the lip shape parameter sequences to drive the lip shape of the stylized 3D face model to obtain a lip shape animation sequence; performing time alignment processing on the facial animation sequence and the lip shape animation sequence to obtain complete animation motion data; and synchronously rendering the animation motion data and the audio information to generate a corresponding digital human video.

[0007] As a preferred embodiment, the step of semantically parsing the text information to obtain text-driven expressive features includes: segmenting the text information into sentences to obtain a set of sentence elements and inter-sentence dependencies; extracting intra-sentence semantic features based on the set of sentence elements; extracting a global semantic context vector and inter-sentence sentiment flow features based on the inter-sentence dependencies; fusing the intra-sentence semantic features with the global semantic context vector to obtain a multi-level semantic feature vector; performing cross-attention mapping between the inter-sentence sentiment flow features and the multi-level semantic feature vector to obtain an emotion-enhanced semantic vector and an emotion feature representation; and using the emotion feature representation to perform style adjustment on the emotion-enhanced semantic vector to obtain a style that aligns with the text meaning. Figure 1 The semantic feature vector and the sentiment feature vector are obtained by fusing the semantic feature vector and the sentiment feature vector to obtain text-driven expression features.

[0008] As a preferred embodiment, the step of extracting speech features from the audio information to obtain prosodic feature vectors and articulation dynamic vectors, and fusing the prosodic feature vectors and articulation dynamic vectors to obtain audio-driven expression features includes: performing frame segmentation processing on the audio information, extracting temporal envelope features and frequency domain spectrogram features for each frame, extracting short-term prosodic patterns based on the temporal envelope features, fusing the short-term prosodic patterns with rhythm information from neighboring frames to obtain a temporally continuous prosodic feature vector; extracting frequency domain articulation features using the frequency domain spectrogram features; performing principal component decoupling processing on the frequency domain articulation features, extracting articulation state sequences and articulation boundary change information, encoding and mapping the articulation state sequences and articulation boundary change information to obtain an articulation dynamic vector; and fusing the temporally continuous prosodic feature vectors and the articulation dynamic vectors in the time dimension to obtain audio-driven expression features.

[0009] As a preferred embodiment, the step of co-modulating the text-driven expression features and the audio-driven expression features to obtain a facial expression parameter sequence and a lip shape parameter sequence includes: inputting the text-driven expression features into an expression channel encoder to extract multi-level facial expression driving vectors; inputting the audio-driven expression features into a lip shape channel encoder to extract multi-band pronunciation mapping vectors; performing temporal alignment processing on the multi-level facial expression driving vectors and the multi-band pronunciation mapping vectors to obtain expression feature pairs with the same time axis; constructing a fused expression state map based on the expression feature pairs; performing feature decoding on the fused expression state map to obtain facial expression parameters and lip shape parameters; and co-modulating the facial expression parameters and the lip shape parameters to eliminate expression conflicts and rhythm misalignments to obtain a facial expression parameter sequence and a lip shape parameter sequence.

[0010] As a preferred embodiment, the steps of generating a stylized 3D face model of a digital human based on the style control parameters, driving the stylized 3D face model to generate facial animation sequences using the facial expression parameter sequence, and driving the stylized 3D face model to generate lip animation sequences using the lip shape parameter sequence include: loading a corresponding basic 3D face model and style mapping weights from a preset digital human parameter template library based on the style control parameters; performing style deformation and material mapping on the basic 3D face model using the style mapping weights to obtain a stylized 3D face model; inputting the facial expression parameter sequence into a preset muscle control module in the stylized 3D face model to drive changes in facial expression control points in the stylized 3D face model to obtain expression-driven results; inputting the lip shape parameter sequence into a preset lip deformation module in the stylized 3D face model to drive geometric deformation of the lip and teeth region in the stylized 3D face model to obtain lip shape-driven results; performing frame-by-frame pose fusion processing on the expression-driven results to generate facial animation sequences; and performing frame-by-frame pose fusion processing on the lip shape-driven results to generate lip animation sequences.

[0011] As a preferred embodiment, the step of performing time alignment processing on the facial animation sequence and the lip-sync animation sequence to obtain complete animation motion data, and synchronously rendering the animation motion data with the audio information to generate a corresponding digital human video includes: resampling the facial animation sequence and the lip-sync animation sequence along the time axis, establishing a unified frame rate and a reference starting frame based on the sampling results, performing time alignment processing on the facial animation sequence and the lip-sync animation sequence using the unified frame rate and the reference starting frame, and extracting the facial expression motion vector and lip-sync motion vector for each frame; performing fusion weight modeling on the facial expression motion vector and the lip-sync motion vector to construct a fusion driving vector sequence, inputting the fusion driving vector sequence into a preset motion combination module in the stylized 3D face model to generate complete animation motion data; performing alignment and slicing processing on the audio information, extracting audio frame segments of the corresponding frames, and synchronously rendering the animation motion data and the audio frame segments at the frame level to generate a corresponding digital human video.

[0012] As a preferred embodiment, the steps of aligning and slicing the audio information, extracting audio frame segments from corresponding frames, and performing frame-level synchronized rendering of the animation motion data and the audio frame segments to generate the corresponding digital human video include: dividing the audio information into segments with the same time interval at a preset frame rate to obtain a set of audio frame segments; extracting audio features from each audio frame segment in the set of audio frame segments to obtain temporal audio features and frequency-domain audio features; constructing an audio feature vector based on the temporal and frequency-domain audio features; aligning the audio feature vector with the motion feature vector of the corresponding frame in the animation motion data in a temporal sequence to obtain an aligned audio feature vector; adjusting the frame-level weights of the aligned audio feature vector and the motion feature vector to obtain synchronized rendering parameters; and inputting the synchronized rendering parameters into a preset rendering module for audio-video synchronized rendering to generate the corresponding digital human video.

[0013] This application also provides a digital human video generation device, comprising: an acquisition module for acquiring text information, audio information, and style control parameters, performing semantic parsing on the text information to obtain text-driven expression features; a fusion module for extracting speech features from the audio information to obtain prosodic feature vectors and pronunciation dynamic vectors, fusing the prosodic feature vectors and the pronunciation dynamic vectors to obtain audio-driven expression features; a modulation module for co-modulating the text-driven expression features and the audio-driven expression features to obtain a facial expression parameter sequence and a lip shape parameter sequence; a driving module for generating a stylized 3D face model of the digital human based on the style control parameters, using the facial expression parameter sequence to drive the expression of the stylized 3D face model to obtain a facial animation sequence, and using the lip shape parameter sequence to drive the lip shape of the stylized 3D face model to obtain a lip shape animation sequence; and a generation module for time-aligning the facial animation sequence and the lip shape animation sequence to obtain complete animation motion data, synchronously rendering the animation motion data and the audio information to generate a corresponding digital human video.

[0014] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor executes the computer program to implement the digital human video generation method described above.

[0015] Compared with existing technologies, this application has the following beneficial effects: high synchronization. Through the coordinated modulation of text and audio information, natural and synchronized facial expressions and lip-sync animations are generated based on the text and audio, achieving a high degree of coordination between the digital human and the text and audio information. At the same time, the introduction of style control parameters can adjust the appearance and performance style of the digital human according to different needs, making it more personalized and expressive, enhancing the synchronization of facial expressions and lip-sync, improving the naturalness and accuracy of digital human video generation, and improving the problem of incomplete synchronization between lip-sync and facial expressions when dealing with complex contexts and emotional expressions. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] The structures, proportions, sizes, etc., shown in the accompanying drawings of this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed in the specification, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0018] Figure 1 This is a flowchart illustrating the digital human video generation method provided in an embodiment of the present invention;

[0019] Figure 2 This is a schematic block diagram of the structure of the digital human video generation device provided in an embodiment of the present invention;

[0020] Figure 3 This is a schematic block diagram of the structure of a computer device provided in an embodiment of the present invention.

[0021] Explanation of reference numerals in the attached figures:

[0022] 10. Digital human video generation device; 11. Acquisition module; 12. Fusion module; 13. Modulation module; 14. Driving module; 15. Generation module; 20. Computer equipment; 21. Memory; 22. Processor. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] The flowchart shown in the attached diagram is for illustrative purposes only and does not necessarily include all content and operations / steps, nor does it necessarily have to be performed in the order described. For example, some operations / steps can be broken down, combined, or partially merged, so the actual execution order may change depending on the actual situation.

[0025] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of the application. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0026] It should also be further understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0027] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0028] Example 1:

[0029] like Figure 1 As shown, this application provides a digital human video generation method, including steps S100 to S500.

[0030] Step S100: Obtain text information, audio information, and style control parameters; perform semantic parsing on the text information to obtain text-driven expressive features.

[0031] In this step, the input text information is first obtained through the natural language processing module, and preprocessed using methods such as word segmentation and syntactic analysis. Specifically, the syntactic dependency parsing module extracts the grammatical structure from the text, generating a set of sentence elements, and then uses the BERT (Bidirectional Encoder Representations from Transformers) model to perform semantic understanding of the text, obtaining deep semantic features of each sentence. The BERT model is a commonly used deep learning model in natural language processing tasks; it can effectively capture the relationships between words, thus obtaining more accurate semantic representations.

[0032] For example, given the input text "He said happily, 'It's a nice day today!'", the BERT model can extract the sentiment vector of the text, such as associating "happy" with "nice day", and generate a semantic vector with emotional color.

[0033] Step S200: Extract speech features from the audio information to obtain prosodic feature vectors and pronunciation dynamic vectors. Fuse the prosodic feature vectors and pronunciation dynamic vectors to obtain audio-driven expression features.

[0034] In this step, the audio information is first fed into the acoustic feature extraction module, where the frequency domain features of the audio are extracted using Short-Time Fourier Transform (STFT). Specifically, the audio signal is first processed by frame segmentation, and then the temporal envelope and frequency domain spectrogram of each frame are calculated to obtain the audio features of each frame. Prosodic features are extracted using an acoustic model (such as a Deep Neural Network (DNN) or a Long Short-Term Memory Network (LSTM) to obtain the prosodic feature vector of the audio. Furthermore, the articulation dynamic vector is obtained by modeling the articulation state of the audio. The articulation features of the audio are extracted using MFCC (Mel-Frequency Cepstral Coefficients), and combined with the preceding and following speech information to obtain the dynamic change vector of articulation. The prosodic feature vector and the articulation dynamic vector are fused using a weighted average method to obtain the audio-driven expressive features.

[0035] For example, in the audio message "'It's a nice day today!'", by extracting its prosodic features such as pitch, intonation, and speech rate, as well as the boundaries and pauses of pronunciation, an audio-driven expressive feature vector can be generated to describe the phonological changes in the audio.

[0036] Step S300: Co-modulate text-driven expression features and audio-driven expression features to obtain facial expression parameter sequences and lip shape parameter sequences.

[0037] In this step, text-driven and audio-driven expressive features are fused through a dual-channel collaborative modulation network. Specifically, a Transformer-based fusion mechanism is employed to interact the emotional features of the text with the prosodic features of the audio through multiple channels, generating a global expressive feature. This global expressive feature is then fed into the facial expression generation module and the lip-sync generation module. A shared neural network model is used to generate sequences of facial expression parameters and lip-sync parameters, respectively.

[0038] For example, in the text “He said happily, ‘It’s a nice day!’”, the Transformer model captures the emotion of “happiness” and generates appropriate facial expressions and lip-sync parameters based on the “pitch and speed of speech” in the audio, ensuring that facial expressions and lip-sync are synchronized with the speech content.

[0039] Step S400: Generate a stylized 3D face model of the digital human based on style control parameters, use facial expression parameter sequence to drive the expression of the stylized 3D face model to obtain a facial animation sequence, and use lip shape parameter sequence to drive the lip shape of the stylized 3D face model to obtain a lip shape animation sequence.

[0040] In this step, a suitable 3D digital human model is first selected based on style control parameters, and its shape and material are adjusted according to the user-defined style. Specifically, style control parameters include facial features (such as the shape of the eyes and mouth) and facial expression style (such as exaggerated or natural). A style transfer network is used to stylize the base 3D model. Then, facial expression parameter sequences and lip shape parameter sequences are used to drive the expression and lip shape of the stylized 3D face model, respectively. Expression driving adjusts facial muscles through a muscle deformation module, while lip shape driving controls lip movement through a lip deformation module, thereby generating facial animation sequences and lip shape animation sequences.

[0041] For example, by adjusting the style control parameters, the shape of the digital human's mouth can be adjusted to a smiling or closed state, while facial expressions will produce corresponding facial features such as raised eyebrows and smiling eyes based on the emotion of "happiness".

[0042] Step S500: Time-align the facial animation sequence and the lip-sync animation sequence to obtain complete animation motion data. Simultaneously render the animation motion data and audio information to generate the corresponding digital human video.

[0043] In this step, the facial animation sequence and the lip-sync animation sequence are time-aligned to ensure accurate matching on the timeline. Specifically, the Dynamic Time Warping (DTW) algorithm is used to align the timelines of facial expressions and lip movements, ensuring precise synchronization between lip movements and audio content. Next, the synchronized animation data and audio information are temporally synthesized, and a digital human video is generated through a rendering engine. During rendering, based on the facial and lip-sync animation of each frame, combined with the temporal information of the audio, the final digital human video frame sequence is generated and post-processed to enhance it, thereby improving image quality.

[0044] For example, through dynamic time warping algorithms, pauses and stresses in audio can be precisely synchronized with micro-expression changes in facial animation, ensuring that the lip movements of the digital human in the final video perfectly match the speech content and that the expressions are natural.

[0045] In this embodiment, by acquiring text information, audio information, and style control parameters, the text information is first semantically parsed to obtain text-driven expression features. Next, speech features are extracted using the audio information to obtain prosodic feature vectors and pronunciation dynamic vectors, which are then fused to obtain audio-driven expression features. Then, the text-driven and audio-driven expression features are co-modulated to obtain facial expression parameter sequences and lip-sync parameter sequences. Based on the style control parameters, a stylized 3D face model of the digital human is generated, and the facial expression parameter sequences are used to drive the expression of this stylized 3D face model to obtain a facial animation sequence; the lip-sync parameter sequences are used to drive the lip-sync of the stylized 3D face model to obtain a lip-sync animation sequence. Finally, the facial animation sequence and the lip-sync animation sequence are time-aligned to obtain complete animation motion data, which is then synchronously rendered with the audio information to ultimately generate the corresponding digital human video. This technology achieves a high degree of coordination between digital humans and text and audio information, particularly in the synchronization of facial expressions and lip movements. This enhances the naturalness and accuracy of digital human video generation and addresses the issue of incomplete synchronization between lip movements and facial expressions when handling complex contexts and emotional expressions. Furthermore, the introduction of style control parameters allows for adjustments to the digital human's appearance and performance style to suit different needs, making it more personalized and expressive. Through the collaborative modulation of text-driven and audio-driven expressive features, conflicts between lip movements and facial expressions are effectively eliminated, improving the digital human's performance capabilities in multi-emotional and multi-contextual situations. This overcomes the synchronization problems of existing technologies in complex emotional and speech expressions, providing a more precise and flexible solution for the application of virtual characters in film and television production, game interaction, and artificial intelligence.

[0046] Example 2:

[0047] In step S100, the step of semantically parsing the text information to obtain text-driven expressive features specifically includes:

[0048] The text information is segmented into sentences to obtain a set of sentence elements and inter-sentence dependencies. A multi-layer semantic coding network is used to extract intra-sentence semantic features based on the set of sentence elements. A context modeling network is applied to extract global semantic context vectors and inter-sentence sentiment flow features based on inter-sentence dependencies.

[0049] Intra-sentence semantic features are extracted through a multi-layer semantic coding network. Specifically, in this step, the BERT (Bidirectional Encoder Representations from Transformers) model is used to bidirectionally encode the input text, generating contextual information for each sentence and capturing intra-sentence semantic relationships. When processing text, the BERT model can simultaneously consider the left and right context of words, thus generating a deep semantic representation for each word. This model can effectively understand the relationships between words in the sentence set and generate intra-sentence semantic feature vectors.

[0050] For example, when given the input text "He said happily, 'It's a nice day today!'", the BERT model can identify the semantic relationship between "happy" and "nice weather" and generate corresponding semantic feature vectors for these words, reflecting the emotional tone of the text.

[0051] This paper extracts global semantic context vectors and inter-sentence sentiment flow features through a context modeling network. Specifically, in this step, either an LSTM (Long Short-Term Memory) network or a Transformer model is used to handle inter-sentence dependencies in the text. LSTM can record contextual information over a long period of time through memory units, while Transformer uses a self-attention mechanism to capture the relationships between different parts of a sentence, generating a global semantic context vector. Simultaneously, this model extracts inter-sentence sentiment flow features to represent the transmission relationship of sentiment between sentences.

[0052] For example, when processing the text “He said happily, ‘It’s a nice day!’”, the LSTM or Transformer model can capture the “happy” emotion from the previous sentence to the following sentence “It’s a nice day” and generate a global semantic context vector containing emotional information and inter-sentence emotion flow features.

[0053] By fusing intra-sentence semantic features with global semantic context vectors, a multi-level semantic feature vector is obtained. Then, cross-attention mapping is performed between inter-sentence sentiment flow features and the multi-level semantic feature vector to obtain sentiment-enhanced semantic vectors and emotion feature representations.

[0054] The fusion is achieved through a cross-attention mechanism. Specifically, a multi-head self-attention mechanism is used to weight and fuse intra-sentence semantic features with the global semantic context vector. The multi-head attention mechanism, through multiple independent attention computation paths, can simultaneously focus on information from different parts, thereby obtaining multi-level contextual information. Next, inter-sentence sentiment flow features are cross-mapped with the fused semantic features to ensure that sentiment information effectively influences the text's expression, generating sentiment-enhanced semantic vectors and emotional feature representations.

[0055] For example, when processing the text “He said happily, ‘It’s a nice day!’”, the cross-attention mechanism combines the emotional information of “happy” with the text content of “good weather”. The resulting emotion-enhanced semantic vector will highlight the positive emotion, ensuring that the digital human’s expressions and tone of voice in the generated digital human video can accurately reflect this emotion.

[0056] Style adjustment is performed on the sentiment-enhanced semantic vector using emotion feature representation to obtain a style that aligns with the text meaning. Figure 1 The semantic feature vector and the sentiment feature vector are consistent.

[0057] The style adjustment is achieved through a style transfer network, specifically a style transfer network, which adjusts the style of the sentiment-enhanced semantic vectors. The style transfer network adjusts parameters such as activation values ​​and weights to make the sentiment-enhanced semantic vectors more consistent with preset style requirements, such as adjusting the intonation of speech or the facial expressions of a digital human. This ensures that the generated semantic and sentiment features are highly consistent with the sentiment intent of the input text.

[0058] For example, in the text “He said happily, ‘It’s a nice day!’”, style adjustment ensures that the “happy” emotion in the sentiment enhancement semantic vector is amplified, thereby generating digital human expression features that conform to “pleasant” expressions and positive emotions.

[0059] By fusing semantic feature vectors and sentiment feature vectors, text-driven expression features are obtained.

[0060] Through weighted fusion, specifically by using a weighted superposition method, semantic feature vectors and sentiment feature vectors are fused to ensure that the semantics and sentiment of the text complement and reinforce each other when generating text-driven expressive features. The fused expressive features will accurately reflect the text information and drive the digital human's facial expressions, actions, and other performance forms.

[0061] For example, for the sentence "It's a nice day today!", the final text-driven expressive features will include emotional components such as "pleasant" and "positive", and drive the digital human to display natural facial expressions such as a happy smile and a gentle smile.

[0062] In step S200, speech features are extracted from the audio information to obtain prosodic feature vectors and articulation dynamic vectors. The prosodic feature vectors and articulation dynamic vectors are then fused to obtain audio-driven expressive features. This step specifically includes:

[0063] The audio information is divided into frames, and the temporal envelope features and frequency spectrum features of each frame are extracted. Based on the preset acoustic feature extraction network, short-time prosodic patterns are extracted from the temporal envelope features. The short-time prosodic patterns are fused with the rhythm information of neighboring frames to obtain a temporally continuous prosodic feature vector.

[0064] Through frame segmentation and feature extraction, specifically, the audio signal is processed into frames using Short-Time Fourier Transform (STFT), typically with a frame length of 20ms and a frame shift of 10ms. STFT converts the audio signal of each frame into temporal envelope features and frequency domain spectrogram features. The temporal envelope features reflect the amplitude changes of the audio signal, while the frequency domain spectrogram features provide frequency information. Then, the temporal envelope features are processed using a pre-defined acoustic feature extraction network (such as a network based on a Convolutional Neural Network (CNN)) to extract short-time prosodic patterns. Next, the short-time prosodic patterns are fused with the rhythmic information of adjacent frames to obtain a prosodic feature vector containing temporal continuity.

[0065] For example, given the input audio signal "Today is a nice day," multiple audio frames are obtained through STFT framing. Temporal envelope features and frequency spectrum features are extracted from each frame. An acoustic feature extraction network further processes the temporal envelope features, extracting rhythmic patterns and fusing them with rhythmic information from neighboring frames to ultimately generate a prosodic feature vector containing temporal continuity. These prosodic features describe the rhythmic characteristics of the audio signal, such as rhythm and pitch variations.

[0066] Frequency domain articulation features are extracted using frequency domain spectrogram features; principal component decoupling is performed on the frequency domain articulation features to extract articulation state sequences and articulation boundary change information; the articulation state sequences and articulation boundary change information are encoded and mapped to obtain the articulation dynamic vector.

[0067] Frequency domain feature processing, specifically the Mel Frequency Cepstral Coefficients (MFCC) algorithm, is used to process the frequency domain spectrogram features and extract frequency domain articulation features. MFCC is widely used in speech processing; it can extract spectral features representing articulation information and reflecting the acoustic characteristics of audio. Then, principal component analysis (PCA) or other dimensionality reduction algorithms are applied to the extracted frequency domain articulation features for principal component decoupling, extracting articulation state sequences and articulation boundary change information. The articulation state sequence represents the changes in articulation states, while the articulation boundary change information represents pauses, phrasing, and other information in speech. Through encoding and mapping, these features are transformed into dynamic articulation vectors to drive lip-sync generation.

[0068] For example, after performing MFCC processing on the audio signal "Today is a good weather", frequency domain pronunciation features can be obtained, reflecting the frequency characteristics of different syllables. Then, PCA is used to reduce the dimensionality of the pronunciation features, extracting the pronunciation state changes between syllables (such as the pronunciation state change from "today" to "is"), and converting them into pronunciation dynamic vectors through encoding mapping.

[0069] By using an attention mechanism, temporally continuous prosodic feature vectors and pronunciation dynamic vectors are fused along the temporal dimension to obtain audio-driven expressive features.

[0070] Specifically, a multi-head self-attention mechanism is used to fuse prosodic feature vectors and articulation dynamic vectors over time. This attention mechanism effectively combines the prosodic information of each frame with the corresponding articulation state and articulation boundary information to generate a global audio-driven expressive feature. During the fusion process, the attention mechanism automatically assigns weights based on the temporal characteristics of the audio signal, ensuring that important articulation information and prosodic information are effectively combined, thereby obtaining a more accurate audio-driven expressive feature.

[0071] For example, for the audio signal "Today is a good day", prosodic features and articulatory dynamics features are fused through an attention mechanism. In this fusion process, the prosodic information such as rhythm and pitch of the two syllables "today" and "good day" in the audio is combined with the corresponding articulatory dynamics (such as changes in mouth shape and pauses in speech) to obtain a precise audio-driven expressive feature.

[0072] In step S300, the step of co-modulating text-driven and audio-driven expressive features to obtain facial expression parameter sequences and lip shape parameter sequences specifically includes:

[0073] Text-driven facial expression features are input into the facial expression channel encoder to extract multi-level facial expression driving vectors, and audio-driven facial expression features are input into the lip-sync channel encoder to extract multi-frequency pronunciation mapping vectors.

[0074] Multi-level facial expression driving vectors and multi-frequency pronunciation mapping vectors are extracted through an expression channel encoder and a lip-sync channel encoder. Specifically, the expression channel encoder employs a combined model based on a convolutional neural network (CNN) and a Transformer architecture. Text-driven expression features are input into the expression channel encoder, first extracting low-level emotional features (such as mood tone) through convolutional layers, then capturing global emotional patterns using a Transformer module, ultimately generating multi-level facial expression driving vectors. Simultaneously, audio-driven expression features are input into the lip-sync channel encoder, which is based on an LSTM (Long Short-Term Memory) model to capture dynamic changes in pronunciation and generate multi-frequency pronunciation mapping vectors through segmented spectral analysis. The temporal dependency mechanism of LSTM enables it to extract the evolutionary patterns of pronunciation from the audio-driven expression features.

[0075] For example, given the text-driven expression feature "pleasant emotional tone" and the audio signal "today is a good day", the facial expression channel encoder will extract multi-level facial expression features of the "pleasant" emotion from the text-driven expression feature, such as mouth opening and eye changes; while the lip shape channel encoder will extract the pronunciation characteristics within the frequency band from the audio-driven expression feature, such as the time sequence information of lip closing, opening and closing and tongue movement.

[0076] The multi-level facial expression driving vector and the multi-frequency pronunciation mapping vector are temporally aligned to obtain expression feature pairs with the same time axis. Based on the expression feature pairs, a fused expression state map is constructed. The fused expression state map is then decoded to obtain facial expression parameters and lip shape parameters.

[0077] Dynamic Time Warping (DTW) is used to temporally align multi-level facial expression driving vectors with multi-frequency pronunciation mapping vectors. DTW effectively aligns feature sequences with inconsistent time lengths, ensuring consistency between the two sequences in the temporal dimension. Then, a fused expression state map is constructed using the aligned expression features. This fused expression state map combines the emotional and action information of all key nodes on the timeline of facial expressions and lip movements. Subsequently, a Graph Convolutional Network (GCN) is used to decode the features of the fused expression state map to generate facial expression parameters (control parameters for facial muscles) and lip movement parameters (pronunciation parameters used to drive the tongue, lips, etc.).

[0078] For example, in the input text expression "Today is a good day", the multi-level facial expression driving vector representing the emotion of "pleasure" and the multi-frequency pronunciation dynamic vector on the corresponding time axis are aligned through DTW to generate a fused expression state map. This map will assign the mouth opening feature and the eye squinting feature to the two words "today", and assign specific features such as tongue and lip closure to "good day", ultimately predicting facial expression parameters and mouth shape parameters.

[0079] By co-modulating facial expression parameters and lip shape parameters, expression conflicts and rhythm misalignments are eliminated, resulting in facial expression parameter sequences and lip shape parameter sequences.

[0080] By leveraging a collaborative modulation model based on a Multi-Modal Generative Adversarial Network (MM-GAN), facial expression parameters and lip-shape parameters are jointly optimized to proactively detect potential conflicts between expressions and lip shapes. For example, certain facial features may produce unnatural facial tension, or the lip shape may fail to synchronize with the facial expression when expressing a specific pronunciation. The MM-GAN's generative network is responsible for generating consistency parameters between expressions and lip shapes, while the adversarial network is used to evaluate the harmony and naturalness of the generated results. In the adversarial optimization process, conflicts between expressions and lip shapes (such as overly exaggerated expressions or mismatched lip shapes) are fed back and adjusted through a loss function, ultimately generating natural and coherent sequences of facial expression parameters and lip shape parameters.

[0081] For example, in the sentence "Today is a nice day", if the smiling expression generated by the facial expression parameter sequence conflicts with the closed mouth shape of the "weather" syllable in the mouth shape parameter sequence, MM-GAN will detect and optimize to reduce the smile amplitude to match normal speech, while adjusting the fluency of the mouth shape when connecting vowels and consonants, and finally generate a natural and coherent facial expression parameter sequence and mouth shape parameter sequence.

[0082] In step S400, the steps of generating a stylized 3D face model of the digital human based on style control parameters, using facial expression parameter sequences to drive the expression of the stylized 3D face model to obtain a facial animation sequence, and using lip shape parameter sequences to drive the lip shape of the stylized 3D face model to obtain a lip shape animation sequence specifically include:

[0083] Based on style control parameters, the corresponding basic 3D face model and style mapping weights are loaded from the preset digital human parameter template library. The style mapping weights are then used to perform style deformation and material mapping on the basic 3D face model to obtain a stylized 3D face model.

[0084] By loading a base 3D face model and style mapping weights, specifically, firstly, a base 3D face model is selected from a digital human parameter template library based on user-defined style control parameters. The template library contains various 3D models, such as "male," "female," "child," and base models of different races or ages. Next, corresponding style mapping weights are loaded according to the style control parameters, including style-related deformation weights (such as facial proportion adjustments) and material weights (such as skin texture and eye color). Through a style transfer module based on a GAN (Generative Adversarial Network), style deformation and material mapping are applied to the base 3D face model, ultimately generating a stylized 3D face model that meets the style requirements.

[0085] For example, when the user selects style control parameters "young woman" and "fair skin", the template library will load a basic model "female character", and adjust the shape ratios such as eye size and face width through style mapping weights. At the same time, texture optimization is performed on the skin material to make the skin呈现白皙无瑕的效果. The finally generated stylized 3D face model has the characteristics of a young woman and the material effect that meets personal requirements.

[0086] Input the facial expression parameter sequence into the muscle control module preset in the stylized 3D face model to drive the change of facial expression control points in the stylized 3D face model and obtain the expression-driven result.

[0087] Drive the expression control points through the muscle control module. Specifically, the facial expression parameter sequence realizes expression changes by being input into the facial muscle control module in the stylized 3D face model. This muscle control module is built based on Blendshape technology and skeletal animation technology. Each point represents the change amplitude of key facial regions (such as eyebrows, corners of the mouth, corners of the eyes, etc.). By decoding the parameters into the corresponding Blendshape weights, specific expression control points in the stylized 3D face model are driven to change. According to the time sequence of the expression parameters, the expression-driven result is gradually generated.

[0088] For example, if the input facial expression parameter sequence corresponds to the emotion of "pleasure", by driving the facial muscle control module, the corners of the mouth on the model will gradually lift, the eye muscles will呈现自然的微眯状态, and the eyebrows will be slightly raised. Finally, the expression-driven result produces a facial expression matching the emotion of "happy", showing an obvious smile.

[0089] Input the lip shape parameter sequence into the lip deformation module preset in the stylized 3D face model to drive the geometric deformation of the lip and tooth region in the stylized 3D face model and obtain the lip shape-driven result.

[0090] Drive the lip and tooth region to change through the lip deformation module. Specifically, the lip shape parameter sequence is input into the lip deformation module of the stylized 3D face model. This module is based on the MorphTarget Animation method and tongue-tooth dynamic binding technology to perform real-time driving on the lip position, mouth opening angle, tongue movement path, etc. Through the geometric deformation of the lip and tooth region, the pronunciation dynamic sequence is mapped into real lip shape changes, and natural transitions can be achieved from vowels to consonants.

[0091] For example, in the pronunciation dynamic sequence, the phonemes of "Today is a nice day" correspond to the lip shape-driven result. "今" corresponds to the lips closing and then opening; "天" corresponds to the mouth opening slightly and showing the tip of the tongue; "好天气" corresponds to the smooth lip and tooth changes of continuous vowel phonemes. The finally generated lip shape-driven result can be synchronized with the actual pronunciation content.

[0092] The facial expression-driven results are processed frame-by-frame with pose fusion to generate a facial animation sequence, and the lip shape-driven results are processed frame-by-frame with pose fusion to generate a lip shape animation sequence.

[0093] Animation sequences are generated through frame-by-frame pose fusion. Specifically, a pose processing mechanism based on a time-series prediction model (such as GRU, Gated Recurrent Unit) and optimized weights is employed to calculate the expression-driven and lip-sync-driven results frame by frame. This model can predict pose transitions between each frame and optimize the motion coordination between the expression region and the lip-sync region. Through weight allocation, the pose results of facial animation and lip-sync animation are accurately fused to generate a coherent and natural animation sequence.

[0094] For example, when generating facial animation sequences, the process of "gradually deepening smiles" is controlled by frame-by-frame pose prediction, while the dynamic relationship between eyebrows and corners of the mouth is corrected to make facial expressions more natural and context-appropriate. For lip-sync animation sequences, the pronunciation closure state of "weather" is predicted by optimizing lip and teeth coordination, seamlessly connecting with facial expressions. The final generated animation sequence not only expresses pleasant emotions but also accurately matches the lip shape of each phoneme.

[0095] In step S500, the facial animation sequence and the lip-sync animation sequence are time-aligned to obtain complete animation motion data. The animation motion data and audio information are then rendered synchronously to generate the corresponding digital human video. Specifically, this includes:

[0096] The facial animation sequence and lip-sync animation sequence are resampled on the timeline, and a unified frame rate and a reference starting frame are established based on the sampling results. The unified frame rate and the reference starting frame are used to perform time alignment processing on the facial animation sequence and lip-sync animation sequence, and the facial expression motion vector and lip-sync motion vector of each frame are extracted.

[0097] A Dynamic Time Warping (DTW) algorithm is used to resample the timelines of facial and lip-sync animation sequences. First, frame rate normalization is performed on both animation sequences using discretization sampling to generate uniform time sampling points. Next, based on a baseline starting frame, the timestamps in the animation sequences are dynamically aligned using the DTW algorithm to ensure a clear temporal correspondence between each frame. Finally, for each frame of the sequence, the facial expression vector and lip-sync vector are extracted to complete the alignment process.

[0098] For example, given a facial animation sequence and a lip-sync animation sequence that does not perfectly match its timing (e.g., the lip-sync sequence may be shorter due to different syllable change speeds), the system will normalize the two sequences to 30 FPS through resampling, and align the frame "smiling while pronouncing 'tian'" using DTW, while extracting the smiling expression motion vector and the lip-sync motion vector to ensure temporal consistency of the corresponding frames.

[0099] The facial expression motion vector and lip shape motion vector are fused and weighted to build a fusion driving vector sequence. The fusion driving vector sequence is then input into a preset motion combination module in a stylized 3D face model to generate complete animation motion data.

[0100] By employing weighted modeling, specifically, a multimodal fusion mechanism is used to merge facial expression vectors and lip-shape vectors into a single fused driving vector sequence. An attention-based fusion model is used to model the weights of facial expressions and lip-shape movements, optimizing their synergistic relationship over time. This attention mechanism assigns weights to facial expressions and lip-shape movements based on the temporal context of the current frame, ensuring consistency and smoothness of the animation. The fused driving vector sequence is then input into the motion fusion module of the stylized 3D face model, which uses previously generated facial muscle control points and lip shape geometry to define complete animation motion data.

[0101] For example, during the processing of the frame sequence "Today is a nice day," the attention mechanism assigns higher weight to the lip movements in the "Today" frame because opening the mouth to pronounce words is the dynamic focus of that frame; while for the "Yes" frame, the smiling facial expression and the slight curve of the corners of the mouth may both receive higher weight. Ultimately, the motion combination module generates continuous and smooth complete animation motion data, representing a seamless transition from a pleasant smile to clear pronunciation.

[0102] The audio information is aligned and sliced ​​to extract the audio frame segments of the corresponding frames. The animation motion data and audio frame segments are then rendered in frame-level synchronization to generate the corresponding digital human video.

[0103] Through audio slicing processing and frame-level synchronous rendering. Specifically, first, according to a unified frame rate, the audio information is segmented into time segments, and the time stamps of the audio sequence and the animation action data are aligned. Then, based on time-domain signal processing methods (such as short-time Fourier transform, STFT), the audio segment features of each frame are extracted to ensure that the audio of each frame is synchronized with the corresponding animation action. Next, the animation action data and the corresponding audio frame segments are input into the rendering engine for frame-level synchronous rendering. In the rendering engine, based on the real-time rendering algorithm accelerated by GPU, combined with the temporal relationship between consecutive frame animations and audio adjustments, the final video frame sequence is generated frame by frame.

[0104] For example, if each phoneme in the audio "Today is a nice day" occupies two frames, and the animation action data includes more complex expressions and lip movements (such as smiling, opening the mouth, and closing the mouth), the system will render the audio feature "今" of the first frame together with the "open mouth + expression change", and the audio "天" of the second frame will match the expression change of "close mouth + smile transition", completing the time synchronization of audio and video frame by frame, and finally generating a continuous and smooth digital human video.

[0105] Among them, the steps of performing alignment slicing processing on the audio information, extracting the audio frame segments corresponding to the frames, performing frame-level synchronous rendering on the animation action data and the audio frame segments, and generating the corresponding digital human video specifically include:

[0106] The audio information is segmented at the same time interval through a preset frame rate to obtain a set of audio frame segments, and audio feature extraction is performed on each audio frame segment in the set of audio frame segments to obtain time-domain audio features and frequency-domain audio features.

[0107] Through audio slicing and feature extraction. Specifically, based on a set frame rate (such as 30FPS), the audio signal is segmented on the time axis, and the audio information is discretized at the same time interval to obtain a set of audio frame segments. Each audio frame segment corresponds to a frame time interval. For each frame segment, the short-time Fourier transform (STFT, Short-Time Fourier Transform) is used to extract time-domain audio features and frequency-domain audio features. The time-domain audio features include the change of audio amplitude over time, and the frequency-domain audio features include the spectral distribution of the audio signal. The time-domain features are used to characterize the dynamic changes of the audio, and the frequency-domain features are used to describe the frequency components of the audio.

[0108] For example, given a speech signal "Today is a nice day", the system will slice the audio into small segments according to the preset frame rate of 30FPS, and each frame corresponds to 33ms of audio data. For the first frame slice "今", the STFT calculates its time-domain amplitude change curve and frequency-domain spectral distribution. The time-domain features reflect the intensity of the gradually rising pitch in this syllable, and the frequency-domain features show that the main energy is concentrated in the middle frequency band.

[0109] Construct an audio feature vector based on the time-domain audio features and frequency-domain audio features, and perform temporal alignment between the audio feature vector and the action feature vector of the corresponding frame in the animation action data to ensure the consistency of the audio and action data on the time axis, and obtain the aligned audio feature vector.

[0110] Through constructing the audio feature vector and temporal alignment, specifically, use the multi-dimensional feature splicing method to form an audio feature vector from the time-domain audio features and frequency-domain audio features of each frame slice. The audio feature vector contains the amplitude change information in the time dimension and the energy distribution pattern information in the frequency domain. Then, perform time-axis alignment processing on the audio feature vector and the action feature vector of the corresponding frame in the animation action data through the Dynamic Time Warping (DTW) algorithm. DTW can automatically optimize the matching relationship between the audio and action data in the time dimension to ensure that the changes in the audio features are completely synchronized with the action features (such as mouth shapes, expressions) in time.

[0111] For example, for the audio segment of the syllable "tian", the spectral features show that the pronunciation energy of the phoneme is concentrated in the middle and high frequency bands, which is associated with a specific mouth shape (the mouth is slightly open and the tip of the tongue is raised). The DTW algorithm will align the audio feature vector and the animation action feature according to the time stamp, so that the starting sound wave of the "tian" audio exactly matches the mouth-opening action in the animation, and generate the aligned audio feature vector.

[0112] Perform frame-level weight adjustment on the aligned audio feature vector and action feature vector, combine the intensity of the audio and the rhythm of the action to obtain the synchronization rendering parameters, and input the synchronization rendering parameters into the preset rendering module for audio-visual synchronization rendering to generate the corresponding digital human video.

[0113] Through weight adjustment and synchronization rendering, specifically, in the frame-level weight adjustment stage, perform weight allocation on the audio feature vector and the action feature vector through the multi-modal fusion attention mechanism. The attention mechanism assigns appropriate weights to each frame by analyzing the intensity changes of the audio (such as volume size, pitch elevation) and the rhythm and dynamic amplitude of the action, ensuring that the synchronization rendering result of the audio and action conforms to the natural visual and auditory laws. Subsequently, input the synchronization rendering parameters into the preset rendering module. Based on GPU acceleration and real-time shadow calculation, the rendering module fuses the audio features and the animation action feature sequences frame by frame to generate a continuous digital human video frame sequence.

[0114] For example, in the pronunciation segment of "weather," the loud syllable "sky" receives a higher audio weight, and its movement features, such as the open mouth and facial expression, become more prominent; while the lighter syllable "breath" receives a lower weight, with its mouth slightly closed and its facial expression reduced. These weights are blended in real time frame by frame by the rendering module, and the final digital human video naturally displays the intensity and dynamics of the character "sky," while seamlessly transitioning to the gentle movements of "breath."

[0115] In this embodiment, by combining text-driven and audio-driven expressive features and employing multi-layered semantic coding and speech feature extraction techniques, natural and fluent digital human video generation is achieved. First, the BERT model is used to perform semantic parsing of the text, and a multi-layered semantic coding network is used to extract intra-sentence semantic features and emotional information. Simultaneously, a context modeling network is applied to extract inter-sentence dependencies and emotional flow features, resulting in emotionally enhanced semantic vectors and emotional feature representations. Next, LSTM and MFCC-based audio feature extraction techniques are used to extract prosodic features and pronunciation dynamics features, which are then fused using an attention mechanism to ensure temporal consistency between audio features and animation actions. Subsequently, deep fusion technology is used to co-modulate text and audio features to generate facial expression and lip-sync parameters. A stylized 3D face model is generated using style control parameters, and facial expression and lip-sync parameters are used to drive facial and lip-sync animation sequences. Finally, time alignment and weight adjustment are used to optimize audio-video synchronization, generating a high-quality digital human video. Throughout the process, technical means include Dynamic Time Warping (DTW) for audio and animation alignment, Multimodal Generative Adversarial Network (MM-GAN) for coordinated modulation of facial expressions and lip movements, and a GPU-accelerated real-time rendering module, ensuring the naturalness and consistency of the generated video. Through these comprehensive methods, this embodiment effectively solves problems such as inconsistencies between facial expressions and audio, and the inability to meet stylistic requirements in traditional digital human generation technologies, providing a higher-quality, more personalized method for generating digital human videos.

[0116] Example 3:

[0117] like Figure 2 As shown, this application also provides a digital human video generation device 10, including an acquisition module 11, a fusion module 12, a modulation module 13, a driving module 14, and a generation module 15.

[0118] The acquisition module 11 is mainly used to acquire text information, audio information and style control parameters, perform semantic parsing on the text information, and obtain text-driven expressive features.

[0119] The fusion module 12 is mainly used to extract speech features from audio information, obtain prosodic feature vectors and pronunciation dynamic vectors, and fuse the prosodic feature vectors and pronunciation dynamic vectors to obtain audio-driven expression features.

[0120] The modulation module 13 is mainly used for the collaborative modulation of text-driven expression features and audio-driven expression features to obtain facial expression parameter sequences and lip shape parameter sequences.

[0121] The driving module 14 is mainly used to generate a stylized 3D face model of the digital human based on style control parameters. It uses facial expression parameter sequences to drive the stylized 3D face model to obtain facial animation sequences, and uses lip shape parameter sequences to drive the stylized 3D face model to obtain lip shape animation sequences.

[0122] The generation module 15 is mainly used to perform time alignment processing on the facial animation sequence and the lip-sync animation sequence to obtain complete animation motion data, and to synchronously render the animation motion data and audio information to generate the corresponding digital human video.

[0123] In this embodiment, high-quality digital human videos are generated through the collaborative work of the acquisition module 11, fusion module 12, modulation module 13, driving module 14, and generation module 15. The acquisition module 11 performs semantic parsing of text information, extracts text-driven expression features using an advanced BERT model, and captures inter-sentence semantic dependencies and emotional flow features using context modeling techniques. The fusion module 12 extracts speech features based on audio information, extracting prosodic feature vectors and pronunciation dynamic vectors using short-time Fourier transform (STFT) and Mel-frequency cepstral coefficients (MFCC), and fuses them temporally using an attention mechanism to generate accurate audio-driven expression features. The modulation module 13 deeply integrates text-driven and audio-driven expression features through collaborative modulation, generating facial expression parameter sequences and lip shape parameter sequences containing emotional and speech coordination. The driving module 14 generates a stylized 3D face model from a digital human parameter template library based on style control parameters, and drives the muscle control module and lip deformation module in the model through expression and lip shape parameters to generate facial and lip animations. Finally, the generation module 15 aligns the facial animation sequence and lip-sync animation sequence along the timeline using Dynamic Time Warping (DTW), and performs synchronized rendering in conjunction with audio information to generate a high-quality digital human video containing natural speech and facial expressions. This embodiment, through modular design, highly integrates text parsing, audio fusion, facial expression modulation, stylization-driven rendering, and generation processes, significantly improving the expressiveness and personalization support of the digital human, and possessing efficient processing capabilities and stable generation quality.

[0124] It should be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device and each module described above can be referred to the corresponding process in the aforementioned Embodiment 1, and will not be repeated here.

[0125] Example 4:

[0126] like Figure 3 As shown, this application also provides a computer device 20, including a memory 21 and a processor 22. The memory 21 stores a computer program that can run on the processor 22. When the processor 22 executes the computer program, it implements the digital human video generation method of Embodiment 1.

[0127] In this embodiment, the complete functionality of the digital human video generation method is realized through the memory 21 and processor 22 in the computer device 20. The computer program stored in the memory 21 includes modules for acquisition, processing, fusion, driving, and rendering, and the processor 22 executes the logical operations and data processing of these modules. The processor 22, by running the computer program, first performs semantic parsing on the text information, extracting text-driven expressive features using deep learning models such as BERT and context modeling networks. Next, the processor 22 performs scientific frame segmentation on the audio information, extracting time-domain and frequency-domain features using Short-Time Fourier Transform (STFT), and generating prosodic feature vectors and pronunciation dynamic vectors using the Mel Frequency Cepstral Coefficients (MFCC) algorithm. Finally, it uses an attention mechanism to complete audio feature fusion, generating audio-driven expressive features. Subsequently, the processor 22 coordinates the modulation of text-driven and audio-driven expressive features to generate facial expression parameter sequences and lip shape parameter sequences, and generates a stylized 3D face model based on style control parameters. Using the facial expression parameters and lip shape parameters, the processor 22 achieves animation driving by driving the muscle module and lip deformation module in the model. Finally, the processor 22 uses the Dynamic Time Warping (DTW) algorithm to time-align facial and lip-sync animations and combines this with audio information to complete synchronized rendering, generating a high-quality digital human video. This embodiment leverages the high-performance processing capabilities of the computer device 20 to integrate multiple modules into a hardware and software system, efficiently achieving multimodal fusion of text, speech, and animation to generate natural and realistic digital human videos. This provides stable and efficient technical support for applications in film, virtual interaction, and other fields.

[0128] The structures, proportions, sizes, etc., shown in the accompanying drawings of this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed in the specification, and are not intended to limit the conditions under which the present invention can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size, without affecting the effects and objectives that the present invention can produce, should still fall within the scope of the technical content disclosed in the present invention.

[0129] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for generating digital human videos, characterized in that, include: The process involves acquiring text information, audio information, and style control parameters; segmenting the text information into sentences to obtain a set of sentence elements and inter-sentence dependencies; extracting intra-sentence semantic features based on the sentence element set; and extracting global semantic context vectors and inter-sentence sentiment flow features based on inter-sentence dependencies. The intra-sentence semantic features are then fused with the global semantic context vector to obtain a multi-level semantic feature vector. The inter-sentence sentiment flow features are then cross-attention mapped with the multi-level semantic feature vector to obtain an emotion-enhanced semantic vector and an emotion feature representation. The emotion feature representation is used to style-adjust the emotion-enhanced semantic vector to obtain a semantic feature vector and an emotion feature vector consistent with the text intent. Finally, the semantic feature vector and the emotion feature vector are fused to obtain text-driven expression features. Speech features are extracted from the audio information to obtain prosodic feature vectors and pronunciation dynamic vectors. The prosodic feature vectors and pronunciation dynamic vectors are then fused to obtain audio-driven expression features. The text-driven expression features are input into the facial expression channel encoder to extract multi-level facial expression driving vectors. The audio-driven expression features are input into the lip-sync channel encoder to extract multi-band pronunciation mapping vectors. The multi-level facial expression driving vectors and the multi-band pronunciation mapping vectors are time-aligned to obtain expression feature pairs with the same time axis. A fused expression state map is constructed based on the expression feature pairs. The fused expression state map is feature-decoded to obtain facial expression parameters and lip-sync parameters. The facial expression parameters and lip-sync parameters are co-modulated to eliminate expression conflicts and rhythm misalignments, resulting in a facial expression parameter sequence and a lip-sync parameter sequence. A stylized 3D face model of the digital human is generated based on the style control parameters. The facial expression parameter sequence is used to drive the expression of the stylized 3D face model to obtain a facial animation sequence. The mouth shape parameter sequence is used to drive the mouth shape of the stylized 3D face model to obtain a mouth shape animation sequence. The facial animation sequence and the lip-sync animation sequence are resampled along the timeline, and a unified frame rate and a baseline starting frame are established based on the sampling results. The unified frame rate and the baseline starting frame are used to perform time alignment processing on the facial animation sequence and the lip-sync animation sequence, and the facial expression motion vector and the lip-sync motion vector of each frame are extracted. The facial expression motion vector and the lip-sync motion vector are then fused with weights to build a fusion driving vector sequence. The fusion driving vector sequence is input into a preset motion combination module in the stylized 3D face model to generate complete animation motion data. The audio information is aligned and sliced ​​to extract audio frame segments of the corresponding frames. The animation motion data and the audio frame segments are then rendered at the frame level to generate the corresponding digital human video.

2. The digital human video generation method according to claim 1, characterized in that, The step of extracting speech features from the audio information to obtain prosodic feature vectors and articulation dynamic vectors, and fusing the prosodic feature vectors and articulation dynamic vectors to obtain audio-driven expressive features includes: The audio information is divided into frames, and the temporal envelope features and frequency spectrum features of each frame are extracted. Short-term prosodic patterns are extracted based on the temporal envelope features. The short-term prosodic patterns are fused with the rhythm information of neighboring frames to obtain a temporally continuous prosodic feature vector. Frequency domain articulation features are extracted using the frequency domain spectrogram features; principal component decoupling processing is performed on the frequency domain articulation features to extract articulation state sequences and articulation boundary change information; the articulation state sequences and articulation boundary change information are encoded and mapped to obtain an articulation dynamic vector; The temporally continuous prosodic feature vector and the pronunciation dynamic vector are fused along the temporal dimension to obtain audio-driven expressive features.

3. The digital human video generation method according to claim 1, characterized in that, The steps of generating a stylized 3D face model of a digital human based on the style control parameters, using the facial expression parameter sequence to drive the expression of the stylized 3D face model to obtain a facial animation sequence, and using the lip shape parameter sequence to drive the lip shape of the stylized 3D face model to obtain a lip shape animation sequence include: Based on the style control parameters, the corresponding basic 3D face model and style mapping weights are loaded from the preset digital human parameter template library. The style mapping weights are then used to perform style deformation and material mapping on the basic 3D face model to obtain a stylized 3D face model. The facial expression parameter sequence is input into the preset muscle control module in the stylized 3D face model to drive the change of facial expression control points in the stylized 3D face model and obtain the expression driving result. The mouth shape parameter sequence is input into the preset lip deformation module in the stylized 3D face model to drive the geometric deformation of the lip and teeth region in the stylized 3D face model and obtain the mouth shape driving result. The expression-driven results are subjected to frame-by-frame pose fusion processing to generate a facial animation sequence, and the lip-shape-driven results are subjected to frame-by-frame pose fusion processing to generate a lip-shape animation sequence.

4. The digital human video generation method according to claim 1, characterized in that, The steps of aligning and slicing the audio information, extracting audio frame segments of corresponding frames, and performing frame-level synchronized rendering of the animation motion data and the audio frame segments to generate the corresponding digital human video include: The audio information is divided into segments with the same time interval by a preset frame rate to obtain a set of audio frame segments. Audio features are extracted from each audio frame segment in the set of audio frame segments to obtain time-domain audio features and frequency-domain audio features. An audio feature vector is constructed based on the time-domain audio features and the frequency-domain audio features. The audio feature vector is then time-aligned with the motion feature vector of the corresponding frame in the animation motion data to obtain the aligned audio feature vector. The aligned audio feature vector and the motion feature vector are weighted at the frame level to obtain synchronous rendering parameters. The synchronous rendering parameters are then input into a preset rendering module for audio and video synchronous rendering to generate the corresponding digital human video.

5. A digital human video generation device, characterized in that, include: The acquisition module is used to acquire text information, audio information, and style control parameters. It performs sentence segmentation on the text information to obtain a set of sentence elements and inter-sentence dependencies. Based on the sentence element set, it extracts intra-sentence semantic features and extracts a global semantic context vector and inter-sentence sentiment flow features according to the inter-sentence dependencies. It then fuses the intra-sentence semantic features with the global semantic context vector to obtain a multi-level semantic feature vector. Finally, it performs cross-attention mapping between the inter-sentence sentiment flow features and the multi-level semantic feature vector to obtain an emotion-enhanced semantic vector and an emotion feature representation. The emotion feature representation is used to style-adjust the emotion-enhanced semantic vector to obtain a semantic feature vector and an emotion feature vector consistent with the text intent. Finally, it fuses the semantic feature vector and the emotion feature vector to obtain text-driven expression features. The fusion module is used to extract speech features from the audio information to obtain prosodic feature vectors and pronunciation dynamic vectors, and to fuse the prosodic feature vectors and the pronunciation dynamic vectors to obtain audio-driven expression features; The modulation module is used to input the text-driven expression features into the facial expression channel encoder to extract multi-level facial expression driving vectors, and input the audio-driven expression features into the lip-sync channel encoder to extract multi-band pronunciation mapping vectors; perform time-series alignment processing on the multi-level facial expression driving vectors and the multi-band pronunciation mapping vectors to obtain expression feature pairs with the same time axis; construct a fused expression state map based on the expression feature pairs; perform feature decoding on the fused expression state map to obtain facial expression parameters and lip-sync parameters; and perform co-modulation on the facial expression parameters and the lip-sync parameters to eliminate expression conflicts and rhythm misalignments, thereby obtaining a facial expression parameter sequence and a lip-sync parameter sequence. The driving module is used to generate a stylized 3D face model of the digital human based on the style control parameters, drive the stylized 3D face model to perform facial expression using the facial expression parameter sequence to obtain a facial animation sequence, and drive the stylized 3D face model to perform lip movement using the lip movement parameter sequence to obtain a lip movement animation sequence. The generation module is used to resample the facial animation sequence and the lip-sync animation sequence along the time axis, establish a unified frame rate and a reference starting frame based on the sampling results, perform time alignment processing on the facial animation sequence and the lip-sync animation sequence using the unified frame rate and the reference starting frame, and extract the facial expression motion vector and lip-sync motion vector for each frame; perform fusion weight modeling on the facial expression motion vector and the lip-sync motion vector to construct a fusion driving vector sequence, input the fusion driving vector sequence into a preset motion combination module in the stylized 3D face model to generate complete animation motion data; perform alignment and slicing processing on the audio information, extract audio frame segments of the corresponding frames, and perform frame-level synchronous rendering of the animation motion data and the audio frame segments to generate the corresponding digital human video.

6. A computer device, characterized in that, The device includes a memory and a processor, the memory storing a computer program that can run on the processor, and the processor executing the computer program to implement the digital human video generation method according to any one of claims 1 to 4.

Images