A digital human expression driving method, device and equipment

Abstract

The present invention discloses a method, apparatus, and device for driving digital human expressions, involving data interaction technology, to address the problem of mechanically driving existing digital human expressions, which lack appeal. The method includes: receiving a voice signal input by a user and obtaining voice features corresponding to the voice signal; processing the voice features based on a preset emotion encoder to obtain emotion prediction data corresponding to the voice features; fusing the emotion prediction data with the basic features corresponding to the feature map of a preset hybrid encoder through a cross-attention mechanism to obtain hybrid features; inputting the hybrid features into a preset decoder for decoding to obtain facial expression coefficients; and driving the digital human's expressions based on the facial expression coefficients.

Description

Technical Field

[0001] The present application relates to the field of digital human technology, and in particular to a method, device and equipment for driving digital human expressions. Background Art

[0002] As the popularity of digital virtual humans continues to grow, voice-driven 3D facial animation technology has attracted widespread attention and holds broad application prospects in film, gaming, and VR / AR. Voice-driven 3D digital facial animation aims to synthesize synchronized facial movements from a speech input sequence. As the demand for human-computer interaction continues to increase, certain application scenarios are placing higher demands on the authenticity and naturalness of human-computer interaction. Simple voice output can no longer meet users' growing demand for immersive interaction. Therefore, driving digital human expressions has become a key area of ​​digital human optimization.

[0003] The current traditional method of generating expressions through manually designed phoneme-to-lip-shape mapping rules lacks emotional expressiveness and is difficult to adapt to different speech styles, resulting in stiff and unnatural expressions in digital humans. While end-to-end neural network methods such as CNNs or LSTMs, which directly model the speech-to-expression mapping, improve fluency, they lack emotional control and cannot explicitly separate emotion from lip-shape features, resulting in a lack of expressiveness. Furthermore, existing multi-stage separation methods, which first train the emotion classifier and expression generator separately and then fuse them, are not end-to-end training methods, making it easy for digital humans' emotions and expressions to become inconsistent and computationally expensive. Summary of the Invention

[0004] To address the problems of existing digital human expression-driving mechanisms lacking appeal and having high computational overhead, one or more embodiments of the present application provide a digital human expression-driving method, device, and apparatus.

[0005] One or more embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, one or more embodiments of the present application provide a method for driving digital human expressions, the method comprising:

[0007] Receive a voice signal input by a user and obtain voice features corresponding to the voice signal;

[0008] Processing the speech features through a preset emotion encoder to obtain emotion prediction data corresponding to the speech features;

[0009] The emotion prediction data is fused with the basic features corresponding to the feature map of the preset hybrid encoder through a cross attention mechanism to obtain a hybrid feature;

[0010] Inputting the mixed features into a preset decoder for decoding to obtain a facial expression coefficient;

[0011] Based on the facial expression coefficient, the digital human expression is driven.

[0012] Optionally, in one or more embodiments of the present application, processing the speech features by a preset emotion encoder to obtain emotion prediction data corresponding to the speech features specifically includes:

[0013] Adding corresponding position codes to the speech features based on the time sequence of the speech features;

[0014] Based on the linear transformation of the speech feature, obtaining a self-attention calculation matrix corresponding to the speech feature;

[0015] Determining the self-attention architecture of the preset emotion encoder through the self-attention calculation matrix and the preset activation function; wherein the preset emotion encoder includes: an emotion encoder and an expression encoder;

[0016] The speech features corresponding to the position codes are processed based on the self-attention architecture to obtain emotion prediction data corresponding to the speech features; wherein the emotion prediction data includes: emotion vectors and expression features.

[0017] Optionally, in one or more embodiments of the present application, before processing the speech features to obtain emotion prediction data corresponding to the speech features, the method further includes:

[0018] Determine the application scenario corresponding to the digital human to obtain the emotional needs corresponding to the application scenario; wherein the emotional needs include: core emotional needs and conventional emotional needs;

[0019] Identifying the emotion type and emotion expression target corresponding to the core emotion need, and determining a corresponding encoder calculation model based on the emotion type and the emotion expression target;

[0020] Speech signal samples matching the core emotional requirements and the conventional emotional requirements are collected, so as to train the encoder calculation model based on the speech signal samples and obtain a preset emotion encoder.

[0021] Optionally, in one or more embodiments of the present application, the emotion prediction data is fused with the basic features corresponding to the feature map of the preset hybrid encoder through a cross-attention mechanism to obtain a hybrid feature, specifically including:

[0022] Extracting a current feature map in the preset hybrid encoder, and copying the emotion vector to the same spatiotemporal dimension as the feature map to obtain a copied emotion vector;

[0023] Converting the copied emotion vector into a query vector of the cross-attention mechanism, and converting the expression feature into a key vector and a value vector of the cross-attention mechanism;

[0024] Based on the query vector, the key vector and the value vector, the attention output is fused to the basic features corresponding to the feature map to obtain a weighted fused hybrid feature.

[0025] Optionally, in one or more embodiments of the present application, the mixed features are input into a preset decoder for decoding to obtain facial expression coefficients, specifically including:

[0026] Inputting the mixed features into a preset decoder, so that the preset decoder captures the inter-frame dependency of the facial animation based on a self-attention mechanism, and obtains decoded output data;

[0027] The decoded output data is mapped to the facial expression coefficient dimension of the digital human based on the fully connected layer of the preset decoder to obtain the facial expression coefficient corresponding to each frame of the facial expression animation.

[0028] Optionally, in one or more embodiments of the present application, receiving a voice signal input by a user and obtaining a voice feature corresponding to the voice signal specifically includes:

[0029] Receive a voice signal input by a user, perform denoising on the voice signal, and obtain a denoised voice signal;

[0030] Based on a preset peak value and a preset audio energy, normalizing the denoised speech signal to obtain a processed speech signal; wherein the preset peak value and the preset audio energy are determined based on the recorded speech of the user;

[0031] The processed speech signal is extracted based on the pre-trained speech model to obtain speech features of preset dimensions.

[0032] Optionally, in one or more embodiments of the present application, before integrating the emotion prediction data into the feature map of the preset hybrid encoder through the cross attention mechanism to obtain the hybrid feature, the method further includes:

[0033] determining a speech text corresponding to the processed speech signal and an additional text corresponding to the processed speech signal;

[0034] Determining the phonemes corresponding to the voice text and the personalized content corresponding to the additional text;

[0035] Aligning the phoneme with the personalized content based on timestamp information and embedding them into the voice feature;

[0036] The embedded speech features are extracted by the preset hybrid encoder to obtain basic features integrated with lip shape features, so as to generate a feature map of the preset hybrid encoder according to the basic features.

[0037] Optionally, in one or more embodiments of the present application, the emotion classification loss determined by the cross entropy loss function is optimized during the training of the preset emotion encoder, and the expression regression loss determined by the L1 smoothing loss function is optimized during the training of the preset decoder.

[0038] In a second aspect, one or more embodiments of the present application provide a digital human expression driving device, the device comprising:

[0039] An acquisition unit, configured to receive a voice signal input by a user and acquire a voice feature corresponding to the voice signal;

[0040] a processing unit, configured to process the speech feature using a preset emotion encoder to obtain emotion prediction data corresponding to the speech feature;

[0041] a fusion unit, configured to fuse the emotion prediction data with basic features corresponding to feature maps of a preset hybrid encoder through a cross-attention mechanism to obtain hybrid features;

[0042] A decoding unit, configured to input the mixed features into a preset decoder for decoding to obtain a facial expression coefficient;

[0043] A driving unit is used to drive the expression of the digital human based on the facial expression coefficient.

[0044] In a third aspect, one or more embodiments of the present application provide a digital human expression driving device, the device comprising:

[0045] at least one processor; and,

[0046] a memory communicatively connected to the at least one processor; wherein,

[0047] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: execute any of the above-mentioned digital human expression driving methods.

[0048] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects:

[0049] By pre-setting the emotion encoder to extract emotion prediction data from the speech signal, the emotion prediction vector is then incorporated into the feature map as an additional control factor in the hybrid encoder using a cross-attention mechanism. This increases the interpretability of emotions and ensures that the final driven expression is highly consistent with the emotions contained in the speech content and the transition is natural, enhancing the realism of the digital human's expression. In addition, the direct generation of drivable expression coefficients from the speech signal enables end-to-end processing, avoiding the tedious steps of manually designing rules or manually adjusting expression parameters in traditional processes, significantly improving the efficiency of content generation or real-time interaction. In addition, obtaining emotion prediction data from the speech signal and then driving expression adjustment can ensure that the digital human's expression is synchronized with the emotion in the audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments described in the present application. For those skilled in the art, other drawings can be obtained based on these drawings without inventive work. In the drawings:

[0051] Figure 1 A schematic diagram of the architecture of a digital human expression drive provided in an embodiment of the present application;

[0052] Figure 2 A flowchart of a method for driving digital human expressions provided in an embodiment of the present application;

[0053] Figure 3 A technical flow chart of expression-driven digital human in an application scenario provided by an embodiment of the present application;

[0054] Figure 4 A schematic diagram of a digital human expression-driven scenario provided in an embodiment of the present application;

[0055] Figure 5 A schematic diagram of a process for obtaining mixed features provided in an embodiment of this specification;

[0056] Figure 6 A schematic structural diagram of a digital human expression driving device provided in an embodiment of the present application;

[0057] Figure 7 A schematic diagram of the structure of a digital human expression driving device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0058] The embodiments of the present application provide a method, apparatus, and device for driving the expression of a digital human.

[0059] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0060] Unless otherwise defined, technical or scientific terms used in this application shall have the ordinary meanings as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "the," "these," and similar expressions in this application do not denote limitations on quantity and may be singular or plural. The terms "comprise," "include," "have," and any variations thereof, as used in this application, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include unlisted steps or modules (units) or other steps or modules (units) inherent to the process, method, product, or device. The terms "connected," "connected," "coupled," and similar expressions used in this application are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used in this application, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone; A and B exist simultaneously; or B exists alone. Generally, the character " / " indicates that the objects in the preceding and following relationship are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0061] The embodiment of the present application provides a method for driving the expression of a digital human, which can be applied to Figure 1 In the application environment shown. Figure 1 As shown, the application environment may include: client terminal 101, communication network 102, server 103, and database 104. Communication network 102 can serve as a data transmission channel, providing a communication link between client terminal 101 and server 103. This allows client terminal 101 to record voice signals and transmit them to server 103 via communication network 102. Furthermore, facial expression coefficients processed by server 103 can be fed back to client terminal 101 to drive the digital human's expressions. Server 103 can provide various services. To improve processing efficiency, server 103 can be connected to multiple servers 103 to form a master-slave server cluster. This allows information collected by smart glasses 101 to be received via communication network 102 and sent to the corresponding server 103 in the server cluster for voice signal processing. After processing, server 103 feeds back the facial expression coefficients to the client terminal via communication network 102 to achieve expression-driven display of the digital human.

[0062] The client terminal 101 may be a computer, a mobile phone, or other terminal devices capable of displaying a digital human.

[0063] The database 104 is connected to each server 103 and can be used to store and manage the data of the preset emotion encoder, preset hybrid encoder and preset decoder. The database 104 can be integrated on the server, or placed on the cloud or other network servers.

[0064] like Figure 2 The embodiment of the present application shows a flowchart of a method for driving the expression of a digital human. Figure 2 The method shown can be applied to various scenarios requiring virtual digital human interaction, including but not limited to scenarios involving interaction between virtual anchors and real-time broadcasters, such as inputting live broadcast voice to drive the virtual anchor to generate emotional expressions; pre-visualization scenarios for film and television animation, such as directly generating character animation drafts with emotions such as anger and sadness from the voice actor's voice to accelerate the production process; and supporting psychotherapy scenarios, such as analyzing the patient's voice emotions through emotion vectors to drive the digital human to respond with empathetic expressions, enhancing the realism of the interaction. The method includes the following steps: S201-S205.

[0065] S201: Receive a voice signal input by a user, and obtain voice features corresponding to the voice signal.

[0066] Traditional digital human expressions may rely on preset animations, resulting in unnatural expressions. However, voice-driven expressions can respond in real time and improve the interactivity of digital humans. Figure 3 As shown, the present application is based on voice-driven digital human expressions, and in order to achieve voice-driven digital human feedback expressions and enhance the realism of interaction. In the present application, a voice signal input by the user is received, thereby obtaining the voice features corresponding to the voice signal. The voice signal received from the user can be a continuous sound waveform captured by a microphone or other audio input device when the user speaks. The voice signal is then subjected to feature extraction, such as extracting acoustic features including fundamental frequency, loudness, spectral features, etc., linguistic features including intonation, word boundaries, and prosodic features, etc., and the extracted features are sorted based on chronological order to obtain a set of voice features.

[0067] In some embodiments, the above step S201 receives a voice signal input by a user and obtains a voice feature corresponding to the voice signal, specifically including the following steps:

[0068] S301: Receive a voice signal input by a user, perform denoising on the voice signal, and obtain a denoised voice signal;

[0069] S302: Normalizing the denoised speech signal based on a preset peak value and a preset audio energy to obtain a processed speech signal; wherein the preset peak value and the preset audio energy are determined based on the recorded speech of the user;

[0070] S303: Extract the processed speech signal based on the pre-trained speech model to obtain speech features of preset dimensions.

[0071] The original voice signal input by the user through a microphone or other device will inevitably contain environmental noise and electronic noise such as background human voices, keyboard sounds, and fan sounds. Therefore, in order to identify and attenuate or eliminate these noise components that are irrelevant to the user's voice content as much as possible, in order to improve the accuracy of subsequent emotion and feeling recognition. After receiving the voice signal input by the user, the present application will first perform denoising on the voice signal to effectively filter out environmental noise and non-voice signal interference, obtain the denoised voice signal, and make it closer to the real voice content. Then, since the volume of the recording of the same user at different times and distances may be different, in order to ensure that the feature extraction process is not affected by the absolute size of the volume, the standard values ​​of the preset peak value and the preset audio energy will be set according to the specific recorded voice of the user. For example: based on the maximum amplitude value of the voice signal representing the normal speaking level recorded by the user in a quiet environment, the preset peak value corresponding to the user is determined; based on the average energy or RMS energy of the voice signal representing the normal speaking level recorded by the user in a quiet environment, the preset audio energy is used. Then, using the preset peak value and preset audio energy as a benchmark, the denoised speech signal is scaled to eliminate volume differences between users or between different input sessions of the same user. In addition, user-adaptive normalization can better reflect the user's normal speaking level than global fixed normalization, avoiding over-amplification or compression problems.

[0072] After the denoised speech signal is normalized and the processed speech signal is obtained, the processed speech signal is extracted according to the pre-trained speech model to obtain speech features of preset dimensions. Among them, the pre-trained speech model refers to a deep neural network model that has been pre-trained on a large-scale speech data set, which can automatically learn the rich information contained in the speech. After the pre-trained speech model extracts the processed speech signal, it outputs a vector representation of a fixed length. For example, in a certain application scenario of this application, 1024-dimensional audio features will be extracted, corresponding to 50 audio extractions per second, that is, the fps of the speech features is 50.

[0073] In some embodiments, denoising is performed on a speech signal to obtain a denoised speech signal, which includes the following steps: first, for speech signals in complex environments, filtering is performed based solely on estimating the noise of the speech signal itself, making it difficult to perceive the physical source of the noise and achieve full-dimensional filtering of the noise. Therefore, in an embodiment of the present application, a speech signal segment of the starting segment of the speech signal uploaded by the user is obtained, and the environmental image and motion data corresponding to the speech signal segment are synchronously collected. Then, the speech signal segment is converted into the frequency domain through Fourier transform, and the average energy of each frequency band is calculated to obtain the initial noise spectrum corresponding to the speech signal, and the noise visual characteristics of each noise source are determined based on the environmental image, and the coupling relationship between the vibration spectrum of each noise source and the noise is determined based on the operating data.

[0074] The initial noise spectrum, visual noise features, and the coupling relationship between the vibration spectrum and noise of each noise source are cross-attentively fused to obtain a multimodal noise vector. Based on the vibration spectrum coupling relationship in the multimodal noise vector, the initial noise spectrum is further adjusted to obtain an updated noise spectrum. Simultaneously, the short-term power spectrum corresponding to the speech signal is obtained, and non-speech frames are detected based on the speech presence probability in the short-term power spectrum. Based on these non-speech frames, the updated noise spectrum is then updated a second time to obtain a second updated noise spectrum. After the initial noise spectrum is modified based on the vibration spectrum coupling relationship in the multimodal noise vector to achieve an initial update, non-speech frames are detected based on the speech presence probability. Further iterations are performed using real-time noise data. Compared to the method of recursively averaging the noise based solely on non-speech frames, this overcomes the current inability to predict noise changes using vibration data in advance and avoids noise smearing in burst noise scenarios. Then, based on the noise visual features within the multimodal noise vector, the noise source direction is determined to enhance the noise suppression weight in the corresponding frequency band to obtain spatial filter coefficients. This applies a higher noise suppression weight to the frequency band in that direction while preserving speech signals in other directions. Then, the speech signal is subjected to denoising filtering according to the spatial filter coefficient and the noise spectrum after the second update to obtain a filtered speech signal. Then, the wavelet threshold denoising is performed on the wavelet speech signal to obtain a denoised speech signal.

[0075] S202: Processing the speech feature through a preset emotion encoder to obtain emotion prediction data corresponding to the speech feature.

[0076] The current artificially designed phoneme-to-mouth mapping rules for generating expressions lack emotional expression and are difficult to adapt to different voice styles, resulting in stiff and unnatural expressions. Figure 4As shown, speech features are processed using a pre-set emotion encoder to obtain emotion prediction data corresponding to the speech features. Specifically, after the pre-trained large model extracts 1024-dimensional speech features from the speech signal in a specific scenario, the pre-set emotion encoder, based on a multi-layer self-attention mechanism, processes the speech features to obtain emotion prediction data. During this process, the emotion vector is constrained based on the sentiment classification loss function to ensure a strong correlation between the emotion vector and human-understandable semantic labels. The pre-set emotion encoder with a multi-layer self-attention mechanism can recognize cross-frame dependencies. For example, in the speech clip "This is so... disappointing!", the emotion vector in the emotion prediction data dynamically evolves from disappointment to anger, solving the long-term forgetting problem of traditional RNNs and enabling the capture of continuous emotions.

[0077] In some embodiments, before processing the speech features by presetting an emotion encoder to obtain emotion prediction data corresponding to the speech features, the following specific steps are further performed:

[0078] S401: Determine the application scenario corresponding to the digital human to obtain the emotional needs corresponding to the application scenario; wherein the emotional needs include: core emotional needs and conventional emotional needs;

[0079] S402: Identifying the emotion type and emotion expression target corresponding to the core emotion requirement, and determining a corresponding encoder calculation model based on the emotion type and the emotion expression target;

[0080] S403: Collecting speech signal samples that match the core emotional requirements and the conventional emotional requirements, and training the encoder calculation model based on the speech signal samples to obtain a preset emotional encoder.

[0081] Different application scenarios have different requirements for the emotional expression of digital humans. For example, in the current customer service consultation scenario in the e-commerce field, it has core requirements for patience and professional emotions to avoid user anxiety, and also has conventional emotional requirements for friendly empathy; in the current game field, in the NPC scenario, it has core requirements for exaggerated, changeable and other emotions to enhance user immersion, and also has conventional emotional requirements for basic emotions that are consistent with the character settings. Therefore, in order to adapt to the differentiated requirements of digital humans' emotions in different application scenarios and achieve accurate matching of emotion prediction and application scenarios, this application will determine the application scenario corresponding to the digital human through scene tags or parsing keywords in the scene description text, thereby obtaining the core emotional requirements and conventional emotional requirements corresponding to the application scenario. It can be understood that the core emotional requirements are the key emotional dimensions that must be met in the scene, and the conventional emotional requirements are auxiliary emotional characteristics that serve as supplementary constraints to the core requirements.

[0082] After obtaining the core emotional needs and common emotional needs, the core emotional needs are mapped to specific emotion types, such as anger, joy, sadness, surprise, fear, calmness, comfort, encouragement, etc. The specific expression requirements of these core emotional needs are then used as the emotional expression targets. The corresponding encoder computation model is determined based on the emotion types corresponding to the core emotional needs and the emotional expression targets. For example, if the core need is to identify the intensity of several basic emotions, a CNN or Transformer-based model may be selected. If the need is to identify complex, subtle emotional changes or mixed emotions, a more complex model, such as a Transformer model with a self-attention mechanism, may be required. If the core need places high demands on the temporal dynamics of emotions, the model may require stronger sequence modeling capabilities, such as an LSTM. To generate the final preset emotion encoder, a large amount of real or simulated speech data expressing the core emotions in the target scenario is collected, as well as speech data expressing common emotions in the target scenario. The collected speech data, labeled with emotion categories, is used to train the model determined in S402, allowing the model to learn to predict the corresponding emotion prediction data from the input speech features, thereby obtaining a preset emotion encoder that meets the requirements.

[0083] In some embodiments, such as Figure 5 In the above step S202, the speech features are processed by the preset emotion encoder to obtain emotion prediction data corresponding to the speech features, which specifically includes:

[0084] S501: Adding corresponding position codes to the speech features based on the time sequence of the speech features;

[0085] S502: Obtaining a self-attention calculation matrix corresponding to the speech feature based on a linear transformation of the speech feature;

[0086] S503: Determine the self-attention architecture of the preset emotion encoder through the self-attention calculation matrix and the preset activation function; wherein the preset emotion encoder includes: an emotion encoder and an expression encoder;

[0087] S504: Processing the speech features corresponding to the position codes based on the self-attention architecture to obtain emotion prediction data corresponding to the speech features; wherein the emotion prediction data includes: emotion vectors and expression features.

[0088] The self-attention mechanism itself, by default, does not consider the positional order of elements when processing a sequence. To address the temporal order of speech signals, it first adds corresponding positional encodings to speech features based on their temporal order. This allows subsequent processing to understand the temporal context of the speech signal and improve the accuracy of emotion prediction data. Then, based on the linear transformation of the speech features, the self-attention calculation matrix corresponding to each speech feature is obtained. The self-attention calculation matrix and a preset activation function are then used to determine the self-attention architecture of the preset emotion encoder. It should be noted that the preset emotion encoder includes an emotion encoder and an expression encoder. The speech features corresponding to each positional encoding are then processed according to a unified self-attention architecture to generate emotion prediction data corresponding to the speech features. This emotion prediction data includes emotion vectors and expression features. During this process, the self-attention mechanism enables the model to autonomously calculate the global dependencies between speech feature frames, ensuring that emotion prediction considers both local details and the overall context. The dual-branch encoder design of the emotion encoder and expression encoder divides emotion prediction into emotion vector acquisition and expression feature acquisition. It can learn the mapping between emotion semantics and physical expression of expression respectively, which helps to obtain the physical parameters driving micro-expressions and lays a reliable data foundation for generating synchronous, natural and rich digital human expressions.

[0089] That is Figure 3 The emotion encoder shown uses a structure based on the self-attention calculation mechanism to process speech features and analyzes the emotional information in the speech through a multi-layer self-attention mechanism. The formula for self-attention calculation is as follows:

[0090]

[0091] The query matrix Q, key matrix K and value matrix V are obtained by linear transformation of input speech features. represents the transpose of the key matrix, and the query matrix and the transpose of the key matrix are matrix multiplied. The vector dimension represents the key matrix. The Softmax function calculates the attention scale for each position in the original sequence, ultimately outputting an attention-enhanced feature vector. The emotion encoder ultimately outputs a compact emotion vector. This vector can represent the emotional states of speech, such as joy, anger, sadness, and happiness, and maintains semantic associations with predefined emotion categories, providing emotional guidance for subsequent expression generation.

[0092] The expression encoder, also based on a self-attention computational architecture, focuses on extracting temporal information related to lip shape and facial movements from speech features. This self-attention mechanism models the long-term dependencies between speech frames and outputs the corresponding expression features at each moment, ensuring that the generated expressions are natural and fluent, synchronized with the speech rhythm. Both the emotion encoder and the expression encoder utilize the self-attention mechanism to model the long-term temporal dependencies of speech features, ensuring the coherence and naturalness of the generated expressions.

[0093] S203: The emotion prediction data is fused with the basic features corresponding to the feature map of the preset hybrid encoder through a cross-attention mechanism to obtain a hybrid feature.

[0094] The current multi-stage separation method is to first train the emotion classifier and expression generator separately and then fuse them. However, because it is not an end-to-end training, it is easy for emotions and expressions to be inconsistent, and the computational overhead is high. Therefore, in order to achieve the dynamic fusion of the emotion vector and the expression feature in the above step S504, the embodiment of the present application uses a cross-attention mechanism to integrate the emotion prediction data into the feature map of the preset hybrid encoder to obtain a hybrid feature. Figure 3 The hybrid encoder shown is responsible for fusing emotion and expression information, and uses a cross-attention mechanism to allow the emotion vector to dynamically adjust the weight of the expression feature. The formula is as follows:

[0095]

[0096] Unlike the self-attention mechanism, where the query, key, and value matrices are transformed from the same feature matrix, cross-attention uses emotional information as the query and expression features as the key. The query and key are generated from two separate matrices and then fused together. This hybrid encoder ensures that the generated expressions accurately match the speech content and reflect the emotional tone, enabling more vivid digital human-like driving.

[0097] In some embodiments, the emotion prediction data is fused with the basic features corresponding to the feature map of the preset hybrid encoder through a cross-attention mechanism. Before obtaining the hybrid features, the following process is also included:

[0098] S601: Determine the speech text corresponding to the processed speech signal and the additional text corresponding to the processed speech signal;

[0099] S602: Determine the phonemes corresponding to the voice text and the personalized content corresponding to the additional text;

[0100] S603: Align the phoneme with the personalized content based on the timestamp information and embed them into the voice feature;

[0101] S604: extracting features from the embedded speech features through the preset hybrid encoder to obtain basic features integrated with lip shape features, and generating a feature map of the preset hybrid encoder based on the basic features.

[0102] First, all text information related to the semantic content currently being driven by the digital human is collected. Speech text is the text obtained through speech recognition of the input speech signal. Additional text is text related to the current speech signal but not directly derived from speech recognition. It can include text corresponding to trigger options or user-uploaded text. The speech text obtained in step S601 is then converted to phonemes to obtain the phonemes corresponding to the speech text. It should be understood that phonemes are the smallest speech units in a language that can distinguish meaning and are the fundamental unit for driving accurate lip synchronization. Each phoneme corresponds to a specific oral, lip, and tongue movement pattern. While determining the phonemes corresponding to the speech text, it is also necessary to parse or associate personalized information from the additional text that influences the digital human's speaking style and performance, such as personalized speaking rhythm characteristics and lip shape-related features. To embed the phonemes and personalized content into speech features, discrete phoneme sequences and personalized features are accurately mapped to continuous, time-framed speech features on a timeline based on timestamp information. That is, each phoneme can be represented as a vector, and personalized content can also be represented as a vector or a sequence of vectors that changes over time. Based on the timestamp, the phoneme corresponding to the current frame is found. Based on the timestamp and context, the personalized feature vector to be applied to the current frame is determined. The found phoneme embedding vector and the personalized embedding vector are then concatenated or added to the original speech feature vector of the time frame to obtain the embedded speech features. By breaking down the speech text into phonemes and embedding speech features by timestamp, the hybrid encoder can learn the precise mapping of phonemes and lip shapes frame by frame, significantly improving the realism of the digital human's pronunciation. By embedding contextual text semantics, the intention not explicitly conveyed in the speech is supplemented, allowing facial expressions to transcend acoustic limitations, better align with the essence of the interaction, and providing rich input data for subsequent hybrid encoders.

[0103] Then, feature extraction is performed on the embedded speech features based on the preset hybrid encoder. That is, the embedded speech features with embedded phonemes and personalized content in each time frame outputted in the above step S603 are inputted into the hybrid encoder. The encoder processes the information through a multi-layer neural network, gradually abstracting and fusing the information, and obtaining the intermediate or final multi-dimensional data organization form of the fused lip shape features as the basic features, thereby generating a feature map of the preset hybrid encoder based on the basic features.

[0104] This process converts speech signals into text and phonemes, while also extracting personalized content from the attached text. This transforms the speech signal from a sonic form to a semantic level, enabling the model to understand the semantic structure of speech and additional personalized information. This provides a semantic foundation for subsequent feature fusion and avoids semantic bias caused by processing based solely on sonic features. Phonemes and personalized content are embedded in speech features via timestamps, ensuring strict synchronization between the speech signal's time domain information and the text's semantics and personalized elements.

[0105] In some embodiments, in step S203, the emotion prediction data is integrated into the feature map of the preset hybrid encoder through the cross attention mechanism to obtain the hybrid feature, specifically including:

[0106] S701: extracting a current feature map from the preset hybrid encoder, and copying the emotion vector to the same spatiotemporal dimension as the feature map to obtain a copied emotion vector;

[0107] S702: Convert the copied emotion vector into a query vector of the cross-attention mechanism, and convert the expression feature into a key vector and a value vector of the cross-attention mechanism;

[0108] S703: Based on the query vector, the key vector and the value vector, the attention output is fused to the basic features corresponding to the feature map to obtain a weighted fused hybrid feature.

[0109] Existing multi-stage separation methods, which first train the emotion classifier and expression generator separately and then fuse them, are not end-to-end training, which can easily lead to inconsistencies between the digital human's emotions and expressions, and they also have high computational overhead. To overcome this problem, in the embodiments of the present application, emotion information is used as the query and expression features as the key value within the cross-attention model. The query and key values ​​are separated from two different matrices and then fused together. Specifically, the current feature map in the preset hybrid encoder, i.e., the feature map obtained in step S604, is first extracted. The emotion vector is then copied and expanded so that it is completely consistent with the feature map in both spatial and temporal dimensions. The copied emotion vector is used as the query vector to guide the model to focus on emotion-related features, and the expression features are converted into key and value vectors. The key vector is used to calculate the degree of match with the query vector, and the value vector is used to provide feature information after the match. Then, attention weights are calculated based on the query vector and key vector, and the value vectors are weighted summed using these weights to obtain the attention output. The attention output is fused with the original feature map, such as by element-by-element addition or concatenation, to form a hybrid feature containing emotion-guided expression features. This process uses a pre-installed hybrid encoder to fuse lip shape features, linking speech signals with visual representations. This helps ensure the synchronization of lip shape and pronunciation during subsequent expression generation, allowing the digital human's speech, lip shape, and facial expressions to form a unified whole, enhancing the realism of interactions. Furthermore, by generating queries using emotion vectors and calculating similarity with the keys generated from expression features, the model only weights emotion-related visual features. This ensures that key emotional cues dominate the fusion results, achieving a lossless mapping of speech emotions to facial muscle movements, and resolving the pain points of stiff expressions and emotional misalignment found in general-purpose driver solutions.

[0110] In a feasible embodiment, the process of obtaining mixed features may include: the current speech features are not sufficiently associated with the emotional features, resulting in inconsistency between the expression and the speech content. Therefore, the current feature map in the preset mixed encoder is extracted, and the emotion vector is copied to the same spatiotemporal dimension as the feature map to obtain the copied emotion vector, so that the global emotion information is corresponding to the local spatiotemporal features. The copied emotion vector is then linearly projected based on the preset first linear projection layer to convert the emotion vector into a query vector of the cross-attention mechanism. The expression features are linearly projected based on the preset second linear projection and third linear projection layers respectively to convert the expression features into a key vector and a value vector of the cross-attention mechanism. Thus, based on the query vector, the key vector and the value vector, the attention output is fused into the feature map to obtain a weighted fused mixed feature.

[0111] S204: Input the mixed features into a preset decoder for decoding to obtain facial expression coefficients.

[0112] After obtaining the mixed features, they are fed into a pre-decoder for interpolation, yielding facial expression coefficients. The pre-decoder converts the fused features into specific facial expression parameters. Still based on the Transformer architecture, self-attention is used to ensure coherence between expression frames. The output layer maps the high-dimensional features into control parameters that drive the 3D digital human, ensuring that the final expression is both consistent with the speech content and emotionally expressive.

[0113] In some embodiments, step S704 inputs the mixed features into a preset decoder for decoding to obtain a facial expression coefficient, specifically including:

[0114] S801: Inputting the mixed features into a preset decoder, so that the preset decoder captures the inter-frame dependency of the facial animation based on a self-attention mechanism to obtain decoded output data;

[0115] S802: Mapping the decoded output data to the facial expression coefficient dimension of the digital human based on the fully connected layer of the preset decoder to obtain the facial expression coefficient corresponding to each frame of the facial expression animation.

[0116] The mixed features are input into a pre-built decoder, which uses a self-attention mechanism to analyze the associations between different frames in the facial animation. The self-attention mechanism captures the temporal continuity of facial expression changes by calculating the weighted relationship between each frame's features and those of all other frames. For example, the mouth shape features of the current frame are associated with the mouth corner movement features of the previous frame, ensuring smooth and seamless facial animation. After the mixed features are input as a sequence, the self-attention mechanism generates decoded output data that includes inter-frame dependencies. The decoded output data undergoes linear transformation and nonlinear mapping through the fully connected layers of the pre-built decoder, converting the abstract feature space into the specific parameter dimensions of the digital human's facial expression. It should be noted that facial expression coefficients typically correspond to facial action units of the digital human, such as "mouth corner lift coefficient" and "frown coefficient." Each coefficient directly controls the deformation amplitude of a specific facial region.

[0117] During this process, the decoder uses a self-attention mechanism to analyze the cross-frame dependencies of mixed features, accurately capturing the physical continuity of facial expressions and motion, helping to eliminate animation stuttering caused by traditional frame-by-frame prediction. Dimensional mapping through the fully connected layer ensures that the target digital human's expression coefficient dimensions are directly used in the 3D engine, avoiding secondary conversion issues.

[0118] In some embodiments, in the above steps, the emotion classification loss determined by the cross entropy loss function is optimized during the training of the preset emotion encoder, and the expression regression loss determined by the L1 smoothing loss function is optimized during the training of the preset decoder. Figure 3The training process shown uses a dual-loss supervision mechanism. The expression loss ensures that the generated facial movements are accurate and natural, while the emotion classification loss constrains the emotion vectors to conform to predefined emotion categories. These two losses are optimized synergistically, enabling the model to learn the mapping from speech to expression end-to-end while maintaining interpretability and controllability of the emotions. The expression regression loss is shown below:

[0119] , which uses L1 smoothing loss to calculate the L1 smoothing distance between the predicted expression and the real expression, Bt represents the real expression coefficient, Represents the predicted expression coefficient, which is the L1 smoothed distance of the blendshape at each time frame from t to T according to the time series.

[0120] The calculation of sentiment classification loss is as follows:

[0121] First, the emotion vector is mapped to the category space to obtain the predicted probability p. The true emotion category is y. The emotion prediction loss is calculated using the category cross-entropy loss function. c represents the cth emotion category. The total emotion loss is summed up for the total emotion loss of C categories. The final total training loss function of the entire network combines the expression loss and emotion loss, and a hyperparameter is introduced to control the difference between the two to achieve a balance.

[0122] S205: Based on the facial expression coefficient, perform expression driving of the digital human.

[0123] After obtaining the facial expression coefficients, the digital human expression can be driven according to the facial expression coefficients, for example, the facial expression coefficients are input into the corresponding engine, and the digital human expression is driven according to the engine output instructions. Figure 4 As shown, in some embodiments, after the expression drive of the digital human is realized, the expression can be fed back to the user end in real time, and then the voice signal uploaded by the user end can be continuously obtained to realize the iterative update of the expression.

[0124] The embodiment of the present application also provides a digital human expression driving device, such as Figure 6 As shown, a digital human expression driving device includes:

[0125] The acquisition unit 601 is configured to receive a voice signal input by a user and acquire a voice feature corresponding to the voice signal;

[0126] The processing unit 602 is configured to process the speech feature using a preset emotion encoder to obtain emotion prediction data corresponding to the speech feature;

[0127] A fusion unit 603 is configured to fuse the emotion prediction data with basic features corresponding to a feature map of a preset hybrid encoder through a cross-attention mechanism to obtain a hybrid feature;

[0128] A decoding unit 604 is configured to input the mixed features into a preset decoder for decoding to obtain a facial expression coefficient;

[0129] The driving unit 605 is used to drive the expression of the digital human based on the facial expression coefficient.

[0130] In some embodiments, the speech features are processed by a preset emotion encoder to obtain emotion prediction data corresponding to the speech features, specifically including:

[0131] Adding corresponding position codes to the speech features based on the time sequence of the speech features;

[0132] Based on the linear transformation of the speech feature, obtaining a self-attention calculation matrix corresponding to the speech feature;

[0133] Determining the self-attention architecture of the preset emotion encoder through the self-attention calculation matrix and the preset activation function; wherein the preset emotion encoder includes: an emotion encoder and an expression encoder;

[0134] The speech features corresponding to the position codes are processed based on the self-attention architecture to obtain emotion prediction data corresponding to the speech features; wherein the emotion prediction data includes: emotion vectors and expression features.

[0135] In some embodiments, before processing the speech features based on a preset emotion encoder to obtain emotion prediction data corresponding to the speech features, the method further includes:

[0136] Determine the application scenario corresponding to the digital human to obtain the emotional needs corresponding to the application scenario; wherein the emotional needs include: core emotional needs and conventional emotional needs;

[0137] Identifying the emotion type and emotion expression target corresponding to the core emotion need, and determining a corresponding encoder calculation model based on the emotion type and the emotion expression target;

[0138] Speech signal samples matching the core emotional requirements and the conventional emotional requirements are collected, so as to train the encoder calculation model based on the speech signal samples and obtain a preset emotion encoder.

[0139] In some embodiments, the emotion prediction data is fused with the basic features corresponding to the feature map of the preset hybrid encoder through a cross-attention mechanism to obtain a hybrid feature, specifically including:

[0140] Extracting a current feature map in the preset hybrid encoder, and copying the emotion vector to the same spatiotemporal dimension as the feature map to obtain a copied emotion vector;

[0141] Converting the copied emotion vector into a query vector of the cross-attention mechanism, and converting the expression feature into a key vector and a value vector of the cross-attention mechanism;

[0142] Based on the query vector, the key vector and the value vector, the attention output is fused to the basic features corresponding to the feature map to obtain a weighted fused hybrid feature.

[0143] In some embodiments, the mixed features are input into a preset decoder for decoding to obtain facial expression coefficients, specifically including:

[0144] Inputting the mixed features into a preset decoder, so that the preset decoder captures the inter-frame dependency of the facial animation based on a self-attention mechanism, and obtains decoded output data;

[0145] The decoded output data is mapped to the facial expression coefficient dimension of the digital human based on the fully connected layer of the preset decoder to obtain the facial expression coefficient corresponding to each frame of the facial expression animation.

[0146] In some embodiments, receiving a voice signal input by a user and obtaining a voice feature corresponding to the voice signal specifically includes:

[0147] Receive a voice signal input by a user, perform denoising on the voice signal, and obtain a denoised voice signal;

[0148] Based on a preset peak value and a preset audio energy, normalizing the denoised speech signal to obtain a processed speech signal; wherein the preset peak value and the preset audio energy are determined based on the recorded speech of the user;

[0149] The processed speech signal is extracted based on the pre-trained speech model to obtain speech features of preset dimensions.

[0150] In some embodiments, before integrating the emotion prediction data into a feature map of a preset hybrid encoder through a cross-attention mechanism to obtain a hybrid feature, the method further includes:

[0151] determining a speech text corresponding to the processed speech signal and an additional text corresponding to the processed speech signal;

[0152] Determining the phonemes corresponding to the voice text and the personalized content corresponding to the additional text;

[0153] Aligning the phoneme with the personalized content based on timestamp information and embedding them into the voice feature;

[0154] The embedded speech features are extracted by the preset hybrid encoder to obtain basic features integrated with lip shape features, so as to generate a feature map of the preset hybrid encoder according to the basic features.

[0155] In some embodiments, the emotion classification loss determined by the cross entropy loss function is optimized during the training of the preset emotion encoder, and the expression regression loss determined by the L1 smoothing loss function is optimized during the training of the preset decoder.

[0156] The embodiment of the present application also provides a digital human expression driving device, such as Figure 7 As shown, a digital human expression driving device includes:

[0157] at least one processor; and,

[0158] a memory communicatively connected to the at least one processor; wherein,

[0159] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any of the above-mentioned digital human expression driving methods.

[0160] The various embodiments in this application are described in a progressive manner. Similar portions between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from the other embodiments. In particular, the device, apparatus, and non-volatile computer storage medium embodiments are generally similar to the method embodiments, so their descriptions are relatively simple. For relevant portions, refer to the descriptions of the method embodiments.

[0161] The foregoing description describes specific embodiments of the present application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0162] The foregoing is merely one or more embodiments of the present application and is not intended to limit the present application. For those skilled in the art, various modifications and variations may be made to one or more embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present application shall be included within the scope of the claims of the present application.

Claims

1. A method for driving digital human expression, characterized in that: The method comprises: Receive a voice signal input by a user and obtain voice features corresponding to the voice signal; Processing the speech features through a preset emotion encoder to obtain emotion prediction data corresponding to the speech features; The emotion prediction data is fused with the basic features corresponding to the feature map of the preset hybrid encoder through a cross-attention mechanism to obtain a hybrid feature; wherein the feature map of the preset hybrid encoder is formed by embedding the phonemes corresponding to the speech signal into the speech features, and the embedded speech features are subjected to feature extraction by the preset hybrid encoder to obtain the basic features fused with the lip shape features, so as to generate the hybrid feature based on the basic features; Inputting the mixed features into a preset decoder for decoding to obtain a facial expression coefficient; Based on the facial expression coefficient, driving the digital human's expression; The speech features are processed by a preset emotion encoder to obtain emotion prediction data corresponding to the speech features, specifically including: Adding corresponding position codes to the speech features based on the time sequence of the speech features; Based on the linear transformation of the speech feature, obtaining a self-attention calculation matrix corresponding to the speech feature; Determining the self-attention architecture of the preset emotion encoder through the self-attention calculation matrix and the preset activation function; wherein the preset emotion encoder includes: an emotion encoder and an expression encoder; The speech features corresponding to the position codes are processed based on the self-attention architecture to obtain emotion prediction data corresponding to the speech features; wherein the emotion prediction data includes: emotion vectors and expression features.

2. The expression driving method of a digital human according to claim 1, characterized in that: Before processing the speech feature based on a preset emotion encoder to obtain emotion prediction data corresponding to the speech feature, the method further includes: Determine the application scenario corresponding to the digital human to obtain the emotional needs corresponding to the application scenario; wherein the emotional needs include: core emotional needs and conventional emotional needs; Identifying the emotion type and emotion expression target corresponding to the core emotion need, and determining a corresponding encoder calculation model based on the emotion type and the emotion expression target; Speech signal samples matching the core emotional requirements and the conventional emotional requirements are collected, so as to train the encoder calculation model based on the speech signal samples and obtain a preset emotion encoder.

3. The expression driving method of a digital human according to claim 1, characterized in that: The emotion prediction data is fused with the basic features corresponding to the feature map of the preset hybrid encoder through a cross-attention mechanism to obtain hybrid features, specifically including: Extracting a current feature map in the preset hybrid encoder, and copying the emotion vector to the same spatiotemporal dimension as the feature map to obtain a copied emotion vector; Converting the copied emotion vector into a query vector of the cross-attention mechanism, and converting the expression feature into a key vector and a value vector of the cross-attention mechanism; Based on the query vector, the key vector and the value vector, the attention output is fused to the basic features corresponding to the feature map to obtain a weighted fused hybrid feature.

4. The expression driving method of a digital human according to claim 1, characterized in that: The mixed features are input into a preset decoder for decoding to obtain facial expression coefficients, specifically including: Inputting the mixed features into a preset decoder, so that the preset decoder captures the inter-frame dependency of the facial animation based on a self-attention mechanism, and obtains decoded output data; The decoded output data is mapped to the facial expression coefficient dimension of the digital human based on the fully connected layer of the preset decoder to obtain the facial expression coefficient corresponding to each frame of the facial expression animation.

5. The method for driving digital human expression according to claim 1, characterized in that: Receiving a voice signal input by a user and obtaining voice features corresponding to the voice signal specifically includes: Receive a voice signal input by a user, perform denoising on the voice signal, and obtain a denoised voice signal; Based on a preset peak value and a preset audio energy, normalizing the denoised speech signal to obtain a processed speech signal; wherein the preset peak value and the preset audio energy are determined based on the recorded speech of the user; The processed speech signal is extracted based on the pre-trained speech model to obtain speech features of preset dimensions.

6. The expression driving method of a digital human according to claim 5, characterized in that: Before obtaining the hybrid features, the method further includes fusing the emotion prediction data with the basic features corresponding to the feature map of the preset hybrid encoder through a cross-attention mechanism: determining a speech text corresponding to the processed speech signal and an additional text corresponding to the processed speech signal; Determining the phonemes corresponding to the voice text and the personalized content corresponding to the additional text; Aligning the phoneme with the personalized content based on timestamp information and embedding them into the voice feature; The embedded speech features are extracted by the preset hybrid encoder to obtain the basic features fused with the lip shape features, so as to generate a feature map of the preset hybrid encoder according to the basic features.

7. The method for driving digital human expression according to claim 1, characterized in that: During the training of the preset emotion encoder, the emotion classification loss determined based on the cross entropy loss function is optimized, and during the training of the preset decoder, the expression regression loss determined based on the L1 smoothing loss function is optimized.

8. A digital human expression driving device, characterized in that: The device comprises: An acquisition unit, configured to receive a voice signal input by a user and acquire a voice feature corresponding to the voice signal; a processing unit, configured to process the speech feature using a preset emotion encoder to obtain emotion prediction data corresponding to the speech feature; a fusion unit, configured to fuse the emotion prediction data with basic features corresponding to a feature map of a preset hybrid encoder through a cross-attention mechanism to obtain hybrid features; wherein the feature map of the preset hybrid encoder is formed by embedding phonemes corresponding to the speech signal into the speech features, performing feature extraction on the embedded speech features through the preset hybrid encoder, obtaining the basic features fused with lip shape features, and generating the hybrid features based on the basic features; A decoding unit, configured to input the mixed features into a preset decoder for decoding to obtain a facial expression coefficient; A driving unit, configured to drive the expression of the digital human based on the facial expression coefficient; The speech features are processed by a preset emotion encoder to obtain emotion prediction data corresponding to the speech features, specifically including: Adding corresponding position codes to the speech features based on the time sequence of the speech features; Based on the linear transformation of the speech feature, obtaining a self-attention calculation matrix corresponding to the speech feature; Determining the self-attention architecture of the preset emotion encoder through the self-attention calculation matrix and the preset activation function; wherein the preset emotion encoder includes: an emotion encoder and an expression encoder; The speech features corresponding to the position codes are processed based on the self-attention architecture to obtain emotion prediction data corresponding to the speech features; wherein the emotion prediction data includes: emotion vectors and expression features.

9. A digital human expression driving device, characterized in that: The device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: execute the digital human expression driving method described in any one of claims 1 to 7.

Images