Deep Learning-Based Intelligent Interaction Methods and Systems for Digital Humans

Abstract

This invention proposes a deep learning-based intelligent interaction method and system for digital humans. The method includes: acquiring user interaction data, including speech streams, corresponding text, and text history context; inputting the text into a BERT-base pre-trained model to output a current-round semantic vector; determining the user's prosodic style vector based on the current-round semantic vector; obtaining a current-round user style vector using a neural network based on the current-round semantic vector and prosodic style vector; constructing a semantic trajectory model based on the current-round user style vector and the current-round semantic vector, and outputting a semantic intent offset vector; and generating a unified control signal using a control vector generator based on the current-round user style vector, the current-round semantic vector, and the semantic intent offset vector to form the final multimodal response of the digital human interaction.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of digital human intelligent interaction, and particularly relates to a digital human intelligent interaction method and system based on deep learning. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, digital humans, as a kind of human-computer interaction carrier with virtual appearance, voice output, action feedback and language understanding ability, have been widely applied in virtual customer service, intelligent assistants, online education, digital companions and other fields. Especially under the promotion of multi-modal deep learning, semantic modeling and synthesis technology, the natural language processing ability, voice synthesis accuracy and action generation synchronization of digital humans have been significantly improved, making them gradually possess the basis for multi-round natural interaction with users. However, the current mainstream digital human interaction system still faces multiple challenges in real applications.

[0003] Firstly, the existing system generally adopts a pre-trained large language model combined with traditional intent recognition methods, which has great limitations in dealing with complex expressions, multi-round questioning or emotional fluctuations of users. Although these systems can perform semantic understanding on user input and generate reasonable answers, they often lack the ability to recognize and adapt to the user's expression style, language tone and personal inclination in the interaction process, resulting in a rigid and stereotyped system response that lacks personalization. Secondly, in the multi-round dialogue scenario, the user's interactive intent is often dynamically changing, i.e., the "intent drift" phenomenon frequently occurs, for example, the user changes from question A to question B in the consultation process, but does not explicitly indicate the switch. Traditional interaction systems rely on keyword matching or static context analysis, making it difficult to accurately identify subtle changes in intent, and easily causing problems such as answering the wrong question, topic confusion or context disorder. In addition, most existing systems separate the deployment of modules such as voice synthesis, text generation, expression and action driving, lack of unified control signals, often causing semantic expression and tone mismatch, language content and facial expression disconnection, further weakening the naturalness and immersion of the interaction.

[0004] Furthermore, current methods mostly focus on semantic correctness or emotion recognition, but lack in-depth modeling in the key aspect of "how to establish a personalized communication style with a specific user". The user's speaking manner, speech tone, and wording preference often affect their acceptance and trust of the digital human response. Existing interaction systems are difficult to learn the user's expression habits and form an output style that fits their preferences in a short time, making it difficult to simulate the "personification empathy" communication experience in the interaction process. In summary, the current digital human intelligent interaction technology still has significant technical shortcomings in expression style unification, dynamic intent perception, modality consistency, personalized response generation, etc., and it is urgent to build a systematic and multi-module collaborative intelligent interaction mechanism to improve the naturalness, continuity and user satisfaction of digital human interaction. SUMMARY

[0005] The purpose of the present application is to propose a digital human intelligent interaction method and system based on deep learning, which constructs an interaction framework that can perceive user expression habits and has semantic adaptability, to solve the defects of the prior art in expression personalization, semantic continuity and multi-modal coordination.

[0006] In order to achieve the above purpose, in the first aspect of the present application, a digital human intelligent interaction method based on deep learning is provided, which comprises the following steps:

[0007] S1, obtaining user interaction data including voice stream, corresponding text and text history context, inputting the text into a BERT-base pre-training model, outputting a current round semantic vector, and determining a user's prosody style vector according to the current round semantic vector;

[0008] S2, based on the current round semantic vector and prosody style vector, using a neural network to obtain a current round user style vector;

[0009] S3, constructing a semantic trajectory model according to the current round user style vector and combining the current round semantic vector, outputting a semantic intention offset vector;

[0010] S4, based on the current round user style vector, current round semantic vector and semantic intention offset vector, using a control vector generator to generate a unified control signal to form a final multi-modal response of digital human interaction.

[0011] Further, the text is input into the BERT-base pre-training model, the output vector at the [CLS] position is extracted as the current round semantic vector, and the user's prosody style vector is determined according to the current round semantic vector.

[0012] The text is input into the BERT-base pre-training model, and the output vector at the [CLS] position is extracted as the current round semantic vector;

[0013] The prosody style vector is determined in combination with the speech rate, fundamental frequency, pause ratio and emotional fluctuation rate, and the prosody style vector mutation leading to style drift anomaly is controlled based on a regularization term; wherein the regularization term is determined based on the semantic vector and the prosody style vector for style consistency measurement.

[0014] Further, the neural network is a two-layer feedforward network, wherein the first layer is upgraded to 64 with ReLU activation; the second layer is upgraded to 768 and aligned with the semantic vector.

[0015] Further, the neural network is a two-layer feedforward network, wherein the first layer is upgraded to 64 with ReLU activation; the second layer is upgraded to 768 and aligned with the semantic vector.

[0016] inputting the prosody style vector into the neural network to obtain a prosody style vector aligned with the current round semantic vector, splicing the prosody style vector and the current round semantic vector, normalizing the spliced vector, and generating a current round style embedding;

[0017] introducing an evolution-aware smoothing mechanism to the current round style embedding to enhance the temporal consistency of the current style representation, and generating a final current round style embedding;

[0018] The evolution-aware smoothing mechanism represents a semantic stability decay function of the current round semantic vector. If the current round style embedding fluctuates the most, it will be punished the most.

[0019] Further, according to the current round user style vector, a semantic trajectory model is constructed in combination with the current round semantic vector, and a semantic intention offset vector is output, specifically including:

[0020] processing the semantic vector sequence using a GRU structure;

[0021] using the final current round style embedding as an adjustment vector, encoding it into a style gate vector, and participating in semantic weight distribution;

[0022] comparing the semantic trajectory model output with the current round semantic vector to form a semantic intention offset vector.

[0023] Further, the semantic weight is calculated as follows:

[0024]

[0025] wherein β i is the semantic weight, represents the similarity between the current semantic and the historical semantic; represents the consistency degree of the current style and the historical style.

[0026] Further, the control vector generator is composed of a shallow fully connected network and an attention gate structure.

[0027] Further, based on the current round user style vector, the current round semantic vector, and the semantic intention offset vector, a unified control signal is generated using a control vector generator to form a final multi-modal response of digital human interaction, specifically including:

[0028] inputting the current round user style vector, the current round semantic vector, and the semantic intention offset vector into the control vector generator to generate a unified control signal;

[0029] decoding the unified control signal into a text response sequence, a voice control parameter, and an action / emotion vector;

[0030] generating a final multimodal response based on the text response sequence, the voice control parameters and the action / emotion vector;

[0031] converting the final multimodal response into control signals for driving the actual digital human system execution module to perform:

[0032] inputting the text response sequence into a subtitle display layer or UI rendering module; converting the voice control parameters into prosody and emotion parameters required by the TTS model, and synthesizing WAV format audio, and pushing to the audio playback system; mapping the action / emotion vector to the digital human expression / action template library, and driving the digital human character to perform the corresponding action in the rendering engine.

[0033] Further, the final current round style embedding is used to control the style template selection of the text sentence of the digital human, the speed and tone in the TTS module, and the detailed parameters of the expression control.

[0034] In a second aspect of the present application, a deep learning-based digital human intelligent interaction system is provided, which comprises:

[0035] An interaction data acquisition module is configured to acquire user interaction data, including a voice stream, corresponding text and a text history context, input the text into a BERT-base pre-training model, output a current round semantic vector, and determine a user prosody style vector based on the current round semantic vector.

[0036] An interaction style analysis module is configured to obtain a current round user style vector based on the current round semantic vector and the prosody style vector using a neural network.

[0037] An interaction intention analysis module is configured to construct a semantic trajectory model based on the current round user style vector and the current round semantic vector, and output a semantic intention offset vector.

[0038] A digital human interaction module is configured to generate a unified control signal using a control vector generator based on the current round user style vector, the current round semantic vector and the semantic intention offset vector, so as to form a final multimodal response of digital human interaction.

[0039] The present application has at least the following beneficial technical effects:

[0040] This invention addresses the shortcomings of existing technologies in terms of personalized expression, semantic continuity, and multimodal coordination by proposing a novel deep learning-based intelligent interaction method and system for digital humans. The invention approaches the issue from two core dimensions: user language style modeling and dynamic recognition of interaction intent. It constructs an interaction framework that can perceive user expression habits and possess semantic adaptability. In this invention, the system can analyze user speech and text input in real time to extract style information such as speaking rhythm, sentence structure, and tone features, and use this information to drive language generation and speech synthesis processes. This makes the digital human's interaction style more closely match the user's, enhancing its personalized adaptability. Simultaneously, the system can monitor changes in user intent during dialogue, identify potential topic shifts or semantic deviations, and dynamically adjust response strategies to avoid semantic gaps and contextual confusion, ensuring the coherence of the interaction. Furthermore, this invention constructs a unified control signal mechanism to achieve multimodal consistency in text, speech, facial expressions, and action generation, ensuring the coordinated unity of interactive output in both content and presentation.

[0041] Through the synergistic integration of the above mechanisms, this invention realizes an intelligent interactive system with empathy, intention adaptation, and natural expression capabilities, which significantly improves the interaction quality and user experience of digital humans in real-world application scenarios. Attached Figure Description

[0042] The present invention will be further described with reference to the accompanying drawings, but the embodiments in the drawings do not constitute any limitation on the present invention. For those skilled in the art, other drawings can be obtained based on the following drawings without creative effort.

[0043] Figure 1 This is a flowchart of the deep learning-based digital human intelligent interaction method of the present invention. Detailed Implementation

[0044] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0045] like Figure 1 As shown in the embodiment of the present invention, a deep learning-based digital human intelligent interaction method is provided, the method comprising:

[0046] S1. Obtain user interaction data, including voice stream, corresponding text, and text history context. Input the text into the BERT-base pre-trained model, output the current round semantic vector, and determine the user's prosodic style vector based on the current round semantic vector.

[0047] Specifically, this step aims to structure the user's current speech and text input, outputting two types of input variables that can be directly used in subsequent style modeling (step 2) and semantic drift recognition (step 3): semantic vector E. t With style feature vector F t .

[0048] Input source:

[0049] The current round of interactive input is:

[0050] Voice Stream V t User voice, acquired via a 16kHz device, in 64-dimensional Mel spectrum format.

[0051] Text x t : By V t Translated using ASR (using Conformer + CTCBeamSearch).

[0052] Context H 1:t-1 : Text records of the user's first t-1 rounds of conversation.

[0053] Text semantic vector E t Extraction (for semantic modeling):

[0054] Current text x t The vector is fed into the BERT-base pre-trained model, and the output vector at the [CLS] position is extracted as the global representation of the semantic expression in this round:

[0055]

[0056] Where: E t This represents the semantic vector for the current round, which will be used for subsequent style modeling (step 2) and intent drift recognition (step 3); x t It is the text transcribed by ASR in the current round; the BERT model is used to extract semantic representations that contain context dependencies.

[0057] Furthermore, to reflect the salient features of user style in speech expression, this invention introduces a vector consisting of the following four dimensions:

[0058]

[0059] Where: r t Speech rate, measured in syllables per second, is defined as follows:

[0060]

[0061] N sT represents the number of syllables (obtained from the number of words output by ASR via dictionary mapping). d The duration of the speech (calculated from the start and end times of the frame);

[0062] f0: Fundamental frequency, representing the reference frequency of speech, extracted from the Mel spectrum by the YIN algorithm;

[0063] p t Pause Ratio: This represents the proportion of frames with energy below a threshold out of the total number of frames (reflecting the rhythm of the expression).

[0064] σ e Emotional volatility, representing the variance of the emotional label distribution in the current speech segment, is defined as:

[0065]

[0066] Where s i Let be the emotion score of the i-th frame (predicted by BiLSTM). The average sentiment value across N frames.

[0067] Furthermore, the emotion-style joint regularization term is constructed as follows: To control style drift caused by abrupt changes in the style vector, this invention designs the following regularization term as a style change control factor (used during training):

[0068]

[0069] Wherein: F t ,F t-1 E represents the style vectors for the current and previous rounds. t E t-1 represents the semantic vectors of the current and previous texts; cos(·,·) is the cosine similarity of the vectors; when the semantics remain unchanged but the style changes abruptly, this cost increases, prompting style changes to keep semantic consistency in sync.

[0070] Understandably, this step completes the decoupling and structural alignment of the underlying information for style and semantic modeling in digital human interaction, laying an executable, scalable, and controllable multimodal input foundation for the subsequent construction of a style-semantic collaborative control model.

[0071] S2. Based on the current round semantic vector and prosodic style vector, obtain the current round user style vector using a neural network.

[0072] Specifically, this step aims to transform the semantic vector E extracted in step one... t And rhythmic style vector F t Further integrate and transform into a unified user language style representation S u,tIn particular, digital human interaction faces challenges such as highly personalized input, strong fluctuations in tone, and inconsistent expression rhythm. This invention, considering these characteristics, proposes a user style modeling mechanism with low-dimensional interpretability, continuously evolving style, and strong controllability. This mechanism is constructed through the following three key paths:

[0073] Furthermore, current round user style fusion modeling:

[0074] To address the challenges of separating semantic and style information during style control and the difficulty in expressing high-dimensional control objectives due to low style dimensionality, this step employs a style-semantic guided fusion structure:

[0075] S u =LayerNorm(W s ·(φ(F t )⊕E t )+b s (6)

[0076] in, Embed the current wheel style; φ(F t ) is a two-layer feedforward network: the first layer is increased to 64 dimensions and activated as ReLU; the second layer is increased to 768 dimensions and activated by E. t Alignment; ⊕ indicates vector concatenation operation; These are trainable linear mapping parameters; b s The bias term is defined by LayerNorm(·), which normalizes the fused vector to improve stability and convergence speed.

[0077] Understandably, the advantages of this structure are:

[0078] Instead of using prosody as classification or labeling information, it directly guides the generation of language generation control vectors;

[0079] This avoids the risk of style information loss or weight imbalance in high-dimensional semantic space;

[0080] It allows for precise control over the rhythm, tone, emotional context, and other aspects of the text generation stage, enabling more natural anthropomorphic language generation.

[0081] Furthermore, a style evolution smoothing mechanism (historical memory enhancement) is introduced: To overcome the problems of style instability in short dialogues and the gradual emergence of user style in multiple rounds of dialogue, an evolution-aware smoothing mechanism is introduced to enhance the temporal consistency of the current style representation.

[0082]

[0083] Wherein: S u : The final style vector generated in the current round; S u,t: The style vector ultimately used to generate stage control; S u,t-j : Historical style vector; α∈[0,1]: Controls the weight of merging the current and historical styles, usually set to 0.6–0.8; k: Window length, usually 3 to 5 rounds of dialogue.

[0084] Understandably, this mechanism effectively prevents style jumps while also allowing the system to respond to changes in the user's expressive style, achieving "evolvable, continuous, and controllable" style, and possessing significant industrialization potential.

[0085] Furthermore, to address the abnormal fluctuations in style vectors that may occur during digital human interaction due to environmental changes (such as voice quality degradation), this invention designs a robust regularization term based on joint control of style change rate and semantic stability:

[0086]

[0087] The meaning of this regular expression is: when the semantic change is small (E) t With E t-1 When similar), if S u Large fluctuations will be penalized; exp(-cos(·)) serves as a semantic stability decay function (the more stable, the stronger the penalty); controlling style to change synchronously with semantics rather than drifting is an innovative regularization term proposed in this invention for "style-semantic dynamic balancing".

[0088] It should be noted that this factor is incorporated into the total loss function during the training phase through weighting, and in practice, it can reduce style fluctuations exceeding 0.15 Euclidean distance by about 36%.

[0089] Final output variable: Final current round style embedding Current round end-user style vector.

[0090] S3. Based on the current round user style vector and the current round semantic vector, construct a semantic trajectory model and output a semantic intent offset vector.

[0091] Specifically, this invention addresses phenomena such as semantic jumps, ambiguous expressions, and indirect transitions of intent in digital human interaction by proposing a method that integrates user style representation S. u,t With semantic embedding sequence {E t E t-1 ,...,E t-k The drift trajectory modeling method improves the sensitivity and interpretability of semantic offsets.

[0092] Furthermore, a style-adjustment-based semantic trajectory modeling network:

[0093] To achieve robust modeling of semantic misjudgment caused by style jumps, this step constructs a style-adjusting semantic variation encoding network with the following structure:

[0094] Process using a GRU architecture (or a lightweight Transformer)

[0095] S u,t As an adjustment vector, it is encoded as a style gating vector G. t It participates in the allocation of historical semantic weights;

[0096] The final output and the semantics of the current round E t Comparison, forming a semantic offset embedding vector ΔI t .

[0097] The core calculations defined in this invention are as follows:

[0098]

[0099] Where the weight β i Determined by the following formula:

[0100]

[0101] in, Similarity between current semantics and historical semantics; The degree of consistency between current and historical styles; the sum of these two is used as the scoring criterion for attention weighting; ΔI t This is the offset representation of "current semantics - weighted historical semantics center", preserving direction and magnitude.

[0102] The advantage of this mechanism is that by adjusting the reference weight of historical semantics through style consistency, it effectively avoids semantic misjudgment caused by sudden changes in expression, and improves the ability of digital humans to perceive drift in ambiguous language scenarios.

[0103] The output result is The semantic offset vector of the current round relative to the historical trajectory; this vector is not a "classification label" or "strategy selection", but is used as a control signal in subsequent generation steps to participate in the selection of output content; the ΔI output in this step t It is the core modeling component for digital humans to "accurately respond to phenomena such as user topic switching, sudden changes in intent, and contextual breaks".

[0104] S4. Based on the current round user style vector, the current round semantic vector, and the semantic intent offset vector, a unified control signal is generated using a control vector generator to form the final multimodal response of digital human interaction.

[0105] Specifically, the goal of this step is to transform the user language style vector S obtained from the previous steps into a vector that represents the user's language style. u,t (Step 2) and semantic intent offset vector ΔI t (Step 3) Integrate the control to generate a natural, continuous, and personalized multimodal response output Y. t It is then mapped to the text rendering, speech synthesis, and motion-driven modules in the digital human system to complete truly executable intelligent interactive operations.

[0106] The input for this step comes from the previous step, and the meaning of the variables is exactly the same as before, ensuring a closed loop in the information transmission chain:

[0107] The semantic vector of the current round extracted in step one;

[0108] Step two involves constructing a user-defined language style representation;

[0109] The semantic offset vector modeled in step three.

[0110] Furthermore, the design of the multimodal generation control module:

[0111] This invention employs a unified decoding control vector h t This drives the generation of text, speech, and action modal outputs, with the following specific structure:

[0112] h t =f decode (E t ,S u,t ,ΔI t (11)

[0113] Among them, f decode (·) represents the control vector generator, which consists of a shallow fully connected network and an attention gating structure; E t Provides semantic core content; S u,t As a style adjustment factor, it is injected into attention bias and generation structure parameters; ΔI t Control the direction of semantic strategies (topic continuation / transition / clarification, etc.); output h t To unify control signals, the dimension is

[0114] Furthermore, during the multimodal generation stage, the control vector is decoded as follows:

[0115] Text response sequence;

[0116] Voice control parameters (such as prosody curve, speech rate, tone, emotion label);

[0117] Motion / expression vectors (used to control facial blendshape, head movements, and gesture selection).

[0118] The final multimodal response is expressed as:

[0119]

[0120] Semantic-style joint regulation mechanism of response generation:

[0121] To ensure that the multimodal output not only matches the user input in content but also conforms to the user's individual expression style, this invention constructs the following control path:

[0122] When ||ΔI t When the value is small (indicating no deviation in intent), the system will select topic-continuing output, such as adding natural transition markers like "Continue..." or "Indeed."

[0123] When ||ΔI t When the || is large (indicating a semantic change), the system adjusts to a topic shift or clarification, such as "Let me explain..." or "So, next, this invention...";

[0124] At the same time, S u,t control:

[0125] Choose a text sentence style template (e.g., plain or empathetic);

[0126] The speech rate in the TTS module (by r) t Inference), intonation (controlled by f0);

[0127] Detailed parameters in facial expression control include the degree of smile, eye focus, and gesture range.

[0128] This mechanism enables "the strategy response path to be determined by ΔI". t The decision is made by S. u,t The bivariate control generation of the "decision" solves the problem of "semantic correctness but stylistic fragmentation" in traditional interactive systems.

[0129] Furthermore, integration with digital human systems enables the execution of operations:

[0130] This invention features a specially designed output interface that enables Y t Not just the internal results of the model, but control signals that can directly drive the execution modules of a real digital human system:

[0131] Input to the subtitle display layer or UI rendering module;

[0132] The prosody and sentiment parameters required to convert to a TTS model (such as FastSpeech+HiFi-GAN);

[0133] Synthesize WAV format audio and push it to the audio playback system;

[0134] Mapped to digital human expression / motion template libraries (such as blendshape controllers and skeletal animation players); driven in rendering engines (such as Unity3D / Unreal) to perform facial expression changes, gaze adjustments, head movements, gesture expressions, etc.

[0135] This design ensures that the invention has a complete closed loop from "deep learning modeling → response generation → system control", and can be truly implemented as a deployable digital human intelligent interaction system.

[0136] This invention also provides a deep learning-based digital human intelligent interaction system, the system comprising:

[0137] The interaction data acquisition module is used to acquire user interaction data, including voice stream, corresponding text and text history context, input the text into the BERT-base pre-trained model, output the current round semantic vector, and determine the user's prosodic style vector based on the current round semantic vector;

[0138] The interaction style analysis module is used to obtain the current round user style vector based on the current round semantic vector and prosodic style vector using a neural network;

[0139] The interaction intent analysis module is used to construct a semantic trajectory model based on the current round user style vector and the current round semantic vector, and output a semantic intent offset vector.

[0140] The digital human interaction module is used to generate a unified control signal based on the current round user style vector, the current round semantic vector, and the semantic intent offset vector, using a control vector generator to form the final multimodal response of the digital human interaction.

[0141] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0142] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or units may be electrical, mechanical, or other forms.

[0143] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0144] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A digital human intelligent interaction method based on deep learning, characterized in that, The method comprises the following steps: S1, obtaining user interaction data including voice stream, text corresponding to the voice stream, and text history context of the voice stream, inputting the text into a BERT-base pre-training model, outputting a current round semantic vector, and determining a user's prosody style vector according to the current round semantic vector, specifically comprising: inputting the text into the BERT-base pre-training model, extracting the output vector at the [CLS] position as the current round semantic vector; introducing speech speed, fundamental frequency, pause ratio, and emotion fluctuation rate, combining the current round semantic vector to determine the prosody style vector, and based on a regularization term to control the style drift anomaly caused by the sudden change of the prosody style vector; wherein the regularization term is determined based on the semantic vector and the prosody style vector for style consistency measurement, and is used for training the BERT-base pre-training model; S2, based on the current round semantic vector and the prosody style vector, using a neural network to obtain a current round user style vector, specifically comprising: inputting the prosody style vector into the neural network to obtain a prosody style vector aligned with the current round semantic vector, splicing the prosody style vector with the current round semantic vector, normalizing the spliced vector, and generating a current round style embedding; introducing an evolution perception smoothing mechanism to the current round style embedding to enhance the time consistency of the current style representation, and generating a final current round style embedding; wherein the evolution perception smoothing mechanism represents that the semantic stability decay function of the current round semantic vector is minimum, and if the current round style embedding fluctuates maximum, it will be punished maximum; S3, according to the current round user style vector, combining the current round semantic vector, constructing a semantic trajectory model, and outputting a semantic intention offset vector; S4, based on the current round user style vector, the current round semantic vector, and the semantic intention offset vector, using a control vector generator to generate a unified control signal to form a final multi-modal response of digital human interaction.

2. The deep learning-based digital human intelligent interaction method according to claim 1, characterized in that, The neural network is a two-layer feedforward network, wherein the first layer is upgraded to 64 and the activation is ReLU; the second layer is upgraded to 768 and aligned with the semantic vector.

3. The deep learning-based digital human intelligent interaction method of claim 1, wherein, According to the current round user style vector, combining the current round semantic vector, constructing a semantic trajectory model, and outputting a semantic intention offset vector, specifically comprising: using GRU structure to process the current round semantic vector; using the final current round style embedding as an adjustment vector, encoding it into a style gate vector to participate in semantic weight distribution; comparing the output of the semantic trajectory model with the current round semantic vector to form a semantic intention offset vector.

4. The deep learning-based digital human intelligent interaction method according to claim 3, characterized in that, The semantic weight is calculated as follows: ; wherein, is a semantic weight, represents a similarity between the current semantic and the historical semantic; represents a consistency degree between the current style and the historical style; k is a window length.

5. The deep learning based digital human intelligent interaction method according to claim 1, characterized in that, The control vector generator is composed of a shallow fully connected network and an attention gate structure.

6. The deep learning based digital human intelligent interaction method of claim 1, wherein, Based on the current round user style vector, the current round semantic vector, and the semantic intention offset vector, using the control vector generator to generate a unified control signal to form a final multi-modal response of digital human interaction, specifically comprising: inputting the current round user style vector, the current round semantic vector, and the semantic intention offset vector into the control vector generator to generate a unified control signal; decoding the unified control signal into a text response sequence, a voice control parameter, and a motion / emotion vector; generating a final multi-modal response in combination with the text response sequence, the voice control parameter, and the motion / emotion vector; transforming the final multi-modal response into a control signal for driving an actual digital human system execution module to execute: inputting the text response sequence into a subtitle display layer or a UI rendering module; converting the voice control parameter into prosody and emotion parameters required by a TTS model, and synthesizing WAV format audio, and pushing to an audio playback system; mapping the motion / emotion vector to a digital human expression / motion template library, and driving the digital human character to perform corresponding actions in the rendering engine.

7. The deep learning based digital human intelligent interaction method of claim 1, wherein, The final current round style embedding is used to control the style template selection of the text sentence of the digital human, the speed and tone in the TTS module, and the detailed parameters of the expression control.

8. The system for performing the deep learning-based digital human intelligent interaction method of claim 1, wherein The system comprises: An interactive data acquisition module for acquiring user interaction data, including voice stream, corresponding text, and text history context, inputting the text into a BERT-base pre-training model, outputting a current round semantic vector, and determining a user's rhythm style vector according to the current round semantic vector; An interactive style analysis module for obtaining a current round user style vector based on the current round semantic vector and the rhythm style vector using a neural network; An interactive intention analysis module for constructing a semantic trajectory model according to the current round user style vector in combination with the current round semantic vector, and outputting a semantic intention offset vector; A digital human interaction module for generating a unified control signal using a control vector generator based on the current round user style vector, the current round semantic vector, and the semantic intention offset vector, to form a final multi-modal response of digital human interaction.

Images