A digital human dynamic interaction method and system based on deep learning
Through multimodal feature fusion and deep reinforcement learning, the digital human system can accurately understand users' emotions and generate personalized dynamic responses, solving the problems of single emotion understanding and stiff interaction in existing systems, and achieving a natural and smooth personalized interactive experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUANGZHOU JINGCUI EDUCATION TECH CO LTD
- Filing Date
- 2025-08-07
- Publication Date
- 2026-05-19
AI Technical Summary
Existing digital human interaction systems struggle to accurately capture complex emotions, lack flexibility and adaptability in behavioral patterns, exhibit poor real-time dynamic response, and lack personalized adaptability, resulting in stiff, mechanical interactions that fail to achieve natural and smooth dynamic interaction.
Employing multimodal feature fusion, deep reinforcement learning strategies, and a real-time rendering architecture, this system acquires real-time user interaction data, performs multimodal sentiment analysis, generates dynamic response strategies for digital humans, and optimizes the behavioral decision-making model through reinforcement learning to achieve personalized interaction.
It achieves accurate emotional insight, intelligent dynamic response, and provides a personalized interactive experience, improving user experience and system adaptability. It is suitable for fields such as intelligent customer service, virtual assistants, and online education.
Smart Images
Figure CN120688535B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of AI digital human technology, and more particularly to a method and system for dynamic interaction of digital humans based on deep learning. Background Technology
[0002] With the development of artificial intelligence and virtual reality technologies, digital human interaction applications are becoming increasingly widespread, but existing interaction systems have shortcomings.
[0003] Traditional methods often rely on simple keyword matching or pre-defined rule bases to understand user intent and emotions, making it difficult to accurately capture complex emotions. Their emotional understanding is limited in scope, resulting in stiff and mechanical interactions. When users use words that imply deep emotions, digital humans often fail to recognize and respond appropriately. When faced with diverse and contextualized user needs, their behavioral patterns lack flexibility and adaptability, failing to achieve truly natural and fluid dynamic interactions, thus limiting user experience and application expansion. Dynamic response is also poor in real-time; traditional rule-based decision systems require predefined interaction templates, leading to significantly increased response latency in complex dialogue scenarios, with measured average latency exceeding 800ms. Behavioral performance is mechanical and rigid: existing action generation schemes often use linear interpolation animation, lacking modeling of the non-linear relationship between emotional intensity and action amplitude, resulting in digital human actions that do not conform to human social habits. Furthermore, static models lack continuous learning capabilities: they struggle to optimize interaction strategies based on user feedback, failing to achieve personalized adaptation and lacking online update mechanisms. Summary of the Invention
[0004] To address the aforementioned issues, this invention provides a deep learning-based method and system for dynamic interaction of digital humans. Through multimodal feature fusion, deep reinforcement learning strategy optimization, and real-time rendering architecture, it effectively overcomes the technical shortcomings of traditional methods, such as the single dimension of emotional understanding in digital humans, stiff and mechanical interaction, and incompatibility with human social habits.
[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0006] In a first aspect of the present invention, a deep learning-based method for dynamic interaction of digital humans is provided, comprising the following steps:
[0007] S1. Acquire real-time interactive data input by the user and preprocess it. The interactive data includes at least text information, voice information and video image information.
[0008] S2. The preprocessed interaction data is analyzed using a multimodal sentiment analysis model to extract user emotion feature vectors, which include emotion polarity values, emotion intensity values, and intention category labels.
[0009] S3. Input the emotion feature vector into the digital human behavior decision model. The digital human behavior decision model generates a dynamic response strategy for the three-dimensional digital human through a deep reinforcement learning algorithm. The response strategy includes a set of facial expression parameters, a sequence of body movements, and voice feedback content.
[0010] S4. Drive the digital human to perform real-time interaction according to the dynamic response strategy, synchronously monitor user feedback data and establish a reinforcement learning reward function to dynamically update the network parameters of the behavior decision model.
[0011] Preferably, in step S1, the text information includes character-level text data input by the user, including but not limited to chat logs, command statements, and comment content; the voice information includes the user's voice data, including acoustic physical features, prosodic emotional features, and speech semantic features; and the video image information includes the user's video data, including facial dynamic features, body behavior features, and scene context features.
[0012] Preferably, in step S2, the construction of the multimodal sentiment analysis model includes:
[0013] The text processing module uses a BERT-TextCNN hybrid architecture to extract semantic features;
[0014] The image processing module extracts dynamic features of facial micro-expressions using a 3D-CNN network;
[0015] The speech processing module uses Mel spectrogram features combined with Attention-LSTM to extract acoustic features;
[0016] The feature fusion layer uses a gating attention mechanism to dynamically weight multimodal features.
[0017] Preferably, in step S3, the emotion feature vector is input into the digital human behavior decision model to understand the user's real-time emotions and intentions, and to generate corresponding dynamic response behaviors of the digital human according to preset rules and strategies.
[0018] More preferably, the dynamic response behavior of the digital human includes, but is not limited to, facial expression changes, body movements, voice tone adjustments, and dialogue content generation, and a dynamic response strategy score is calculated based on user feedback. The specific calculation formula for the dynamic response strategy score is as follows:
[0019] ,
[0020] Where S represents the dynamic response strategy score, Re w This represents the text bias coefficient of user feedback, where e is a constant, α1 represents the text interaction weight, and d wThis represents the ability score of the multimodal sentiment analysis model on the calculated preliminary text data, α2 represents the voice interaction weight, and d s Re represents the ability score of the multimodal sentiment analysis model on the calculated preliminary speech data. s α represents the voice deviation coefficient of user feedback, α3 represents the video interaction weight, and d v Re represents the capability score of the multimodal sentiment analysis model on the preliminary video data calculated. v This represents the video deviation coefficient as reported by the user.
[0021] Preferably, in step S4, the user's emotion feature vector and dynamic response strategy score are input into a reinforcement learning-based recommendation model, which outputs the optimal response strategy recommendation for each user based on the current dynamic response strategy and historical behavior. The specific process is as follows:
[0022] ,
[0023] Where, r i Indicates the first Let f represent the optimal personalized recommendation for a given user, and let f be the recommendation strategy generated by the reinforcement learning model. Indicates the first A user's emotional feature vector, Indicates the first A dynamic response strategy score for each digital person.
[0024] Even better, the optimal response strategy recommendation is combined with the system execution strategy. During the strategy execution process, each recommended operation is transformed into actual data human actions or language, and a reinforcement learning reward function is introduced. When the emotion matching accuracy and user satisfaction are low, the system automatically adjusts the recommendation strategy to select a digital human interaction mode that is more suitable for the current interaction state.
[0025] In a second aspect of the invention, a deep learning-based digital human dynamic interaction system is provided, comprising: sequentially connected...
[0026] The data acquisition and processing module is used to acquire real-time interactive data input by the user and perform preprocessing.
[0027] The multimodal sentiment analysis module is used to parse preprocessed interaction data based on a multimodal sentiment analysis model and extract user emotion feature vectors.
[0028] The digital human behavior decision generation module is used to input emotional feature vectors into the digital human behavior decision model. The digital human behavior decision model generates dynamic response strategies for the three-dimensional digital human through deep reinforcement learning algorithms, and combines the optimal response strategy recommendation with the system execution strategy to drive the digital human to perform real-time interaction.
[0029] The feedback update module is used to synchronously monitor user feedback data and establish a reinforcement learning reward function to dynamically update the network parameters of the behavioral decision model.
[0030] The beneficial effects of this invention are as follows:
[0031] 1. Precise Emotional Insight: Deeply analyzes the emotions expressed by users' words, accurately grasping their emotional state. Compared to traditional methods, it significantly improves the recognition rate of complex and subtle emotions, making the digital human better understand users' emotions.
[0032] 2. Intelligent Dynamic Response: Real-time understanding of user emotions and intentions, quickly generating dynamic response behaviors that fit the context, including intelligent adjustment of facial expressions, actions, voice and dialogue content, making the interaction natural and coherent, as smooth as real-person communication.
[0033] 3. Personalized interactive experience: Customize exclusive interaction modes based on different user emotions and needs, provide personalized services for each user, and enhance user stickiness and satisfaction.
[0034] 4. Wide applicability: It can be applied to multiple fields such as intelligent customer service, virtual assistant, and online education, providing industries with efficient intelligent interaction solutions, helping enterprises improve service quality and efficiency, and expand business boundaries. Attached Figure Description
[0035] Figure 1 This is a flowchart of a deep learning-based dynamic interaction method for digital humans according to the present invention.
[0036] Figure 2 This is a block diagram of a deep learning-based digital human dynamic interaction system according to the present invention. Detailed Implementation
[0037] Please see Figure 1 As shown, in a first aspect of the present invention, a deep learning-based method for dynamic interaction of digital humans is provided, comprising the following steps:
[0038] S1. Acquire real-time interactive data input by the user and preprocess it. The interactive data includes at least text information, voice information and video image information.
[0039] The core objective of step S1 is to preprocess the real-time interactive data (text, voice, and video / image information) input by the user, transforming it into a unified set of input features to facilitate subsequent modeling. Since the interactive data comes from multiple sources with varying timestamps, data granularity, and formats, time alignment, normalization, and missing value imputation are necessary to ensure the data can be fused into a standardized feature vector.
[0040] The text information includes character-level text data input by the user, including but not limited to chat logs, command statements, and comments; the voice information includes the user's voice data, including acoustic physical features, prosodic emotional features, and speech semantic features; the video image information includes the user's video data, including facial dynamic features, body behavior features, and scene context features.
[0041] Interactive data includes:
[0042] Text data, including but not limited to chat logs, command statements, and comment content;
[0043] Speech data, including acoustic physical features, prosodic and emotional features, and speech semantic features;
[0044] Video data, including facial dynamic features, body behavior features, and scene context features.
[0045] The data preprocessing process consists of three core steps: time alignment, normalization, and missing value imputation, to ensure data quality and consistency.
[0046] Time alignment:
[0047] Since the interaction data comes from multiple different sources, their sampling frequencies and timestamps are not necessarily perfectly aligned. Time alignment is crucial to ensure that data from different data sources can be effectively combined during model training. The specific steps are as follows:
[0048] Video data ( It records the start and end times of user actions. Each interaction has a defined time period, which will be used to align with device status data.
[0049] Text data ( ) and voice data ( The sampling frequencies of data sources typically differ. For example, voice data might be sampled every second, while text data might be sampled every 5 minutes. All data is aligned using timestamps. Specifically, if the sampling interval for device data is 1 second, the data is sampled to the start timestamp of each user action, and then interpolated according to a time window based on the duration of the action. For text data, it is aligned to the most recent interaction moment, and if any data is missing, linear interpolation is used to fill in the missing data.
[0050] Normalization process:
[0051] The purpose of normalization is to eliminate dimensional differences between different data sources and different features, so that they can be compared and calculated on the same scale.
[0052] The advantage of normalization is that it ensures that there are no differences in scale between different features, so that subsequent model training will not be affected by the scale of a certain feature being too large or too small.
[0053] Missing value imputation:
[0054] During actual data collection, some data may be missing due to equipment malfunction or external factors. To ensure the continuity and stability of subsequent processing, missing value imputation is an essential step. The specific imputation method is as follows:
[0055] The specific method involves filling in missing values by calculating the average of several adjacent data points within the same time window. This effectively avoids bias caused by sudden changes or erroneous data.
[0056] The output of step S1 is a set of structured behavioral samples. The sample set was obtained by merging after preprocessing. It is The matrix is denoted by N, where N is the number of samples (i.e., the number of car washes), and each row represents the feature data of a single user interaction. This output will be used as input for user behavior modeling in subsequent steps. Specifically, Each line in the output contains three types of information: user text, voice, and video / image data. This output will serve as input for subsequent steps in user behavior modeling.
[0057] In step S1, multi-source data from user operations, device status, and the external environment are effectively aligned, normalized, and imputed for missing values, generating input feature vectors in a unified format. .
[0058] During the processing, data consistency and physical validity were ensured, avoiding instability caused by time misalignment, differences in feature scales, or missing data. These processes provided high-quality, reliable input data for subsequent user behavior modeling, ensuring the stability and operability of the entire system.
[0059] S2. The preprocessed interaction data is analyzed using a multimodal sentiment analysis model to extract user emotion feature vectors, which include emotion polarity values, emotion intensity values, and intention category labels.
[0060] The multimodal sentiment analysis model includes a text processing module, an image processing module, and a speech processing module.
[0061] The text processing module outputs a 256-dimensional semantic vector. The training data uses the CMU-MOSEI dataset, which includes 25,000 labeled samples. The semantic features are extracted using a BERT-TextCNN hybrid architecture.
[0062] The image processing module uses SlowFast 3D-CNN (β=0.125 version) to extract facial action units (AUs), and its key parameters are as follows:
[0063] Input resolution: 224×224×16 (16 consecutive video frames).
[0064] Output feature: AU activation intensity vector (dimension 52);
[0065] The speech processing module uses Mel spectrogram features combined with Attention-LSTM to extract acoustic features;
[0066] The feature fusion layer uses a gating attention mechanism to dynamically weight multimodal features.
[0067] The core objective of step S2 is to extract user emotion feature vectors from the structured user interaction data obtained in step S1 using a multimodal sentiment analysis model. These emotion feature vectors include emotion polarity values, emotion intensity values, and intent category labels, while ensuring privacy protection. The user emotion feature vectors will help the system understand and predict user behavior, thereby supporting tasks such as personalized interaction recommendations and service optimization for the digital human. To protect user privacy, differential privacy technology and incremental learning strategies are combined to ensure that the user emotion feature vectors are continuously optimized as the data changes, and that user privacy is protected throughout the entire process.
[0068] First, the input data comes from the output of step S1—the user behavior feature matrix. Each line represents a sample of an interaction behavior, containing the following three features: text data ( ), voice data ( ), and video data ( The input matrix The size is , where N is the number of user samples and 3 is the feature dimension of each sample, which is input into the multimodal sentiment analysis model.
[0069] In the process of extracting user emotion feature vectors, the data is classified and labeled with sentiment based on the collected interactive text data and its corresponding feedback data. Using user satisfaction ratings as the primary basis, text samples are labeled as positive emotions (corresponding to very satisfied or satisfied ratings), negative emotions (corresponding to dissatisfied or very dissatisfied ratings), or neutral emotions (corresponding to average ratings), forming a training sample dataset with sentiment labels.
[0070] Through training, the model can extract deep-seated personalized features from user behavior data and generate a personalized emotional feature vector for each user. The vector has dimension d. This vector will become the input to the subsequent recommendation system.
[0071] To ensure both model performance and privacy protection, step S2 introduces an innovative regularization mechanism. A privacy regularization term is designed to protect users' personal information and prevent the leakage of sensitive data during training.
[0072] This regularization term introduces noise into the loss function, making it impossible for an attacker to infer detailed information about an individual user from the model's updates. Specifically, Gaussian noise is injected with each gradient update during training. The intensity of the noise is determined by a hyperparameter. This control ensures a balance between the strength of privacy protection and the accuracy of model fitting.
[0073] Building upon this, the model employs an incremental learning strategy. This means that when new user behavior data arrives, the model is not retrained from scratch, but rather learns by fine-tuning existing model parameters. This incremental update approach allows the model to adapt to changes in user behavior in a timely manner, improving the model's efficiency and flexibility in practical applications.
[0074] Regarding privacy protection, this step ensures that each user's behavioral data is not leaked through differential privacy and noise injection mechanisms. Furthermore, the introduction of privacy regularization terms further enhances the strength of privacy protection, ensuring the security of user data.
[0075] Finally, the output of step S2 is the user's personalized emotion feature vector. Its dimensions are Each row represents a user's personalized characteristics. This personalized emotional feature vector will serve as input for subsequent behavioral decision-making models of digital humans, helping the system accurately understand user needs and preferences, and providing personalized decision-making models and recommendations.
[0076] Through this step S2, the entire system can accurately generate personalized emotional feature vectors for users while protecting user privacy, and ensure that the model continuously optimizes and adapts to changes in user behavior as new data is added.
[0077] S3. Input the emotion feature vector into the digital human behavior decision model. The digital human behavior decision model generates a dynamic response strategy for the three-dimensional digital human through a deep reinforcement learning algorithm. The response strategy includes a set of facial expression parameters, a sequence of body movements, and voice feedback content.
[0078] The core objective of step S3 is to establish a behavioral decision-making model for the digital human and input emotional feature vectors into the model. The behavioral decision-making model is tasked with understanding the user's real-time emotions and intentions, and generating corresponding dynamic response behaviors for the digital human based on preset rules and strategies.
[0079] An innovative decision-making model is used to dynamically evaluate user interaction behavior, ensuring that the system can operate efficiently and accurately.
[0080] Emotional feature vectors are input into the digital human behavior decision model to understand the user's real-time emotions and intentions, and generate corresponding dynamic response behaviors of the digital human according to preset rules and strategies.
[0081] First, the emotion feature vector generated in step S2 is... and the video data in step S1 Merge into a new feature matrix This serves as the input to the equipment capacity prediction model. The combined feature matrix has the following dimensions: ,in These are the dimensions of video data (e.g., facial dynamic features, body behavior features, and scene context features).
[0082] After the data is prepared, the next step is to input the feature matrix. The input is fed into the behavioral decision-making model for training, generating corresponding dynamic response behaviors of the digital human.
[0083] The digital human's dynamic response behavior includes, but is not limited to, facial expression changes, body movements, voice tone adjustments, and dialogue content generation. A dynamic response strategy score is calculated based on user feedback, and the specific formula for calculating this score is as follows:
[0084] ,
[0085] Where S represents the dynamic response strategy score, Re w This represents the text bias coefficient of user feedback, where e is a constant, α1 represents the text interaction weight, and d w This represents the ability score of the multimodal sentiment analysis model on the calculated preliminary text data, α2 represents the voice interaction weight, and d s Re represents the ability score of the multimodal sentiment analysis model on the calculated preliminary speech data.s α represents the voice deviation coefficient of user feedback, α3 represents the video interaction weight, and d v Re represents the capability score of the multimodal sentiment analysis model on the preliminary video data calculated. v This represents the video deviation coefficient as reported by the user.
[0086] The dynamic response strategy score is used to score the user's reaction after the digital human interacts with the user. For example, if the user repeats the previous interaction action or denies the digital human's interaction action, it means that the user does not approve of the interaction action. When the dynamic response strategy score is less than a preset threshold, the user's intent is re-analyzed through a multimodal sentiment analysis model. A better dynamic response strategy is regenerated through the new intent recognition and required to be executed by the digital human.
[0087] Assuming the digital human performs well initially, but as usage increases, the number of text interactions gradually rises, the number of voice interactions increases, and excessive video data interactions lead to a decrease in system efficiency. In this scenario, the computational load increases over time, thus dynamically adjusting the dynamic response strategy scoring threshold to more accurately reflect the device's actual performance.
[0088] To ensure the behavioral decision-making model can adapt to changes in interaction states, an incremental learning strategy is introduced. Specifically, when new interaction data arrives, the model is not trained from scratch, but updated by fine-tuning the existing model. This reduces computational resource consumption while ensuring the model's real-time performance.
[0089] The key to incremental learning is to fine-tune the model using only the new data each time it arrives. With each update, the model updates its weights through mini-batch learning, without retraining the entire model. This allows the device capability model to quickly adapt to changes in device status, improving prediction accuracy.
[0090] The output of this step is the dynamic response strategy score S, which is a... The vector represents the score given by the digital human for each interaction. A higher score indicates a better dynamic response strategy.
[0091] S4. Drive the digital human to perform real-time interaction according to the dynamic response strategy, synchronously monitor user feedback data and establish a reinforcement learning reward function, and dynamically update the network parameters of the behavior decision model.
[0092] The user's sentiment feature vector and dynamic response strategy score are input into a reinforcement learning-based recommendation model, which outputs the optimal response strategy recommendation for each user based on the current dynamic response strategy and historical behavior. The specific process is as follows:
[0093] ,
[0094] Where, r i Let f represent the optimal personalized recommendation for the i-th user, and let f represent the recommendation strategy generated by the reinforcement learning model. Let i represent the i-th user's emotional feature vector. This represents the dynamic response strategy score for the i-th digital person.
[0095] After generating recommendations, the next step is to integrate these recommendations with the device's execution strategy. This strategy must not only be based on the recommended actions but also adjusted in real-time according to the digital human's current interaction state.
[0096] To achieve this goal, the policy gradient method from reinforcement learning is applied to the policy execution. During policy execution, each recommended operation is translated into an actual digital human interaction action. When executing the interaction action, the user's input (such as text, voice, and video / image information) will influence the task assignment. The dynamic response policy score S will serve as one of the inputs to the policy execution, thus ensuring that the system is not overloaded when device performance is insufficient.
[0097] The optimal response strategy recommendation is combined with the system execution strategy. During the strategy execution process, each recommended operation is transformed into actual data human actions or language, and a reinforcement learning reward function is introduced. When the sentiment matching accuracy and user satisfaction are low, the system automatically adjusts the recommendation strategy to select a digital human interaction mode that is more suitable for the current interaction state.
[0098] The core objective of step S4 is to combine the user sentiment feature vector generated in step S2 with the response strategy and dynamic response strategy score obtained in step S3, and use reinforcement learning (RL) methods to generate personalized recommendations, which are then closely integrated with the strategy execution process. The goal of this process is to dynamically adjust recommendations and execute tasks based on the user's behavioral characteristics and the digital human's current interaction capabilities, thereby maximizing system efficiency, improving user experience, and optimizing system usage. The ultimate goal is to achieve intelligent and personalized services from the self-service digital human system through intelligent decision-making.
[0099] Please see Figure 2 As shown, in a second aspect of the present invention, a deep learning-based digital human dynamic interaction system is provided, comprising:
[0100] The data acquisition and processing module is used to acquire real-time interactive data input by the user and perform preprocessing.
[0101] The multimodal sentiment analysis module is used to parse preprocessed interaction data based on a multimodal sentiment analysis model and extract user emotion feature vectors.
[0102] The digital human behavior decision generation module is used to input emotional feature vectors into the digital human behavior decision model. The digital human behavior decision model generates dynamic response strategies for the three-dimensional digital human through deep reinforcement learning algorithms, and combines the optimal response strategy recommendation with the system execution strategy to drive the digital human to perform real-time interaction.
[0103] The feedback update module is used to synchronously monitor user feedback data and establish a reinforcement learning reward function to dynamically update the network parameters of the behavioral decision model.
[0104] The above embodiments are merely descriptions of preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.
Claims
1. A deep learning-based dynamic interaction method for digital humans, characterized in that, Includes the following steps: S1. Acquire real-time interactive data input by the user and preprocess it. The interactive data includes at least text information, voice information and video image information. S2. The preprocessed interaction data is analyzed using a multimodal sentiment analysis model to extract user emotion feature vectors, which include emotion polarity values, emotion intensity values, and intention category labels. S3. Input the emotion feature vector into the digital human behavior decision model. The digital human behavior decision model generates a dynamic response strategy for the three-dimensional digital human through a deep reinforcement learning algorithm. The response strategy includes a set of facial expression parameters, a sequence of body movements, and voice feedback content. S4. Drive the digital human to perform real-time interaction according to the dynamic response strategy, synchronously monitor user feedback data and establish a reinforcement learning reward function, and dynamically update the network parameters of the behavior decision model. In step S3, the emotion feature vector is input into the digital human behavior decision model to understand the user's real-time emotions and intentions, and to generate corresponding dynamic response behaviors of the digital human according to preset rules and strategies. The digital human's dynamic response behaviors include, but are not limited to, facial expression changes, body movements, voice tone adjustments, and dialogue content generation. A dynamic response strategy score is calculated based on user feedback, and the specific formula for calculating the dynamic response strategy score is as follows: ， Where S represents the dynamic response strategy score, Re w This represents the text bias coefficient of user feedback, where e is a constant, α1 represents the text interaction weight, and d w This represents the ability score of the multimodal sentiment analysis model on the calculated preliminary text data, α2 represents the voice interaction weight, and d s Re represents the ability score of the multimodal sentiment analysis model on the calculated preliminary speech data. s α represents the voice deviation coefficient of user feedback, α3 represents the video interaction weight, and d v Re represents the capability score of the multimodal sentiment analysis model on the preliminary video data calculated. v This represents the video deviation coefficient reported by users.
2. The deep learning-based digital human dynamic interaction method according to claim 1, characterized in that, In step S1, the text information includes character-level text data input by the user, including but not limited to chat logs, command statements, and comment content; the voice information includes the user's voice data, including acoustic physical features, prosodic emotional features, and speech semantic features; and the video image information includes the user's video data, including facial dynamic features, body behavior features, and scene context features.
3. The deep learning-based dynamic interaction method for digital humans according to claim 1, characterized in that, In S2, the construction of the multimodal sentiment analysis model includes: The text processing module uses a BERT-TextCNN hybrid architecture to extract semantic features; The image processing module extracts dynamic features of facial micro-expressions using a 3D-CNN network; The speech processing module uses Mel spectrogram features combined with Attention-LSTM to extract acoustic features; The feature fusion layer uses a gating attention mechanism to dynamically weight multimodal features.
4. The deep learning-based dynamic interaction method for digital humans according to claim 1, characterized in that, In step S4, the user's emotion feature vector and dynamic response strategy score are input into the reinforcement learning-based recommendation model, which outputs the optimal response strategy recommendation for each user based on the current dynamic response strategy and historical behavior. The specific process is as follows: Where, r i Let f represent the optimal personalized recommendation for the i-th user, and let f represent the recommendation strategy generated by the reinforcement learning model. This represents the emotional feature vector of the i-th user. This represents the dynamic response strategy score for the i-th digital human.
5. The deep learning-based dynamic interaction method for digital humans according to claim 4, characterized in that, The optimal response strategy recommendation is combined with the system execution strategy. During the strategy execution process, each recommended operation is transformed into actual data human actions or language, and a reinforcement learning reward function is introduced. When the sentiment matching accuracy and user satisfaction are low, the system automatically adjusts the recommendation strategy to select a digital human interaction mode that is more suitable for the current interaction state.
6. A deep learning-based dynamic interaction system for digital humans, characterized in that, Including sequentially connected: The data acquisition and processing module is used to acquire real-time interactive data input by the user and perform preprocessing. The multimodal sentiment analysis module is used to parse preprocessed interaction data based on a multimodal sentiment analysis model and extract user emotion feature vectors. The digital human behavior decision generation module is used to input emotional feature vectors into the digital human behavior decision model. The digital human behavior decision model generates dynamic response strategies for the three-dimensional digital human through deep reinforcement learning algorithms, and combines the optimal response strategy recommendation with the system execution strategy to drive the digital human to perform real-time interaction. The feedback update module is used to synchronously monitor user feedback data and establish a reinforcement learning reward function to dynamically update the network parameters of the behavioral decision model. The emotional feature vector is input into the digital human behavior decision model to understand the user's real-time emotions and intentions, and to generate corresponding dynamic response behaviors of the digital human according to preset rules and strategies. The digital human's dynamic response behaviors include, but are not limited to, facial expression changes, body movements, voice tone adjustments, and dialogue content generation. A dynamic response strategy score is calculated based on user feedback, and the specific formula for calculating the dynamic response strategy score is as follows: ， Where S represents the dynamic response strategy score, Re w This represents the text bias coefficient of user feedback, where e is a constant, α1 represents the text interaction weight, and d w This represents the ability score of the multimodal sentiment analysis model on the calculated preliminary text data, α2 represents the voice interaction weight, and d s Re represents the ability score of the multimodal sentiment analysis model on the calculated preliminary speech data. s α represents the voice deviation coefficient of user feedback, α3 represents the video interaction weight, and d v Re represents the capability score of the multimodal sentiment analysis model on the preliminary video data calculated. v This represents the video deviation coefficient reported by users.