Multi-modal adaptive digital human system based on deep learning and intelligent emotion optimization method thereof

Abstract

The invention discloses a multi-modal adaptive digital human system based on deep learning, and the system comprises a facial expression extraction module which extracts facial information from a video image through employing a convolutional neural network in deep learning; the speaking style modeling module is used for analyzing and understanding the language mode of the user by using a long-term short-term memory network so as to predict the intention and emotional state of the user; and the personalized recommendation module is combined with collaborative filtering and a content-based recommendation algorithm to deeply mine and learn historical data of a user, and the invention relates to the technical field of a multi-modal adaptive digital human system. The multi-modal adaptive digital human system based on deep learning combines rich sensor information and advanced artificial intelligence technology, has high-precision emotion recognition ability and flexible adaptive dialogue ability, can effectively increase the sense of reality and intimacy in the process of human-computer interaction, and improves the accuracy of human-computer interaction. And the use satisfaction of the user is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal adaptive digital human systems, and specifically to a multimodal adaptive digital human system based on deep learning and an emotional intelligence optimization method thereof. Background Art

[0002] With the rapid development of artificial intelligence technology, digital humans are widely used in education, medical care, entertainment and other fields, providing people with various virtual interactions and services. However, traditional digital humans are limited to imitating human images and voices and lack the ability to respond to real emotions, which limits in-depth communication and connection with users. To solve this problem, we propose a multimodal adaptive digital human system based on artificial intelligence. It combines deep learning and affective computing technologies to understand and respond to users' emotional states, while realizing the functions of personalized services and dynamic adjustments. Summary of the invention

[0003] To achieve the above objectives, the present invention is implemented through the following technical solutions: A multimodal adaptive digital human system based on deep learning, comprising:

[0004] The facial expression extraction module uses the convolutional neural network in deep learning to extract facial information from video images;

[0005] A speaking style modeling module that uses long short-term memory networks to analyze and understand users’ language patterns to predict their intentions and emotional states;

[0006] The personalized recommendation module combines collaborative filtering and content-based recommendation algorithms to provide tailored service suggestions through in-depth mining and learning of user historical data;

[0007] The emotion automatic adjustment module uses a deep emotion analysis network to quickly and accurately classify and evaluate the user's emotional expression;

[0008] The response generation decision module uses a generative adversarial network to simulate complex conversation logic in real scenarios and generate diverse response content;

[0009] Real-time feedback update unit, which uses Bayesian optimization method to monitor and control the system training process in real time;

[0010] The multimodal integration module integrates multiple independent sub-functions in the system into a unified whole, thereby improving the system's processing capabilities and simplifying the expansion process.

[0011] Preferably, the workflow of the facial expression extraction module is as follows:

[0012] First, the system preprocesses the input image to improve the data quality;

[0013] Next, face detection is performed using a pre-trained CNN model to locate the facial area in the image;

[0014] Then, the convolutional layer of the convolutional neural network extracts features such as facial muscle movement and skin texture, the activation layer introduces nonlinearity to capture complex expression changes, and the pooling layer reduces the feature dimension to reduce the computational burden;

[0015] Finally, these features are integrated into high-dimensional vectors through a fully connected layer for further processing by the sentiment analysis model to generate the user’s emotional state data, which is then used to optimize the emotional intelligence of the system’s response to make it more in line with the user’s real emotional needs.

[0016] Preferably, the speaking style modeling module first preprocesses the user's speech data, including speech segmentation and feature extraction, and then the long short-term memory network captures the long-term and short-term dependencies in the language by processing these time series data, and learns the user's speaking style and emotional expression. Finally, the output generated by the model is used for further sentiment analysis and personalized response.

[0017] Preferably, the workflow of the personalized recommendation module is as follows:

[0018] First, collaborative filtering algorithms analyze user behaviors and preferences and recommend similar items that other users like based on the similarities between users or items.

[0019] Secondly, the content-based recommendation algorithm recommends items related to the user's past interests based on the user's historical records and the characteristic attributes of the items;

[0020] Finally, the module combines these two recommendation methods and comprehensively considers users' personal interests and behavior patterns to generate accurate recommendation results.

[0021] Preferably, the automatic emotion adjustment module uses the deep neural network in the deep emotion analysis network to analyze the user's voice or text input, extracts emotion-related features and classifies them into different emotion types, and then combines the attention mechanism. The module can focus on the key emotional information in the input and enhance the understanding of the user's emotions. The attention mechanism dynamically adjusts the weights to highlight important emotional clues, so that the digital human can better respond to the user's emotional state.

[0022] Preferably, the generative adversarial network consists of a generator and a discriminator, and generates natural and targeted dialogue responses through adversarial training. On this basis, the module introduces a reinforcement learning algorithm to optimize the dialogue generation strategy. Reinforcement learning continuously interacts with the environment and adjusts the generation strategy according to the reward mechanism, thereby improving the performance and adaptability of the model in different dialogue scenarios.

[0023] Preferably, the real-time feedback update unit evaluates and updates the system performance in each iteration by dynamically adjusting the hyperparameters of the model based on the statistical learning theory of Bayesian optimization.

[0024] Preferably, the multimodal integration module coordinates different functional components to ensure that they can work together seamlessly. In actual operation, the multimodal integration module first collects data and results from each sub-module, and then processes and fuses them through an integration algorithm to form a comprehensive output.

[0025] The method for optimizing the emotional intelligence of a multimodal adaptive digital human system based on deep learning includes the following steps:

[0026] Step 1: Use a convolutional neural network to preprocess the user's facial image into expression and action feature vectors, output them to the next layer of long short-term memory network and Transformer network, further obtain the emotional state information of speech and text, normalize it into a probability distribution representation, and obtain the final prediction result;

[0027] Step 2: Use the attention mechanism at the decision layer to effectively capture key semantic words and determine weight coefficients by fusing information from each sensory channel, thereby assigning different focus points;

[0028] Step 3: Finally, a combination of deep learning models and reinforcement learning algorithms was used to implement a responsive robotic system that could better understand and interact with the user’s intentions.

[0029] Step 4: Based on the above, the Bayesian optimization method is used to update the model parameters in a timely manner to keep the system in the best state;

[0030] Step 5: Finally, the integrated collaborative filtering and content-based recommendation algorithms fully consider factors such as the user's historical preferences and personal interests and hobbies, providing a more accurate and effective service solution.

[0031] The present invention provides a multi-modal adaptive digital human system based on deep learning. It has the following beneficial effects:

[0032] 1. Improve the accuracy of emotional communication by deeply exploring the key factors of the user's emotional state, accurately understand and respond to the needs in different scenarios, and avoid simple mechanical replies and meaningless repetitive conversations.

[0033] 2. Enhance user engagement and introduce a combination of attention mechanism and generative adversarial network in the decision-making stage, which improves the interactivity and coherence between digital humans and users.

[0034] 3. The innovative integrated optimization architecture combines the perception layer and the decision-making layer to form a closed feedback loop, which can automatically adjust parameter configuration according to actual conditions, so that the entire system is always in the optimal state.

[0035] In short, this multimodal adaptive digital human system based on deep learning combines rich sensor information and advanced artificial intelligence technology, has high-precision emotion recognition capabilities and flexible adaptive dialogue capabilities, can effectively increase the sense of reality and intimacy in the human-computer interaction process, and greatly improves user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 Schematic diagram of the framework structure of the multimodal adaptive digital human system based on deep learning of the present invention. DETAILED DESCRIPTION

[0037] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0038] See also Figure 1 The present invention provides a technical solution: a multimodal adaptive digital human system based on deep learning, comprising a multimodal perception module, a multimodal fusion module and a multimodal decision module, wherein: the multimodal perception module uses a convolutional neural network as a backbone layer to perform face detection and face key point positioning respectively, and uses the human posture estimation obtained by training the YOLOv4 model to obtain the spatial position information of various parts of the human body, and then uses ResNet-101 as a backbone model to convert multi-channel images of different resolutions into RGB images of the same resolution and compress them into a fixed size;

[0039] The multimodal fusion module includes an encoder composed of a convolutional neural network and a memory decoder composed of a long short-term memory network;

[0040] First, the ResNet-101 model is used to extract the feature representation of each component in the face area, such as the feature vectors of the eyes and mouth. These features are then fed into the long short-term memory network together with the character's status information and timestamp. After the memory decoder is warmed up, the memory unit status is continuously updated in the memory decoder based on the annotated database, ultimately generating accurate speech output results.

[0041] In addition, text information will be added synchronously during the speech output process, so it is necessary to design a corresponding text conversion function to unify text and speech into a unified dimension, so as to generate a high-quality, semantically rich text sequence, which is then decoded by the recurrent neural network of the long short-term memory network to form a complete spoken language stream, which is directly displayed to the end user in the speech synthesis module;

[0042] Finally, the multimodal decision module uses the generative adversarial network method, which uses the user's emotional state feedback to model the selectivity and diversity of the target response, and selects the appropriate response based on the current speaker's emotional type. In addition, reinforcement learning is used to improve the stability and robustness of the model. If the current target response is recognized by the user, a reward is given, otherwise points are deducted according to the penalty rules. This continuously improves the quality of the response generated by the model, and takes into account the influence of multiple factors to avoid the deviation problem caused by a single model, thus achieving adaptive effects in complex environments.

[0043] The multimodal perception module includes face detection (FaceDetection), face key point positioning (FaceKeypointDetectron), human posture estimation (PoseEstimation), and image segmentation (ImageSegmentation);

[0044] In the specific implementation, the original image is used as input to detect the position of the face and head contours, and estimate the coordinates of the key points corresponding to each pixel. Then we use the DETRON model in Cascade RCNN to predict the relative positions of 29 human joints. Finally, all key points and corresponding labels are saved and stored for subsequent steps. The technologies involved in this process include OpenCV, TensorFlow, Pytorch and other toolkits;

[0045] The multimodal fusion module includes a multi-layer convolutional neural network architecture, a long short-term memory network (LSTM), an attention mechanism and text conversion (TextConversion); wherein the multi-layer convolutional neural network architecture is a relatively general model with good expression ability, which can effectively mine features from source data and perform classification operations. Since this module is mainly responsible for the fusion of multimodal data and the task of text-to-speech, its flexibility and scalability need to be paid attention to when building the model. In addition, the long short-term memory network is a powerful recursive neural network model suitable for sequence data, which can handle nonlinear relationships in time and space well, and has high accuracy and stability. The attention mechanism is an important computing technique that can allocate more weights to pay attention to certain specific details or feature parts according to different needs, so that the output results are more in line with the actual situation. After the entire system is built, a large number of experimental verifications are required to ensure that the model performs well in practical applications, so as to ensure that users get the best experience during use.

[0046] The multimodal decision-making module includes functional components such as generative adversarial network (GAN), reinforcement learning, personalized service and visual interface; GAN is a method that uses two mutually exclusive probability distributions to characterize the difference measurement between real data and synthetic samples. It can effectively solve the problem that large-scale high-dimensional data is difficult to accurately describe, and has been widely used in many fields. Reinforcement learning is an artificial intelligence method that imitates human thinking patterns and learning methods. It can achieve the purpose of maximizing long-term benefits by exploring the environment and taking corresponding actions. The purpose of personalized service is to better meet the needs of specific groups of people, such as developing specially designed applications or service processes for a certain customer group. Such solutions have become increasingly popular and have very broad market prospects.

[0047] Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field and related fields without creative work should fall within the scope of protection of the present invention. The structures, devices and operating methods not specifically described and explained in the present invention are implemented according to the conventional means in the field unless otherwise specified and limited.

Claims

1. A multimodal adaptive digital human system based on deep learning, characterized by: include: The facial expression extraction module uses the convolutional neural network in deep learning to extract facial information from video images; A speaking style modeling module that uses long short-term memory networks to analyze and understand users’ language patterns to predict their intentions and emotional states; The personalized recommendation module combines collaborative filtering and content-based recommendation algorithms to provide tailored service suggestions through in-depth mining and learning of user historical data; The emotion automatic adjustment module uses a deep emotion analysis network to quickly and accurately classify and evaluate the user's emotional expression; The response generation decision module uses a generative adversarial network to simulate complex conversation logic in real scenarios and generate diverse response content; Real-time feedback update unit, which uses Bayesian optimization method to monitor and control the system training process in real time; The multimodal integration module integrates multiple independent sub-functions in the system into a unified whole, thereby improving the system's processing capabilities and simplifying the expansion process.

2. The multimodal adaptive digital human system based on deep learning according to claim 1, characterized in that: The workflow of the facial expression extraction module is as follows: First, the system preprocesses the input image to improve the data quality; Next, face detection is performed using a pre-trained CNN model to locate the facial area in the image; Then, the convolutional layer of the convolutional neural network extracts features such as facial muscle movement and skin texture, the activation layer introduces nonlinearity to capture complex expression changes, and the pooling layer reduces the feature dimension to reduce the computational burden; Finally, these features are integrated into high-dimensional vectors through a fully connected layer for further processing by the sentiment analysis model to generate the user’s emotional state data, which is then used to optimize the emotional intelligence of the system’s response to make it more in line with the user’s real emotional needs.

3. The multimodal adaptive digital human system based on deep learning according to claim 1, characterized in that: The speaking style modeling module first preprocesses the user's speech data, including speech segmentation and feature extraction. Then, the long short-term memory network captures the long-term and short-term dependencies in the language by processing these time series data, and learns the user's speaking style and emotional expression. Finally, the output generated by the model is used for further sentiment analysis and personalized response.

4. The multimodal adaptive digital human system based on deep learning according to claim 1, characterized in that: The workflow of the personalized recommendation module is as follows: First, collaborative filtering algorithms analyze user behaviors and preferences and recommend similar items that other users like based on the similarities between users or items. Secondly, the content-based recommendation algorithm recommends items related to the user's past interests based on the user's historical records and the characteristic attributes of the items; Finally, the module combines these two recommendation methods and comprehensively considers users' personal interests and behavior patterns to generate accurate recommendation results.

5. The multimodal adaptive digital human system based on deep learning according to claim 1, characterized in that: The automatic emotion adjustment module uses the deep neural network in the deep emotion analysis network to analyze the user's voice or text input, extract emotion-related features and classify them into different emotion types, and then combines the attention mechanism. The module can focus on the key emotional information in the input and enhance the understanding of the user's emotions. The attention mechanism dynamically adjusts the weights to highlight important emotional clues, so that the digital human can better respond to the user's emotional state.

6. The multimodal adaptive digital human system based on deep learning according to claim 1, characterized in that: The generative adversarial network consists of a generator and a discriminator, and generates natural and targeted dialogue responses through adversarial training. On this basis, the module introduces a reinforcement learning algorithm to optimize the dialogue generation strategy. Reinforcement learning continuously interacts with the environment and adjusts the generation strategy according to the reward mechanism, thereby improving the performance and adaptability of the model in different dialogue scenarios.

7. The multimodal adaptive digital human system based on deep learning according to claim 1, characterized in that: The real-time feedback update unit evaluates and updates the system performance in each iteration by dynamically adjusting the hyperparameters of the model based on the statistical learning theory of Bayesian optimization.

8. The multimodal adaptive digital human system based on deep learning according to claim 1, characterized in that: The multimodal integration module coordinates different functional components to ensure that they can work together seamlessly. In actual operation, the multimodal integration module first collects data and results from each sub-module, and then processes and fuses them through an integration algorithm to form a comprehensive output.

9. The emotional intelligence optimization method of a multimodal adaptive digital human system based on deep learning according to claim 1, characterized in that: The following steps are involved: Step 1: Use a convolutional neural network to preprocess the user's facial image into expression and action feature vectors, output them to the next layer of long short-term memory network and Transformer network, further obtain the emotional state information of speech and text, normalize it into a probability distribution representation, and obtain the final prediction result; Step 2: Use the attention mechanism at the decision layer to effectively capture key semantic words and determine weight coefficients by fusing information from each sensory channel, thereby assigning different focus points; Step 3: Finally, a combination of deep learning models and reinforcement learning algorithms was used to implement a responsive robotic system that could better understand and interact with the user’s intentions. Step 4: Based on the above, the Bayesian optimization method is used to update the model parameters in a timely manner to keep the system in the best state; Step 5: Finally, the integrated collaborative filtering and content-based recommendation algorithms fully consider factors such as the user's historical preferences and personal interests and hobbies, providing a more accurate and effective service solution.

Images