Digital human emotion perception and dynamic response method, system and device and storage medium

Abstract

The embodiment of the invention discloses an emotion perception and dynamic response method, system and device of a digital human, and a storage medium. According to the embodiment of the invention, accurate emotion perception can be realized, the recognition precision of multi-dimensional emotion information such as voice, expression and text of a customer is improved, and a complex emotion state is accurately captured; according to the emotional intensity, the cultural background and the dialogue context of the customer, anthropomorphic voices, expressions and solutions are generated, and the interaction affinity is enhanced; personalized and highly-targeted emotional services are provided by combining a hotel industry knowledge graph and a high-frequency service scene; and secure storage and processing of customer emotion data are ensured, and requirements of privacy protection laws and regulations are met.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of human-computer interaction technology, and specifically to a method, system, device and storage medium for digital human emotion perception and dynamic response. Background Art

[0002] In hotel service scenarios, traditional human-computer interaction systems mainly rely on rule-based or simple natural language processing (NLP) dialogue systems. These systems usually use keyword matching or predefined dialogue processes to implement customer service functions such as room reservations, information inquiries, and FAQs. In recent years, with the development of artificial intelligence technology, some hotels have begun to introduce voice assistants and digital human images to enhance customer experience. However, existing technologies have significant limitations in emotional perception and response. The specific problems and shortcomings are as follows:

[0003] 1) Insufficient emotional perception: Existing systems mainly rely on unimodal analysis of text or voice, and lack the ability to perceive customer emotions in multiple dimensions. For example, judging customer emotions only by voice intonation or text keywords cannot capture non-verbal clues such as facial expressions and body language;

[0004] 2) Weak context understanding and emotion association capabilities: It is difficult to maintain context consistency in multiple rounds of conversations, and it is impossible to dynamically associate the customer's emotional state with the conversation history and scene background. For example, when a customer complains about room problems multiple times, the system may repeatedly provide the same solution, causing the customer's emotions to escalate;

[0005] 3) Rigid emotional response mechanism: Responses are usually based on fixed rules or simple templates, lacking the ability to dynamically adjust in anthropomorphic ways. For example, when a customer expresses dissatisfaction, the system may only provide a standardized apology statement, and cannot convey empathy through voice tone, facial expressions, etc.

[0006] At the same time, single-modal emotion recognition has certain limitations. Speech emotion recognition (SER) is easily disturbed by environmental noise, and text emotion analysis has difficulty processing implicit emotional expressions in high-context cultures. Traditional dialogue management systems (DMS) lack the ability to remember long-term conversation contexts, and it is difficult to capture emotional change trends after losing context. Existing technologies do not deeply combine emotion analysis with hotel scene knowledge (such as customer preferences and service history), resulting in a lack of pertinence in emotional responses and a problem of emotional-scene disconnection. Existing technologies usually only simply replace explicit sensitive information (such as phone numbers), do not protect implicit sensitive information (such as emotional status), and do not desensitize data thoroughly enough. Summary of the invention

[0007] To this end, the embodiments of the present invention provide a digital human emotion perception and dynamic response method, system, device and storage medium to solve the technical problems of insufficient emotion perception ability, rigid response mechanism and poor cross-cultural adaptability in the prior art.

[0008] In order to achieve the above purpose, the embodiment of the present invention provides the following technical solutions:

[0009] According to a first aspect of an embodiment of the present invention, a method for emotional perception and dynamic response of a digital human is provided, and the method is applied to a hotel digital human, and comprises:

[0010] S1. Receive user interaction information and collect multimodal data, extract multiple modal features from the multimodal data, and use an attention mechanism to perform weighted fusion on each modal feature to generate a comprehensive emotion feature vector;

[0011] S2. Construct a deep learning model and train the deep learning model using preset emotional state data to generate an emotional state classification model;

[0012] S3, using the emotional state classification model and the comprehensive emotional feature vector to judge the emotional intensity, when the emotional intensity is medium or low intensity, combining the conversation history with the hotel scene knowledge graph to generate an emotional response strategy;

[0013] S4. Generate anthropomorphic behavior using the emotional response strategy, output the anthropomorphic behavior and collect interactive feedback from users, detect emotional changes and dynamically adjust the response strategy, store the interactive data and generate a service report.

[0014] Furthermore, the multimodal data includes the user's interactive voice and the user's facial expression data and body language.

[0015] Furthermore, multiple modal features are extracted from the multimodal data and weighted fusion of each modal feature is performed using an attention mechanism to generate a comprehensive emotion feature vector, including:

[0016] Construct a multimodal emotion perception network by integrating speech emotion recognition, facial expression analysis, text emotion analysis and body language capture;

[0017] Use deep convolutional neural network model to extract features of speech spectrum and combine it with long short-term memory network to capture temporal emotional changes;

[0018] Obtain facial expression data and capture the user's facial emotional changes through micro-expression recognition algorithms;

[0019] Use language models to conduct conversation analysis and generate language emotion changes;

[0020] The speech spectrum features, temporal emotion changes, facial emotion changes and language emotion changes are weightedly fused through the attention mechanism to generate a comprehensive emotion feature vector.

[0021] Furthermore, the emotional state classification model and the comprehensive emotional feature vector are used to judge the emotional intensity. When the emotional intensity is medium or low, an emotional response strategy is generated by combining the conversation history and the hotel scene knowledge graph, including:

[0022] Build a conversation context modeling framework based on graph neural network to dynamically associate the user's emotional state with the conversation history and scene background;

[0023] Storing multiple rounds of conversation information through memory-enhanced networks to track emotional states over a long period of time;

[0024] Acquire a hotel scene knowledge graph and use the hotel scene knowledge graph to enhance scene knowledge and generate an emotional response strategy;

[0025] When judging the emotion intensity, the emotion state needs to be classified by an emotion intensity quantification model, and the emotion state of the emotion intensity quantification model is divided into a first preset number of major categories and a second preset number of subcategories.

[0026] Furthermore, the emotional response strategy is used to generate anthropomorphic behavior, output the anthropomorphic behavior and collect interactive feedback from users, detect emotional changes and dynamically adjust the response strategy, including:

[0027] Develop an emotion-driven digital human behavior generation engine to dynamically adjust the digital human's voice intonation, facial expressions, and body movements, including:

[0028] synthesizing speech and adjusting speech synthesis parameters based on the emotion intensity, wherein the speech synthesis parameters include pitch and speech rate;

[0029] Generate facial expressions and body movements that match emotional states through the emotion-action mapping rule base, and render the robot's facial expressions according to the intensity of emotions;

[0030] Dynamically update model parameters through gradient descent method, combine real-time emotional state monitoring algorithm with online learning mechanism, and adjust response strategy according to users' instant interactive feedback data;

[0031] The instant interaction feedback data includes voice interruptions and expression changes.

[0032] Furthermore, the method further comprises:

[0033] The emotion state classification model and the comprehensive emotion feature vector are used to judge the emotion intensity. When the emotion intensity is high, an emergency response mechanism is triggered to soothe the user's emotions.

[0034] Furthermore, storing the interaction data includes:

[0035] Desensitize interactive data in real time and separate emotional data from business data;

[0036] The emotional data and the business data are stored separately in the sub-database, the data is encrypted using a preset encryption algorithm and all data access behaviors are recorded;

[0037] The desensitizing operation is to filter sensitive information dynamically based on role permissions.

[0038] According to a second aspect of an embodiment of the present invention, a digital human emotion perception and dynamic response system is provided, the system comprising:

[0039] A collection module, used to receive user interaction information and collect multimodal data, extract multiple modal features from the multimodal data and use an attention mechanism to perform weighted fusion on each modal feature to generate a comprehensive emotion feature vector;

[0040] A model building module, used to build a deep learning model and train the deep learning model using preset emotional state data to generate an emotional state classification model;

[0041] An emotional response strategy generation module is used to use the emotional state classification model and the comprehensive emotional feature vector to judge the emotional intensity, and when the emotional intensity is medium or low, generate an emotional response strategy by combining the conversation history and the hotel scene knowledge graph;

[0042] The interactive feedback module is used to generate anthropomorphic behavior using the emotional response strategy, output the anthropomorphic behavior and collect interactive feedback from users, detect emotional changes and dynamically adjust the response strategy, store the interactive data and generate a service report.

[0043] According to a third aspect of an embodiment of the present invention, there is provided a digital human emotion perception and dynamic response device, the device comprising: a processor and a memory;

[0044] The memory is used to store one or more program instructions;

[0045] The processor is used to run one or more program instructions to execute the steps of a digital human emotion perception and dynamic response method as described in any one of the above items.

[0046] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of a method for emotional perception and dynamic response of a digital human as described in any of the above items are implemented.

[0047] The embodiments of the present invention have the following advantages:

[0048] An embodiment of the present invention includes: S1, receiving user interaction information and collecting multimodal data, extracting multiple modal features from the multimodal data and using the attention mechanism to weightedly fuse each modal feature to generate a comprehensive emotional feature vector; S2, building a deep learning model and using preset emotional state data to train the deep learning model to generate an emotional state classification model; S3, using the emotional state classification model and the comprehensive emotional feature vector to judge the emotional intensity, when the emotional intensity is medium or low intensity, combining the conversation history with the hotel scene knowledge graph to generate an emotional response strategy; S4, using the emotional response strategy to generate anthropomorphic behavior, outputting the anthropomorphic behavior and collecting user interaction feedback, detecting emotional changes to dynamically adjust the response strategy, storing the interaction data and generating a service report. The embodiments of the present invention can achieve precise emotion perception, improve the recognition accuracy of multi-dimensional emotional information such as customer voice, expression, text, etc., and accurately capture complex emotional states; generate anthropomorphic voice, expression and solutions according to the customer's emotional intensity, cultural background and conversation context, and enhance interactive affinity; combine the hotel industry knowledge graph with high-frequency service scenarios to provide personalized and highly targeted emotional services; ensure the safe storage and processing of customer emotional data to meet the requirements of privacy protection regulations. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0050] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0051] Figure 1 A schematic diagram of the logical structure of a digital human emotion perception and dynamic response system provided by an embodiment of the present invention;

[0052] Figure 2 A schematic diagram of a flow chart of a digital human emotion perception and dynamic response method provided by an embodiment of the present invention;

[0053] Figure 3 A schematic diagram of the overall architecture of a digital human emotion perception and dynamic response method provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The following is a description of the implementation of the present invention by specific embodiments. People familiar with the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0055] In hotel service scenarios, traditional human-computer interaction systems mainly rely on rule-based or simple natural language processing (NLP) dialogue systems. These systems usually use keyword matching or predefined dialogue processes to implement customer service functions such as room reservations, information inquiries, and FAQs. In recent years, with the development of artificial intelligence technology, some hotels have begun to introduce voice assistants and digital human images to enhance customer experience. However, existing technologies have significant limitations in emotional perception and response. The specific problems and shortcomings are as follows:

[0056] 1) Insufficient emotional perception: Existing systems mainly rely on unimodal analysis of text or voice, and lack the ability to perceive customer emotions in multiple dimensions. For example, judging customer emotions only by voice intonation or text keywords cannot capture non-verbal clues such as facial expressions and body language;

[0057] 2) Weak context understanding and emotion association capabilities: It is difficult to maintain context consistency in multiple rounds of conversations, and it is impossible to dynamically associate the customer's emotional state with the conversation history and scene background. For example, when a customer complains about room problems multiple times, the system may repeatedly provide the same solution, causing the customer's emotions to escalate;

[0058] 3) Rigid emotional response mechanism: Responses are usually based on fixed rules or simple templates, lacking the ability to dynamically adjust in anthropomorphic ways. For example, when a customer expresses dissatisfaction, the system may only provide a standardized apology statement, and cannot convey empathy through voice tone, facial expressions, etc.

[0059] At the same time, single-modal emotion recognition has certain limitations. Speech emotion recognition (SER) is easily disturbed by environmental noise, and text emotion analysis has difficulty processing implicit emotional expressions in high-context cultures. Traditional dialogue management systems (DMS) lack the ability to remember long-term conversation contexts, and it is difficult to capture emotional change trends after losing context. Existing technologies do not deeply combine emotion analysis with hotel scene knowledge (such as customer preferences and service history), resulting in a lack of pertinence in emotional responses and a problem of emotional-scene disconnection. Existing technologies usually only simply replace explicit sensitive information (such as phone numbers), do not protect implicit sensitive information (such as emotional status), and do not desensitize data thoroughly enough.

[0060] In order to solve the above-mentioned technical problems of insufficient emotional perception, rigid response mechanism, and poor cross-cultural adaptability.

[0061] refer to Figure 1 The embodiment of the present invention discloses a digital human emotion perception and dynamic response system, which includes: a collection module 1; a model building module 2; an emotional response strategy generation module 3; and an interactive feedback module 4.

[0062] The technical scope of the embodiment of the present invention covers: a multimodal emotion computing model based on deep learning, a semantic-emotion association analysis algorithm in a cross-cultural context, a digital human behavior generation engine driven by real-time emotion feedback, and a privacy-protected dialogue data processing architecture. The embodiment of the present invention is directly applied to hotel customer service scenarios, integrating speech emotion recognition (SER), micro-expression capture, dialogue text emotion tendency analysis, and context-aware multi-round dialogue management technology, and innovatively integrating the hotel industry knowledge graph and the cross-cultural emotion expression rule library. It is suitable for the emotional interaction of intelligent digital humans in hotel front desk consultation, room service, complaint handling and other links, and realizes a full-link closed loop from emotion recognition (such as anger, satisfaction, anxiety) to anthropomorphic response (voice intonation adjustment, facial expression rendering, solution recommendation), meeting the credibility and affinity requirements of the emotional human-computer interaction system in the ISO 9241-210 standard.

[0063] Corresponding to the above-disclosed digital human emotion perception and dynamic response system, the embodiment of the present invention further discloses a digital human emotion perception and dynamic response method. The following describes in detail the digital human emotion perception and dynamic response method disclosed in the embodiment of the present invention in combination with the above-described digital human emotion perception and dynamic response system.

[0064] The embodiment of the present invention aims to solve the problem of insufficient emotional perception and response capabilities of the digital human interaction system in hotel service scenarios. By innovatively integrating multimodal emotional computing, context-aware technology, and anthropomorphic behavior generation engines, the technical problems of insufficient emotional perception capabilities and rigid response mechanisms in the existing technologies are improved. The invention is committed to creating an intelligent, emotional, safe and reliable hotel digital human perception and dynamic response system, which significantly improves customer service experience and hotel operation efficiency.

[0065] refer to Figures 2 to 3 The present invention discloses a method for emotional perception and dynamic response of a digital human, which is applied to a hotel digital human and comprises:

[0066] S1. Receive user interaction information and collect multimodal data, extract multiple modal features from the multimodal data, and use an attention mechanism to perform weighted fusion on each modal feature to generate a comprehensive emotion feature vector;

[0067] S2. Construct a deep learning model and train the deep learning model using preset emotional state data to generate an emotional state classification model;

[0068] S3, using the emotional state classification model and the comprehensive emotional feature vector to judge the emotional intensity, when the emotional intensity is medium or low intensity, combining the conversation history with the hotel scene knowledge graph to generate an emotional response strategy;

[0069] S4. Generate anthropomorphic behavior using the emotional response strategy, output the anthropomorphic behavior and collect interactive feedback from users, detect emotional changes and dynamically adjust the response strategy, store the interactive data and generate a service report.

[0070] Furthermore, the multimodal data includes the user's interactive voice and the user's facial expression data and body language.

[0071] Furthermore, multiple modal features are extracted from the multimodal data and each modal feature is weightedly fused using an attention mechanism to generate a comprehensive emotion feature vector, including: building a multimodal emotion perception network by integrating speech emotion recognition, facial expression analysis, text emotion analysis and body language capture; using a deep convolutional neural network model to extract features of the speech spectrum and combining it with a long short-term memory network to capture temporal emotion changes; acquiring facial expression data and capturing the user's facial emotion changes through a micro-expression recognition algorithm; using a language model to perform dialogue analysis and generate language emotion changes; weightedly fusion of speech spectrum features, temporal emotion changes, facial emotion changes and language emotion changes through an attention mechanism to generate a comprehensive emotion feature vector.

[0072] Integrate speech emotion recognition (SER), facial expression analysis, text sentiment analysis and body language capture technology to build a multimodal emotion perception network.

[0073] Speech emotion recognition: Speech spectrum feature extraction based on deep convolutional neural network (CNN) is used, combined with long short-term memory network (LSTM) to capture temporal emotional changes.

[0074] Facial expression analysis: Capture subtle emotional changes of customers through facial action unit (AU) detection and micro-expression recognition algorithms.

[0075] Text Sentiment Analysis: Context-aware sentiment classification based on pre-trained language models (such as BERT), supporting implicit sentiment parsing in high-context cultures.

[0076] The embodiment of the present invention proposes a multimodal feature fusion mechanism and dynamically weights each modal feature through an attention mechanism to improve the accuracy of emotion recognition (>92% F1 value). It introduces an emotion intensity quantification model, divides the emotion state into 9 categories and 32 subcategories, and supports more fine-grained emotion classification.

[0077] Furthermore, the emotion state classification model and the comprehensive emotion feature vector are used to judge the emotion intensity. When the emotion intensity is medium or low, the conversation history and the hotel scene knowledge graph are combined to generate an emotional response strategy, including: constructing a conversation context modeling framework based on a graph neural network to dynamically associate the user's emotion state with the conversation history and scene background; storing multiple rounds of conversation information through a memory enhancement network to track the emotion state over a long period of time; obtaining a hotel scene knowledge graph and using the hotel scene knowledge graph to enhance scene knowledge and generate an emotional response strategy.

[0078] Build a conversation context modeling framework based on graph neural network (GNN) to dynamically associate the customer's emotional state with the conversation history and scene background.

[0079] Dialogue context modeling: Multi-round dialogue information is stored through the Memory Network, supporting long-term emotional state tracking.

[0080] Scenario knowledge enhancement: Combined with the hotel industry knowledge graph (such as customer preferences and service history), emotional service recommendations are provided.

[0081] Propose an emotion-scenario association rule engine to dynamically adjust the emotion response strategy. For example, for customers who have made multiple complaints, the upgrade service process is automatically triggered.

[0082] The emotion attenuation model is introduced to simulate the characteristics of human emotions changing over time and avoid delayed emotional response.

[0083] Among them, when judging the intensity of emotion, it is also necessary to classify the emotional state through the emotional intensity quantification model. The emotional state of the emotional intensity quantification model is divided into a first preset number of major categories and a second preset number of subcategories. The first preset number is 9 and the second preset number is 32.

[0084] Furthermore, the emotional response strategy is used to generate anthropomorphic behavior, output the anthropomorphic behavior and collect user interaction feedback, and detect emotional changes to dynamically adjust the response strategy, including: developing an emotion-driven digital human behavior generation engine, dynamically adjusting the digital human's voice intonation, facial expressions and body movements, specifically including: synthesizing speech and adjusting speech synthesis parameters based on emotion intensity, the speech synthesis parameters including pitch and speaking rate; generating facial expressions and body movements matching the emotional state through an emotion-action mapping rule library, and rendering the robot's facial expressions according to the emotion intensity; dynamically updating model parameters through the gradient descent method, combining the real-time emotional state monitoring algorithm with the online learning mechanism, and adjusting the response strategy according to the user's instant interactive feedback data.

[0085] The instant interaction feedback data includes voice interruptions and expression changes.

[0086] Develop an emotion-driven digital human behavior generation engine to dynamically adjust voice intonation, facial expressions, and body movements:

[0087] Speech synthesis: Adjust speech synthesis parameters (such as pitch and speaking speed) based on the intensity of emotion to convey empathy.

[0088] Expression and action generation: Generate facial expressions and body movements that match emotional states through the emotion-action mapping rule library.

[0089] Propose an emotional response diversity mechanism to generate differentiated response content based on emotional intensity and cultural background. Introduce a real-time emotional feedback mechanism to dynamically adjust the response strategy based on customer expressions and voice changes.

[0090] Furthermore, the method also includes: using the emotional state classification model and the comprehensive emotional feature vector to judge the emotional intensity, and when the emotional intensity is high intensity, triggering an emergency response mechanism and soothing the user's emotions.

[0091] Furthermore, storing the interaction data includes: desensitizing the interaction data in real time and separating the emotional data from the business data; isolating and storing the emotional data and the business data in separate databases, encrypting the data using a preset encryption algorithm and recording all data access behaviors.

[0092] The desensitizing operation is to filter sensitive information dynamically based on role permissions.

[0093] Design a layered data desensitization and encryption storage architecture to ensure the security and compliance of customer sentiment data. Propose sentiment data anonymization processing technology to ensure that customer privacy is not leaked during sentiment analysis. Introduce a data access audit mechanism to record all data access behaviors to meet the requirements of privacy protection regulations such as GDPR.

[0094] Real-time data desensitization: Dynamically filter sensitive information (such as identity information, payment information) based on role permissions.

[0095] Sub-database storage mechanism: emotional data and business data are stored separately, and the AES-256 encryption algorithm is used to protect data security.

[0096] The embodiment of the present invention also has a cross-cultural support function, adapts to the differences in emotional expression in multi-language and multi-cultural backgrounds, and improves customer satisfaction in global hotel service scenarios.

[0097] Build a cross-cultural emotion expression rule library to support emotion interaction in multi-language and multi-cultural contexts. Propose a cultural context perception mechanism to dynamically adjust the emotion recognition threshold. For example, improve the sensitivity of implicit emotion recognition for East Asian customers. Introduce cross-cultural emotion mapping rules to ensure that the emotional response is consistent with the customer's cultural background.

[0098] Compensation for cultural differences: Adjust emotion recognition and response strategies based on the emotional expression habits of customers from different cultural backgrounds.

[0099] Multi-language support: Integrates multi-language sentiment analysis models to support sentiment interactions in mainstream languages ​​such as English, Chinese, and Japanese.

[0100] The embodiments of the present invention have the following advantages: high-precision emotion perception: through multimodal fusion and context association, the emotion recognition accuracy and scene adaptation ability are significantly improved; humanized interactive experience: dynamic generation of emotional voice, expression and action to enhance customer interaction experience; security and compliance: using layered data desensitization and encryption storage technology to ensure customer privacy and security; cross-cultural support: adapting to multiple languages ​​and cultural backgrounds to meet the needs of global hotel services.

[0101] Through the above technical solutions, the present invention realizes a full-link closed loop from emotional perception to anthropomorphic response, providing an intelligent, emotional, safe and reliable digital human interaction solution for hotel service scenarios.

[0102] The embodiments of the present invention have the following advantages:

[0103] 1) The integration of multimodal emotion perception is enhanced, significantly improving the accuracy and scene adaptability of emotion recognition:

[0104] Using multimodal data fusion technology, the user's facial expressions, voice intonation, body movements (such as arm posture) and dialogue text are collected synchronously through cameras, microphones and text input, and combined with deep learning models (such as LSTM, attention mechanism) for joint training. For example, micro-expressions are analyzed through the facial action coding system (FACS), combined with the fundamental frequency (F0) of the voice and the semantics of the text to form a comprehensive emotional judgment. The problem that single-modal data is easily disturbed by noise is solved, especially in hotel service scenarios, and customers' implicit emotions (such as dissatisfaction in a neutral state) can be more accurately identified.

[0105] 2) Dynamic real-time emotional response optimization to achieve low-latency personalized feedback and improve the naturalness of interaction:

[0106] The real-time emotional state monitoring algorithm (REM) is introduced to dynamically update the model parameters through the gradient descent method, and combined with the online learning mechanism, the response strategy is adjusted according to the real-time data of user interaction (such as voice interruption and expression change). For example, when the customer expresses his needs, the system optimizes the feedback content by calculating the difference in emotional labels in real time (such as the loss function in formula (1)); the response time is shortened to milliseconds, and it can adapt to sudden emotional changes (such as the customer's change from calm to anxious), providing more scene-appropriate service suggestions.

[0107] 3) Improve empathy based on common sense knowledge graphs to generate more logical and context-relevant conversation content:

[0108] The open-domain dialogue model of the prior art relies on large-scale pre-trained language models (such as GPT and PLATO), but lacks common sense reasoning capabilities and is prone to logical contradictions (such as "cows go to college"). The embodiment of the present invention constructs a social common sense knowledge graph, including event causal relationships (such as "thirst → buy water"), social intentions (such as "propose requires a ring") and other triplets, and integrates the dialogue history and graph knowledge through the BART model. For example, when a customer mentions "birthday gift", the system infers the "expectation" emotion based on the graph and generates an empathetic response of "the other party will definitely like it", reducing responses that violate common sense and enhancing the coherence of the dialogue and user trust.

[0109] 4) Generate service improvement suggestions for multiple scenarios and provide feasible business optimization solutions, which go beyond basic interactive functions:

[0110] Traditional hotel feedback systems rely on manual analysis of questionnaire data, which is inefficient and highly subjective. The embodiment of the present invention improves the coefficient calculation model to quantify the preferences of customers of different age groups for dishes and service experience, and generates improvement suggestions in combination with emotion recognition results (such as high frequency of "neutral" expressions). For example, if it is recognized that young customers frequently "frown" at a certain dish, the system automatically calculates the "negative experience coefficient" of the dish and recommends adjusting the taste or plating. Transforming emotional data into executable operational strategies helps hotels improve customer satisfaction and repurchase rates.

[0111] In addition, an embodiment of the present invention further provides a digital human emotion perception and dynamic response device, the device comprising: a processor and a memory; the memory is used to store one or more program instructions; the processor is used to run one or more program instructions to execute the steps of a digital human emotion perception and dynamic response method as described in any of the above items.

[0112] In addition, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method for emotional perception and dynamic response of a digital human as described in any of the above items are implemented.

[0113] In the embodiment of the present invention, the processor may be an integrated circuit chip having the signal processing capability. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0114] The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware.

[0115] The storage medium may be a memory, which may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.

[0116] Among them, the non-volatile memory can be a read-only memory (ROM), a programmable ROM (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory.

[0117] The volatile memory may be a random access memory (RAM) which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM).

[0118] The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.

[0119] Those skilled in the art will appreciate that in one or more of the above examples, the functions described in the present invention can be implemented using a combination of hardware and software. When software is used, the corresponding functions can be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any media that facilitates the transmission of computer programs from one place to another. Storage media can be any available media that can be accessed by general or special-purpose computers.

[0120] Although the present invention has been described in detail above by general description and specific embodiments, it is obvious to those skilled in the art that some modifications or improvements can be made to the present invention. Therefore, these modifications or improvements made without departing from the spirit of the present invention all belong to the scope of protection claimed by the present invention.

Claims

1. A digital human emotion perception and dynamic response method, characterized in that: The method is applied to a hotel digital human, and comprises: S1. Receive user interaction information and collect multimodal data, extract multiple modal features from the multimodal data, and use an attention mechanism to perform weighted fusion on each modal feature to generate a comprehensive emotion feature vector; S2. Construct a deep learning model and train the deep learning model using preset emotional state data to generate an emotional state classification model; S3, using the emotional state classification model and the comprehensive emotional feature vector to judge the emotional intensity, when the emotional intensity is medium or low intensity, combining the conversation history with the hotel scene knowledge graph to generate an emotional response strategy; S4. Generate anthropomorphic behavior using the emotional response strategy, output the anthropomorphic behavior and collect interactive feedback from users, detect emotional changes and dynamically adjust the response strategy, store the interactive data and generate a service report.

2. The method for emotional perception and dynamic response of a digital human as claimed in claim 1, characterized in that: The multimodal data includes the user's interactive voice and the user's facial expression data and body language.

3. The method for digital human emotion perception and dynamic response as claimed in claim 2, characterized in that: Extract multiple modal features from the multimodal data and use the attention mechanism to perform weighted fusion on each modal feature to generate a comprehensive sentiment feature vector, including: Construct a multimodal emotion perception network by integrating speech emotion recognition, facial expression analysis, text emotion analysis and body language capture; Use deep convolutional neural network model to extract features of speech spectrum and combine it with long short-term memory network to capture temporal emotional changes; Obtain facial expression data and capture the user's facial emotional changes through micro-expression recognition algorithms; Use language models to conduct conversation analysis and generate language emotion changes; The speech spectrum features, temporal emotion changes, facial emotion changes and language emotion changes are weightedly fused through the attention mechanism to generate a comprehensive emotion feature vector.

4. The method for emotional perception and dynamic response of a digital human as claimed in claim 3, characterized in that: The emotional state classification model and the comprehensive emotional feature vector are used to judge the emotional intensity. When the emotional intensity is medium or low, an emotional response strategy is generated by combining the conversation history and the hotel scene knowledge graph, including: Build a conversation context modeling framework based on graph neural network to dynamically associate the user's emotional state with the conversation history and scene background; Storing multiple rounds of conversation information through memory-enhanced networks to track emotional states over a long period of time; Acquire a hotel scene knowledge graph and use the hotel scene knowledge graph to enhance scene knowledge and generate an emotional response strategy; When judging the emotion intensity, the emotion state needs to be classified by an emotion intensity quantification model, and the emotion state of the emotion intensity quantification model is divided into a first preset number of major categories and a second preset number of subcategories.

5. The method for digital human emotion perception and dynamic response as claimed in claim 4, characterized in that: Generate anthropomorphic behavior using the emotional response strategy, output the anthropomorphic behavior and collect interactive feedback from users, detect emotional changes and dynamically adjust the response strategy, including: Develop an emotion-driven digital human behavior generation engine to dynamically adjust the digital human's voice intonation, facial expressions, and body movements, including: synthesizing speech and adjusting speech synthesis parameters based on the emotion intensity, wherein the speech synthesis parameters include pitch and speech rate; Generate facial expressions and body movements that match emotional states through the emotion-action mapping rule base, and render the robot's facial expressions according to the intensity of emotions; Dynamically update model parameters through gradient descent method, combine real-time emotional state monitoring algorithm with online learning mechanism, and adjust response strategy according to users' instant interactive feedback data; The instant interaction feedback data includes voice interruptions and expression changes.

6. The method for digital human emotion perception and dynamic response as claimed in claim 5, characterized in that: The method further comprises: The emotion state classification model and the comprehensive emotion feature vector are used to judge the emotion intensity. When the emotion intensity is high intensity, an emergency response mechanism is triggered to soothe the user's emotions.

7. The method for digital human emotion perception and dynamic response as claimed in claim 6, characterized in that: The interaction data stored includes: Desensitize interactive data in real time and separate emotional data from business data; The emotional data and the business data are stored separately in the sub-database, the data is encrypted using a preset encryption algorithm and all data access behaviors are recorded; The desensitizing operation is to filter sensitive information dynamically based on role permissions.

8. A digital human emotion perception and dynamic response system, characterized in that: The system comprises: A collection module, used to receive user interaction information and collect multimodal data, extract multiple modal features from the multimodal data and use an attention mechanism to perform weighted fusion on each modal feature to generate a comprehensive emotion feature vector; A model building module, used to build a deep learning model and train the deep learning model using preset emotional state data to generate an emotional state classification model; An emotional response strategy generation module is used to use the emotional state classification model and the comprehensive emotional feature vector to judge the emotional intensity, and when the emotional intensity is medium or low, generate an emotional response strategy by combining the conversation history and the hotel scene knowledge graph; The interactive feedback module is used to generate anthropomorphic behavior using the emotional response strategy, output the anthropomorphic behavior and collect interactive feedback from users, detect emotional changes and dynamically adjust the response strategy, store the interactive data and generate a service report.

9. A digital human emotion perception and dynamic response device, characterized in that: The device comprises: a processor and a memory; The memory is used to store one or more program instructions; The processor is used to run one or more program instructions to execute the steps of the digital human emotion perception and dynamic response method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the digital human emotion perception and dynamic response method as described in any one of claims 1 to 7.

Images