Emotion recognition and response system based on deep learning

Abstract

The invention belongs to the field of artificial intelligence, particularly relates to an emotion recognition and response system based on deep learning, and aims to solve the problems of low emotion recognition precision, single response strategy and lack of context perception and personalized adaptation in the prior art. The system comprises a multi-modal perception module, an emotional state modeling module, a context understanding module, a response strategy generation module and a personalized adaptation engine, and high-precision emotion recognition and intelligent dynamic response are achieved through multi-modal fusion, situational reasoning and continuous learning.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically relating to an emotion recognition and response system based on deep learning. Background Technology

[0002] Artificial intelligence (AI) technology has made significant progress in the field of human-computer interaction in recent years. Among these advancements, emotion recognition, as a key component in understanding users' psychological states and behavioral intentions, has become an important research direction for improving the intelligence level of systems. Emotion recognition technology aims to infer a user's current emotional state by analyzing facial expressions, voice tone, physiological signals, or multimodal fusion information, and then generate corresponding interactive responses accordingly. This technology is widely used in scenarios such as intelligent customer service, mental health assistance, educational assessment, and in-vehicle interaction, and is of great significance for improving user experience and system adaptability.

[0003] Among these, deep learning-based emotion recognition and response systems focus on automatically extracting high-dimensional emotion features using neural network models and achieving end-to-end emotion classification and feedback generation. These systems typically rely on large-scale labeled data to train models to capture the non-linear patterns of complex emotional expressions and optimize response strategies by incorporating contextual information.

[0004] Current technologies still face multiple challenges in practical applications: single-modal emotion recognition is susceptible to interference from environmental noise or individual differences in expression, leading to unstable recognition accuracy; while multimodal fusion methods can improve robustness, existing models have limited ability to model the temporal alignment and semantic consistency of heterogeneous data, making it difficult to effectively integrate visual, auditory, and textual cues; furthermore, most systems lack the ability to model the dynamic evolution of emotions, failing to adapt to emotion drift during long-term interactions; simultaneously, existing response mechanisms are mostly based on preset rules or static generation, making it difficult to achieve personalized, context-aware feedback deeply coupled with the recognition results. These problems severely restrict the reliability and practicality of emotion recognition systems in real-world complex scenarios, urgently requiring an integrated solution capable of achieving high-precision recognition and intelligent dynamic response. Summary of the Invention

[0005] The purpose of this invention is to provide a deep learning-based emotion recognition and response system to solve the technical problems of insufficient recognition accuracy, single response strategy, lack of context awareness, and difficulty in achieving personalized dynamic adaptation in existing emotion recognition technologies in complex interactive scenarios.

[0006] The technical solution of this invention includes a multimodal perception module, an emotion state modeling module, a context understanding module, a response strategy generation module, and a personalized adaptation engine. The multimodal perception module synchronously collects the user's voice signals, facial video streams, physiological signals, and text input data, and preprocesses and extracts features from each modality. The emotion state modeling module receives the feature vector output by the multimodal perception module, models the user's current emotional state by fusing a deep neural network, and outputs a structured emotion representation containing emotion category, intensity value, and confidence level. The context understanding module acquires the user's historical interaction records, current task objectives, environmental context information, and social relationship graph, and constructs a dynamic contextual semantic vector. The response strategy generation module combines the emotion representation output by the emotion state modeling module and the contextual semantic vector output by the context understanding module, and generates a response action sequence matching the current context through a policy inference network. The personalized adaptation engine dynamically adjusts the weight parameters of the emotion recognition model and the decision threshold of the response strategy generation module based on the user's long-term behavioral preferences, emotion regulation habits, and feedback history, achieving continuous optimization and personalized adaptation of system behavior.

[0007] Furthermore, the multimodal perception module includes a speech feature extraction unit, a visual feature extraction unit, a physiological signal processing unit, and a text semantic parsing unit; the speech feature extraction unit is used to extract fundamental frequency, speech rate, energy profile, and Mel frequency cepstral coefficients from the speech signal; the visual feature extraction unit is used to extract facial motion unit intensity sequence, micro-expression temporal features, and head posture parameters from the facial video stream; the physiological signal processing unit is used to extract time-domain and frequency-domain features from heart rate variability, skin conductance, and electromyography signals; the text semantic parsing unit is used to perform word segmentation, part-of-speech tagging, sentiment dictionary matching, and syntactic dependency analysis on the input text, and output a text sentiment tendency vector.

[0008] Furthermore, the emotion state modeling module adopts a multi-branch fusion architecture, including an independent emotion recognition sub-network and a cross-modal attention fusion layer; each emotion recognition sub-network processes a single modality feature and outputs the emotion probability distribution under that modality; the cross-modal attention fusion layer is used to calculate the correlation weights between the features of each modality, and to perform weighted fusion of the outputs of each sub-network according to the weights to generate the final emotion state representation; the emotion state representation includes discrete emotion category labels, continuous emotion dimension values, and emotion recognition confidence scores.

[0009] Furthermore, the context understanding module includes a task intent recognition unit, an environment perception unit, and a social relationship modeling unit; the task intent recognition unit is used to analyze the task objective corresponding to the user's current interaction behavior and output the task type code; the environment perception unit is used to access environmental sensor data and identify the current physical environment type, light intensity, noise level, and personnel density; the social relationship modeling unit is used to construct a dynamic social graph based on the user's historical social interaction data and output the intimacy level and role tag of the current interaction object; the context understanding module encodes the above three types of information into a context semantic vector of a unified dimension.

[0010] Furthermore, the response strategy generation module includes a strategy candidate library, a context matching unit, and an action sequence generation unit. The strategy candidate library stores predefined multi-level response action templates, covering language feedback, facial expression simulation, voice tone adjustment, interaction rhythm control, and service behavior triggering. The context matching unit is used to calculate the joint embedding of the current emotion representation and the context semantic vector, and to retrieve the most matching response strategy category from the strategy candidate library. The action sequence generation unit is used to generate specific and executable response action sequences based on the matching results, including action type, execution parameters, and timing arrangement.

[0011] Furthermore, the personalized adaptation engine includes a user profile construction unit, a feedback learning unit, and a model fine-tuning unit. The user profile construction unit integrates the user's historical emotional response patterns, preferred response types, and interaction satisfaction scores to construct a dynamic user profile vector. The feedback learning unit collects explicit and implicit behavioral feedback from users in real time and calculates response effect evaluation indicators. The model fine-tuning unit incrementally updates the fusion weights of the emotional state modeling module and the matching thresholds of the response strategy generation module based on the evaluation indicators and using an online learning mechanism.

[0012] Furthermore, the system operates within a hierarchical decision-making architecture, comprising a perception layer, a cognition layer, and an execution layer. The perception layer is responsible for real-time acquisition and feature extraction of multimodal data. The cognition layer is responsible for emotional state modeling and contextual understanding, and for completing policy reasoning. The execution layer is responsible for transforming the generated response action sequence into specific interactive behaviors and outputting them through a human-computer interaction interface. The layers communicate with each other with low latency through standardized data interfaces to ensure the real-time performance and consistency of the system response.

[0013] Furthermore, the cross-modal attention fusion layer in the emotion state modeling module adopts a gating mechanism to dynamically adjust the fusion weights of each modality based on the confidence level of the current dominant modality. When the data quality of a certain modality decreases or is missing, the system automatically increases the weights of other reliable modalities to maintain the robustness of emotion recognition.

[0014] Furthermore, the action sequence generation unit in the response strategy generation module supports multi-objective optimization, taking into account the effect of emotional soothing, task completion efficiency and user comfort, and optimizes strategy selection in long-term interaction through reinforcement learning mechanism.

[0015] Compared with the prior art, the advantages and positive effects of the present invention are as follows: This invention effectively overcomes the limitations of single-modal recognition in complex scenarios by constructing a multimodal deep fusion emotion recognition architecture, significantly improving the accuracy and robustness of emotion recognition. The introduction of a contextual understanding module enables the system to make contextual judgments based on task objectives, environmental states, and social relationships, avoiding misjudgments of emotions out of context. The response strategy generation module performs joint reasoning based on structured emotion representations and contextual semantic vectors, generating response actions that not only match the user's current emotional state but also fit the interaction context, achieving a closed loop from "emotion recognition" to "intelligent response." The personalized adaptation engine continuously learns from user feedback and dynamically optimizes recognition and response parameters, enabling the system to evolve over the long term and adapt to the unique emotional expressions and interaction preferences of different users. The overall system adopts a hierarchical decision-making architecture, balancing real-time performance and intelligence, providing highly available and adaptable technical solutions for applications such as intelligent customer service, mental health assistance, educational companionship, and human-computer collaboration. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall technical solution architecture of the present invention. Detailed Implementation

[0017] Example 1 Please refer to Figure 1 This invention provides a deep learning-based emotion recognition and response system. Its core objective is to overcome the limitations of existing emotion recognition technologies, such as insufficient accuracy in complex interactive scenarios, simplistic response strategies, lack of context awareness, and difficulty in achieving personalized dynamic adaptation. This system constructs a multimodal deep fusion emotion recognition architecture, combines comprehensive contextual understanding, and introduces a dynamic adaptive response strategy generation and personalized adaptation mechanism to achieve closed-loop management from accurate perception of user emotions to contextualized intelligent responses. The system operates on a hierarchical decision-making architecture, balancing real-time performance and intelligence, and aims to provide high-performance technical solutions for various application scenarios such as intelligent customer service, mental health assistance, educational companionship, and human-computer collaboration.

[0018] The system's overall technical flow begins with the multimodal perception module synchronously collecting and preprocessing user interaction data. Subsequently, the emotion state modeling module receives these processed features and accurately models the user's emotional state using a deep fusion network. Simultaneously, the context understanding module acquires and constructs dynamic contextual semantic vectors in parallel. The emotion representation output by the emotion state modeling module and the contextual semantic vector output by the context understanding module are used as input to the response strategy generation module, which generates a sequence of response actions highly matched to the current context. Finally, the personalized adaptation engine continuously optimizes the system's recognition and response capabilities based on the user's long-term behavioral preferences and historical feedback, achieving dynamic adjustment and personalized optimization of behavior.

[0019] This system employs a hierarchical decision-making architecture, ensuring its efficient operation and collaborative capabilities in complex environments. The perception layer primarily handles real-time acquisition of multimodal data and preliminary feature extraction, its core function being to ensure the diversity, synchronization, and high quality of the raw data. The cognition layer, the system's intelligent core, is responsible for in-depth emotional state modeling and complex contextual understanding of the features output by the perception layer, and based on this, performs policy reasoning. The execution layer concretizes the abstract response action sequences generated by the cognition layer, transforming them into perceptible and operable human-computer interaction behaviors, and outputting them through various human-computer interaction interfaces. Low-latency communication between layers is achieved through standardized data interfaces to ensure the real-time and consistent nature of the system's response. This hierarchical design effectively reduces system complexity and enhances the independence between modules and the system's maintainability.

[0020] The multimodal perception module aims to provide comprehensive, detailed, and multi-dimensional data input for subsequent intelligent decision-making, thereby capturing subtle changes and complex expressions of user emotions. This module is designed as a highly integrated real-time data aggregation and primary processing platform, capable of simultaneously acquiring data streams from different types of sensors or interfaces. Voice signals are acquired through a high-fidelity microphone array, ensuring effective acoustic capture even in noisy environments. Facial video streams are captured in real-time by a high-resolution camera to analyze subtle facial muscle movements and macroscopic facial expression changes. Physiological signal data, such as heart rate variability, skin conductance, and electromyography signals, are acquired in real-time through wearable biosensors or non-contact sensors, providing objective indicators of the user's physiological arousal level. Text input data is acquired through the user input interface or a speech-to-text service. All acquired raw data undergoes time-stamp synchronization and preliminary data cleaning to remove obvious noise and artifacts.

[0021] The multimodal perception module is further subdivided into several functional units, including a speech feature extraction unit, a visual feature extraction unit, a physiological signal processing unit, and a text semantic parsing unit.

[0022] The speech feature extraction unit performs in-depth acoustic feature analysis on the acquired raw speech signal. After digital sampling and frame processing, the speech signal first undergoes a pre-emphasis filter to remove high-frequency noise and enhance high-frequency components. Subsequently, the unit performs a Fourier transform on each speech frame to calculate its spectral characteristics. The fundamental frequency extracted from the spectrum reflects the vocal cord vibration frequency of the pronunciation and is closely related to emotional arousal. Speech rate is determined by analyzing the frequency of syllables or words and the duration of silence intervals; its speed is often related to the degree of emotional activation or stress level. Energy profile is characterized by calculating the short-term energy of the speech signal, revealing the intensity changes of pronunciation. Mel-frequency cepstral coefficients, as a highly compressed acoustic feature, simulate the human ear's perception of sound frequencies and can effectively capture subtle differences in timbre and speech quality. All extracted speech features are organized into a temporal feature vector sequence and normalized for use as input to subsequent deep learning models. During processing, the unit's built-in speech activity detection algorithm filters out prolonged silences, ensuring that only segments containing valid speech information are used for feature extraction, thereby improving processing efficiency and feature quality. Simultaneously, the unit monitors the integrity of the audio stream, marking any missing or corrupted data and handling it through interpolation or discarding strategies to avoid incomplete data affecting subsequent emotion recognition.

[0023] The visual feature extraction unit is specifically designed to extract emotion-related visual features from real-time facial video streams. The video stream first undergoes frame segmentation and face detection to ensure accurate identification of facial regions in each frame. The unit utilizes advanced facial landmark detection algorithms, such as a convolutional neural network-based facial landmark localization model, to accurately track 68 or more feature points on the face. Based on this, a facial motion coding system, such as an OpenFace-based algorithm, calculates the intensity sequence of facial motion units, such as the intensity values ​​of eyebrow raising, mouth drooping, and eye closing. These intensity values ​​can finely quantify the dynamic changes in facial expressions. Simultaneously, the unit extracts micro-expression temporal features. Micro-expressions are involuntary facial expressions with extremely short durations and very low intensity. Through detailed time-series analysis of the high frame rate video stream, it captures rapid, instantaneous contractions or relaxations of facial muscles. Furthermore, the unit analyzes head posture parameters, including head pitch, yaw, and roll angles. These posture changes indirectly reflect the user's attention and emotional engagement. All extracted visual features, such as the action unit intensity matrix and pose angle sequence, are also standardized and encapsulated into temporal feature vectors, ready to be transmitted to the emotion state modeling module. To ensure the accuracy of feature extraction, the unit implements illumination normalization and head pose correction algorithms to reduce the interference of ambient light changes and user head movements on feature extraction.

[0024] The physiological signal processing unit is responsible for preprocessing and feature extraction of physiological signals from various biosensors. Raw physiological signals, such as electrocardiogram (ECG) signals for heart rate variability analysis, skin conductance signals for skin conductance response analysis, and electromyography (EMG) signals for electromyographic activity analysis, often contain significant noise and baseline drift. Therefore, the unit first filters these signals to remove power line interference, motion artifacts, and high-frequency noise. Heart rate variability features are extracted by analyzing continuous heartbeat interval time series (RR intervals), including time-domain indicators (e.g., SDNN, RMSSD) and frequency-domain indicators (e.g., high-frequency HF, low-frequency LF power). These indicators reflect the activity state of the autonomic nervous system and are closely related to emotional arousal and stress responses. Skin conductance response features are obtained by measuring changes in skin conductance levels, such as skin conductance level and the frequency and amplitude of skin conductance response events. These changes are a result of sweat gland activity and are positively correlated with the degree of emotional arousal. EMG signals are characterized by analyzing muscle electrical activity, such as muscle tension and contraction frequency. These data reflect the body's physiological state of tension. All extracted physiological features are time-aligned and normalized to form a multi-dimensional physiological feature vector. The unit continuously monitors the quality of sensor data. In the event of signal interruption, sensor detachment, or data anomalies, an alarm mechanism is triggered and backup data processing logic is initiated, such as filling with the last valid value or performing predictive interpolation based on historical data, to ensure the continuity and reliability of physiological feature data.

[0025] The text semantic parsing unit focuses on deep semantic and sentiment analysis of user-input text data. The raw input text first undergoes a standardized natural language processing workflow, including word segmentation, part-of-speech tagging, and stop word removal. Subsequently, the unit utilizes pre-trained word vector models, such as the Chinese BERT or Word2Vec model, to convert each word into a high-dimensional dense vector. Based on these word vectors, the unit performs sentiment lexicon matching to identify positive, negative, or neutral sentiment words and their intensity within the text, and, combined with a sentiment rule engine, corrects negation words, degree adverbs, etc. Further, the unit performs syntactic dependency analysis to parse the grammatical relationships between words in the sentence, thereby understanding the sentence structure and more complex sentiment expressions, such as irony or satire. Finally, the unit outputs a text sentiment vector that integrates the text's sentiment tendency, topic distribution, and key sentiment words. This vector quantifies the overall sentiment polarity, intensity, and fine-grained sentiment category of the text. During processing, the unit identifies and corrects common typos, performs context disambiguation for ambiguous expressions, and supports multilingual text processing to meet the needs of globalized application scenarios.

[0026] The emotion state modeling module's core function is to receive the feature vectors output by the multimodal perception module and accurately model the user's current emotional state through a deep fusion neural network, ultimately outputting a structured emotion representation that includes emotion category, intensity value, and confidence level. This module adopts a multi-branch fusion architecture to fully utilize the complementarity and redundancy of data from different modalities.

[0027] The multi-branch fusion architecture includes independent emotion recognition sub-networks and a cross-modal attention fusion layer. Each emotion recognition sub-network is responsible for processing feature vector inputs from a single modality. For example, the speech sub-network may use a long short-term memory network or a temporal convolutional network to process speech feature sequences, the vision sub-network may use a 3D convolutional network or a spatiotemporal graph convolutional network to process visual feature sequences, the physiological sub-network may use a recurrent neural network to process physiological signal features, and the text sub-network may use a Transformer or a recurrent network with an attention mechanism to process text feature vectors. After independent training, each sub-network can output the emotion probability distribution for that modality, for example, expressed as probability values ​​for discrete emotion categories such as "happy," "sad," "angry," and "calm," or numerical values ​​for continuous emotion dimensions (such as arousal and valence).

[0028] The cross-modal attention fusion layer is a key innovation in the emotion state modeling module. This layer calculates the correlation weights between features of different modalities and then weights and fuses the outputs of each sub-network based on these weights to generate the final emotion state representation. Specifically, the fusion layer employs a gating mechanism to dynamically adjust the fusion weights of each modality. The gating mechanism adjusts the fusion strategy based on the confidence level of the currently dominant modality. For example, if the visual sub-network's confidence level in recognizing the emotion "anger"—for instance, the maximum probability value output by softmax—is much higher than that of other modalities, the gating mechanism will assign a higher fusion weight to the visual modality. When the data quality of a particular modality deteriorates or is missing, such as when a camera is obstructed causing blurred visual data, or when a microphone malfunction causes interrupted audio data, the system automatically identifies this situation and maintains the robustness of emotion recognition by reducing the weight of that modality while increasing the weights of other reliable modalities. This dynamic weighting strategy ensures that the system can still accurately recognize emotions by relying on the remaining effective modalities even when some data is damaged or unavailable. The final output of the fusion layer is a structured data package containing discrete emotion category labels, such as "calm" and "excited," continuous emotion dimension values, such as valence ranging from -1 to 1 and arousal ranging from 0 to 1, and an overall emotion recognition confidence score. This score comprehensively reflects the model's confidence in its current emotion judgment. During training, the module uses a large-scale labeled emotion dataset and employs techniques such as multi-task learning or adversarial training to improve the model's generalization ability.

[0029] The context understanding module is responsible for acquiring and integrating user history, current task objectives, environmental context information, and social relationship graphs, and constructing them into dynamic contextual semantic vectors. This enables the system to go beyond simple emotion recognition and make deeper contextual judgments, thereby avoiding misjudgments of emotions out of context.

[0030] The context understanding module further includes a task intent recognition unit, an environment perception unit, and a social relationship modeling unit.

[0031] The task intent recognition unit is dedicated to analyzing the task objective corresponding to the user's current interaction and outputting a task type code. For example, in a smart home assistant, if the user says "Please play music," the unit will recognize the task intent as "play music"; if the user says "How's the weather today?", it will recognize the intent as "weather query." The unit uses a deep learning model based on natural language understanding, such as the Transformer model or recurrent neural networks, to semantically parse the user's text or voice input and map it to predefined service domains and intent categories. The model infers the user's intent by analyzing keywords, syntactic structure, and past interaction patterns. The output task type code is a standardized data structure containing the task's unique identifier, priority, and possibly involved entity parameters. The unit can handle task context in multi-turn dialogues, avoid repeatedly recognizing intents in continuous dialogues, and has error correction capabilities, such as confirming with the user when the expression is ambiguous.

[0032] The environmental perception unit is responsible for accessing data from various environmental sensors to identify the type of the current physical environment, light intensity, noise level, and population density. These sensors can include ambient light sensors built into smart devices, microphone arrays, and even third-party environmental monitoring devices connected to an IoT platform. For example, illuminance values ​​obtained through ambient light sensors can quantify light intensity, ambient sound analysis through microphone arrays can quantify noise levels, and target detection using anonymized camera images can roughly estimate population density. Environmental type recognition may be based on geographic location information or preset scene patterns, such as "office," "home," and "outdoors." These environmental parameters are encoded as standardized numerical or categorical data, such as light intensity values ​​ranging from 0 to 20,000 lux and noise levels ranging from 20 to 100 decibels. The unit is designed to provide non-intrusive environmental perception capabilities, and all data processing adheres to privacy protection principles. In particular, for camera data, edge anonymization and feature extraction are prioritized rather than uploading raw images. The unit monitors changes in environmental parameters in real time and triggers updates when parameters exceed preset thresholds to ensure the freshness of contextual information.

[0033] The social relationship modeling unit constructs a dynamic social graph based on users' historical social interaction data and outputs the intimacy level and role tags of the currently interacting objects. Historical social interaction data may come from users' address books, social media interaction records, and historical interaction records with the system. Nodes in the social graph represent users and their contacts, edges represent the interaction relationships between them, and edge weights quantify the intimacy level, such as a range of 1 to 5 from "stranger" to "close friend." Role tags describe the identity of the interacting object in the current context, such as "colleague," "family member," or "customer service representative." The unit dynamically analyzes the social graph using graph neural networks or community detection algorithms, updating intimacy weights in real time and identifying potential social roles. The output intimacy level and role tags are encoded as discrete or continuous feature values. During processing, the unit strictly adheres to user privacy settings, accessing and analyzing social data only with user authorization, and employs techniques such as federated learning to train and update the model without disclosing the original data.

[0034] The context understanding module encodes the three types of information mentioned above—task type encoding, environmental parameter encoding, and intimacy level and role label—into a unified-dimensional contextual semantic vector. This encoding may be achieved through simple feature concatenation or more complex feature fusion networks, such as multilayer perceptrons or attention mechanism networks. The resulting contextual semantic vector is a dense, high-dimensional vector that comprehensively and abstractly represents the complex situation the user is currently in. This vector plays a crucial role in subsequent response strategy generation, enabling the system to generate more accurate and appropriate responses based on the user's specific circumstances, rather than just their emotions.

[0035] The response strategy generation module is a crucial component for achieving intelligent system responses. Its function is to combine the emotion representation output by the emotion state modeling module with the contextual semantic vector output by the context understanding module, and then generate a sequence of response actions that matches the current context through a policy inference network. The design philosophy of this module is to achieve a seamless transition from emotion recognition to specific interactive behaviors, ensuring that the system can provide appropriate and context-sensitive feedback.

[0036] The response strategy generation module includes a strategy candidate library, a context matching unit, and an action sequence generation unit.

[0037] The strategy candidate library is a structured knowledge base that stores predefined, multi-level response action templates. These templates cover a wide range of interaction behavior types, including verbal feedback, such as "I understand you are feeling frustrated"; facial expression simulation, such as expressing "sympathy" or "happiness" by controlling the facial muscles of a virtual avatar or the facial expressions of a physical robot; voice tone adjustment, such as slowing down the system's speech rate and lowering the pitch to express "soothing"; interaction rhythm control, such as increasing response latency or reducing information content when the user is emotionally agitated; and service behavior triggering, such as automatically invoking relevant service interfaces when the user expresses a need for help. Each template contains multiple levels; for example, a "soothing" strategy might include sub-actions such as "gentle tone," "slow speech rate," and "affirmative verbal feedback," along with adjustable parameter ranges and temporal constraints. The strategy candidate library is organized in the form of an ontology or knowledge graph for efficient retrieval and expansion.

[0038] The context matching unit acts as a bridge connecting emotion and response strategy. This unit receives the structured emotion representation output by the emotion state modeling module and the contextual semantic vector output by the context understanding module, and first calculates their joint embedding. This joint embedding process may be implemented through a dedicated fusion network that maps the emotion and context dimensions into a shared high-dimensional space. Within this joint embedding space, the system can comprehensively understand the user's current emotion and context. Subsequently, the context matching unit retrieves the most matching response strategy category from a strategy candidate library. The retrieval process may employ similarity calculation methods, such as cosine similarity, or a deep learning-based matching network to find the strategy category with the smallest distance or highest similarity between the joint embedding and the strategy template embedding. The matching result is one or a set of most suitable response strategy category identifiers, such as "comforting strategy," "encouraging strategy," and "clarifying strategy." During the matching process, the unit also considers the priority and mutual exclusivity of strategies to avoid generating conflicting or inappropriate strategy suggestions.

[0039] The action sequence generation unit generates specific, executable response action sequences based on the matching results from the context matching unit. Once a strategy category is determined, the unit instantiates the corresponding action template from the strategy candidate library and fills in the variable parts of the template according to the detailed parameters of the current context. For example, if a "comfort strategy" is matched, the system may select specific comforting statements based on the intensity of emotion and the level of intimacy of the context, and adjust the parameter values ​​of the voice tone. The generated action sequence is a detailed list of instructions, including the type of each action, execution parameters, execution order, and timing arrangement, such as "adjust the voice tone to a soothing mode within 200 milliseconds, then say 'I understand how you feel' after 1 second, and simultaneously raise the corners of the virtual avatar's mouth by 10 degrees." The action sequence generation unit supports multi-objective optimization, considering the emotional soothing effect, task completion efficiency, and user comfort simultaneously when generating action sequences. This may be achieved through a reinforcement learning mechanism, in which the agent learns through trial and error by interacting with the user and optimizes strategy selection over the long term. The reward function of a reinforcement learning model comprehensively considers multiple dimensions such as user feedback, emotional changes, and task progress, enabling the system to improve the overall quality of its responses through continuous learning. For example, an overly aggressive "soothing" response might reduce task efficiency, while an overly indifferent "clarification" response might increase user discomfort. This unit, through reinforcement learning, can weigh these objectives and generate an optimal sequence of actions.

[0040] The personalized adaptation engine is the core of the system's long-term evolution and self-adaptation capabilities. Its goal is to dynamically adjust the weight parameters of the emotion recognition model and the decision threshold of the response strategy generation module based on the user's long-term behavioral preferences, emotion regulation habits, and feedback history, thereby achieving continuous optimization and highly personalized adaptation of the system's behavior.

[0041] The personalized adaptation engine includes a user profile building unit, a feedback learning unit, and a model fine-tuning unit.

[0042] The user profile building unit integrates users' historical emotional response patterns, preferred response types, and interaction satisfaction scores to construct a dynamic user profile vector. Historical emotional response patterns include the distribution of emotions exhibited by users in different situations, emotional trigger thresholds, and spontaneous emotional regulation behaviors. Preferred response types record the user's acceptance and satisfaction with different system response methods; for example, whether the user prefers verbal reassurance or understanding expressed through facial expressions. Interaction satisfaction scores are derived from the aggregation of explicit or implicit user feedback. The user profile vector is a high-dimensional, dense numerical vector capable of capturing users' unique emotional expression habits and interaction preferences. This vector is dynamically updated, for example, through sliding time windows or exponentially weighted averaging, ensuring it reflects changes in user behavior over time. The establishment of user profiles aims to provide precise, personalized guidance for subsequent model fine-tuning, making the system no longer a one-size-fits-all solution but rather tailored to each user.

[0043] The feedback learning unit is responsible for collecting explicit and implicit behavioral feedback from users in real time regarding the system's responses, and calculating response effectiveness evaluation metrics. Explicit feedback includes user ratings, verbal comments, and questionnaires. Implicit behavioral feedback is more diverse and subtle, such as changes in facial expressions, tone of voice, efficiency in completing subsequent tasks, number of dialogue turns, and even changes in physiological signals after a system response. The unit continuously monitors these implicit signals through a multimodal perception module. These explicit and implicit feedbacks are comprehensively analyzed to calculate multi-dimensional response effectiveness evaluation metrics, such as emotion improvement index, task efficiency improvement rate, and user experience satisfaction. The calculation of evaluation metrics may involve complex statistical models or machine learning classifiers, which map raw feedback data to quantified evaluation values. The feedback learning unit is designed to provide an objective and comprehensive evaluation mechanism to measure the actual effect of each system response. For negative feedback, the unit conducts in-depth analysis to identify the root causes of the problem and passes this information to the model fine-tuning unit.

[0044] The model fine-tuning unit is the core of the personalized adaptation engine. Based on the evaluation metrics calculated by the feedback learning unit, this unit uses an online learning mechanism to incrementally update the fusion weights of the emotion state modeling module and the matching threshold of the response policy generation module. The online learning mechanism means that model parameter updates are continuous, rather than waiting for retraining on a large dataset. For example, the weight parameters of the cross-modal attention fusion layer in the emotion state modeling module are fine-tuned based on user feedback on the system's emotion recognition results to adapt to users' unique facial expressions, tone of voice, or physiological signals. If the system frequently misjudges emotions in a specific user, the fine-tuning unit adjusts the fusion weights of the corresponding modality or the parameters of its sub-networks. Similarly, the matching threshold of the response policy generation module—the decision boundary that determines when to choose which policy—is adjusted based on user satisfaction with the system's response. If users generally feel uncomfortable with a certain type of response, the matching threshold for that response policy is increased, making it harder to select. The update process typically employs incremental gradient descent or policy gradient methods from reinforcement learning to ensure the model can smoothly adapt to new data. The update frequency and learning rate are configurable, for example, they can be fine-tuned every certain time period or after accumulating enough user feedback, and the learning rate can be dynamically adjusted based on experience or meta-learning methods.

[0045] This embodiment constructs a closed-loop intelligent system through the collaborative work of the aforementioned modules. The system is no longer limited to a single emotion recognition function, but organically combines recognition, understanding, response, and personalized optimization. The implementation of this solution effectively improves the accuracy and robustness of emotion recognition in complex, realistic interaction scenarios, avoids misjudgments through contextual understanding, and provides responses that highly match user emotions and context. More importantly, the personalized adaptation engine endows the system with the ability to evolve over the long term, enabling it to adapt to the unique emotional expressions and interaction preferences of different users, thereby providing users with a more natural, comfortable, and effective interactive experience, ultimately significantly improving the quality and depth of human-computer interaction.

[0046] Existing technologies in emotion recognition generally suffer from significant drawbacks, including limited recognition accuracy, rigid response strategies, lack of context awareness, and insufficient personalization capabilities. For example, single-modal emotion recognition systems often perform poorly when faced with complex user emotional expressions or poor quality information from a particular modality. Response systems also tend to rely on preset rules or simple mappings, making it difficult to address the diverse needs of different situations and users' personalized requirements.

[0047] The core difference of this solution lies in its systematic architectural approach to addressing these issues. At the emotion recognition level, through deep feature extraction by the multimodal perception module and a multi-branch fusion architecture of the emotion state modeling module, along with a cross-modal attention fusion layer, it achieves multi-dimensional and high-precision capture of emotions, while a gating mechanism ensures robustness against data quality fluctuations. At the understanding level, the context understanding module innovatively integrates task intent, environmental state, and social relationships to construct a comprehensive contextual semantic vector, significantly enhancing the contextualization of decision-making. At the response level, the response strategy generation module not only possesses a rich and hierarchical policy candidate library, but more importantly, the context matching unit and action sequence generation unit can perform intelligent reasoning based on joint embedding, and optimize action sequences through multi-objective reinforcement learning to ensure the appropriateness and comprehensive effectiveness of the response. Finally, the introduction of a personalized adaptation engine enables the system to dynamically evolve through user profiling, feedback learning, and model fine-tuning, breaking the static limitations of traditional systems. The organic integration of these mechanisms allows this invention to provide a truly intelligent and adaptive emotion recognition and response solution, far exceeding the performance and application scope of existing technologies.

[0048] Example 2 This embodiment further elaborates on the gating mechanism of the cross-modal attention fusion layer in the emotion state modeling module. The fundamental purpose of this mechanism is to dynamically adjust the contribution weights of different modalities in emotion recognition when the quality of the raw data acquired by the multimodal perception module fluctuates, thereby maintaining the robustness and accuracy of emotion recognition.

[0049] The gating mechanism operates as follows: When the emotion state modeling module receives emotion probability distributions or continuous emotion dimension values ​​from each emotion recognition sub-network, the fusion layer first evaluates the current confidence level of each modality. The confidence level of the dominant modality can be evaluated in several ways. For example, the entropy value of the probability distribution output by each sub-network can be calculated; a lower entropy value indicates a higher certainty for the emotion classification, and thus a higher confidence level. Another approach is to directly use the predicted confidence scores obtained during the training of each sub-network, such as the maximum probability value output by the softmax function. The system then compares the confidence levels of all modalities to identify the current dominant modality.

[0050] The gating mechanism dynamically adjusts the fusion weights of each modality based on the confidence level of the currently dominant modality. Specifically, when the confidence level of a certain modality is significantly higher than that of other modalities, the gating mechanism assigns a higher fusion weight to that modality. For example, if a user is speaking in a quiet environment, the confidence level of the speech modality may be very high, and the system will tend to trust the recognition results of the speech subnetwork more. Conversely, when the data quality of a certain modality deteriorates or is missing, the system will automatically identify this situation. Deterioration in data quality can be determined by detecting the signal-to-noise ratio of the raw sensor data, the sparsity of the feature vector, or specific error codes. For example, blurred camera images, interrupted microphone input, and detached physiological sensors can all lead to a deterioration or loss of data quality for the corresponding modality. Once this situation is detected, the gating mechanism will immediately reduce the fusion weight of that modality and correspondingly increase the weights of other reliable modalities. For example, when the speech signal is impaired in a noisy environment, the system will rely more on the visual and text modalities to determine the user's emotions.

[0051] This dynamic weight adjustment mechanism can be implemented using a small feedforward neural network as a gating network. This network takes the confidence scores and data quality metrics of each modality as input and outputs the weighting coefficients for each modality during final fusion. These weighting coefficients sum to 1, ensuring the validity of the probability distribution. The gating network learns during training how to optimally allocate modality weights to maximize the accuracy of emotion recognition under various scenarios of degraded or missing data. In this way, even in complex and variable real-world application environments with partial data defects, the emotion state modeling module maintains high robustness, thus providing consistently stable and accurate emotion state representations.

[0052] The advantage of this embodiment lies in its significantly enhanced practicality and stability in complex real-world environments. In practical applications, sensor data corruption or loss is common; for example, insufficient light can cause visual blurring, excessive background noise can affect speech recognition, or unstable wearable device connections can lead to the loss of physiological data. Traditional fusion methods often experience a sharp deterioration in overall system performance when the quality of data from a single modality declines. By introducing a gating mechanism, this system can intelligently assess the reliability of each modality and dynamically adjust its weight in decision-making. This means that even when some data is unavailable, the system can continue effective emotion recognition by prioritizing high-quality modal information, thus avoiding the "weakest link" effect and ensuring the continuous accuracy and high availability of emotion recognition.

[0053] Example 3 This embodiment further elaborates on the multi-objective optimization and reinforcement learning mechanism supported by the action sequence generation unit in the response strategy generation module. The core purpose of this mechanism is to not only match the user's current emotion when generating a specific response action sequence, but also to weigh multiple interrelated and potentially conflicting objectives such as emotional soothing effect, task completion efficiency, and user comfort, and to continuously optimize the strategy selection through long-term interaction.

[0054] Achieving multi-objective optimization first requires quantifying these three objectives. The effect of emotional soothing can be measured by changes in the user's emotional valence (positive or negative) and arousal (intensity) detected by the emotional state modeling module. For example, a shift from negative high arousal to positive low arousal is considered a good soothing effect. Task completion efficiency can be evaluated by task duration, task error rate, and whether the task was successfully completed. User comfort is a more comprehensive indicator, which can be judged by combining preferences in the user profile, satisfaction scores collected by the feedback learning unit, and implicit behaviors (such as body language and interaction duration).

[0055] The action sequence generation unit optimizes policy selection through long-term interactions using a reinforcement learning mechanism. The reinforcement learning framework includes the agent, environment, state, action, and reward. In this system, the agent is the action sequence generation unit, and the environment is the user and their interaction context. The state is composed of the current emotional representation, contextual semantic vector, and user profile vector, comprehensively describing the user's current state and background information. The action space defines all possible response actions the agent can take, instantiated from a policy candidate library, including language feedback, facial expression simulation, voice tone adjustment, interaction rhythm control, and service behavior triggering. Each action has its own execution parameters and timing.

[0056] The reward function is the core of reinforcement learning. It calculates reward values ​​based on the agent's actions and the resulting environmental changes, thereby guiding the agent to learn the optimal strategy. In this embodiment, the reward function is a multi-objective comprehensive reward function that simultaneously considers emotional soothing effects, task completion efficiency, and user comfort. For example, a successful soothing action may receive a positive reward, as may efficient task assistance, while a response that causes user discomfort will receive a negative penalty. The design of the reward function requires fine-grained weighting and balancing to reflect the importance of different objectives. For example, in a mental health assistance scenario, emotional soothing may have a higher weight; while in an intelligent customer service scenario, task completion efficiency may be more important. These weights can be preset using expert knowledge or dynamically adjusted based on system performance using meta-learning methods.

[0057] Through continuous interaction with the user, the intelligent agent constantly tries different sequences of response actions and learns from environmental feedback, such as changes in the user's subsequent emotions, task completion status, and explicit user feedback, to evaluate the effectiveness of its actions. Using policy gradient algorithms, Q-learning, or deep reinforcement learning (such as deep Q-networks and A2C algorithms), the agent continuously updates its policy network, enabling it to select action sequences that maximize long-term cumulative rewards based on its current state. This means that the system learns, through trial and error, from a large amount of interactive experience which responses yield the best overall results in which situations.

[0058] The positive effect of this embodiment lies in its ability to generate more intelligent and balanced responses. Traditional systems often optimize only for a single objective, such as pursuing emotional soothing while neglecting the effective completion of the task, or vice versa. However, through multi-objective optimization and reinforcement learning, this system can dynamically weigh these objectives in complex situations to find a comprehensive and optimal response strategy. For example, when a user expresses anxiety while simultaneously needing to complete an urgent task, the system will not merely soothe the emotion but may choose a strategy that appropriately alleviates anxiety while guiding the user to complete the task efficiently. This long-term learning and optimization mechanism allows the system to transcend the limitations of preset rules and continuously adapt to the complex needs of different users and in different situations, thereby significantly improving the intelligence, adaptability, and user satisfaction of human-computer interaction.

Claims

1. A deep learning-based emotion recognition and response system, characterized in that, include: The multimodal perception module is used to simultaneously acquire the user's voice signal, facial video stream, physiological signal and text input data, and to preprocess and extract features from each modality of data; The emotion state modeling module receives the feature vectors output by the multimodal perception module, models the user's current emotion state by fusing deep neural networks, and outputs a structured emotion representation that includes emotion category, intensity value, and confidence level. The context understanding module is used to obtain the user's historical interaction records, current task objectives, environmental context information, and social relationship graph, and to construct dynamic context semantic vectors; The response strategy generation module combines the emotion representation output by the emotion state modeling module with the context semantic vector output by the context understanding module, and generates a sequence of response actions that match the current context through the policy reasoning network. The personalized adaptation engine is used to dynamically adjust the weight parameters of the emotion recognition model and the decision threshold of the response strategy generation module based on the user's long-term behavioral preferences, emotion regulation habits and feedback history, so as to achieve continuous optimization and personalized adaptation of system behavior.

2. The emotion recognition and response system based on deep learning according to claim 1, characterized in that, The multimodal perception module includes a speech feature extraction unit, a visual feature extraction unit, a physiological signal processing unit, and a text semantic parsing unit. The speech feature extraction unit is used to extract fundamental frequency, speech rate, energy profile, and Mel-frequency cepstral coefficients from the speech signal. The visual feature extraction unit is used to extract facial motion unit intensity sequences, micro-expression temporal features, and head posture parameters from the facial video stream. The physiological signal processing unit is used to extract time-domain and frequency-domain features from heart rate variability, skin conductance, and electromyography signals. The text semantic parsing unit is used to perform word segmentation, part-of-speech tagging, sentiment dictionary matching, and syntactic dependency analysis on the input text, and output a text sentiment tendency vector.

3. The emotion recognition and response system based on deep learning according to claim 1, characterized in that, The emotion state modeling module adopts a multi-branch fusion architecture, including an independent emotion recognition sub-network and a cross-modal attention fusion layer. Each emotion recognition sub-network processes a single modality feature and outputs the emotion probability distribution under that modality. The cross-modal attention fusion layer is used to calculate the correlation weights between the features of each modality, and performs weighted fusion on the outputs of each sub-network according to the weights to generate the final emotion state representation. The emotion state representation includes discrete emotion category labels, continuous emotion dimension values, and emotion recognition confidence scores.

4. The emotion recognition and response system based on deep learning according to claim 3, characterized in that, The cross-modal attention fusion layer employs a gating mechanism to dynamically adjust the fusion weights of each modality based on the confidence level of the currently dominant modality. When the data quality of a certain modality deteriorates or is missing, the system automatically increases the weights of other reliable modalities to maintain the robustness of emotion recognition.

5. The emotion recognition and response system based on deep learning according to claim 1, characterized in that, The context understanding module includes a task intent recognition unit, an environment perception unit, and a social relationship modeling unit. The task intent recognition unit analyzes the task objective corresponding to the user's current interaction behavior and outputs a task type code. The environment perception unit accesses environmental sensor data to identify the current physical environment type, light intensity, noise level, and personnel density. The social relationship modeling unit constructs a dynamic social graph based on the user's historical social interaction data and outputs the intimacy level and role tag of the current interaction object. The context understanding module encodes the above three types of information into a context semantic vector of a unified dimension.

6. The emotion recognition and response system based on deep learning according to claim 1, characterized in that, The response strategy generation module includes a strategy candidate library, a context matching unit, and an action sequence generation unit. The strategy candidate library stores predefined multi-level response action templates, covering language feedback, facial expression simulation, voice tone adjustment, interaction rhythm control, and service behavior triggering. The context matching unit is used to calculate the joint embedding of the current emotion representation and the context semantic vector, and to retrieve the most matching response strategy category from the strategy candidate library. The action sequence generation unit is used to generate specific and executable response action sequences based on the matching results, including action type, execution parameters, and timing arrangement.

7. The emotion recognition and response system based on deep learning according to claim 6, characterized in that, The action sequence generation unit supports multi-objective optimization, taking into account the effects of emotional soothing, task completion efficiency, and user comfort, and optimizes strategy selection in long-term interaction through a reinforcement learning mechanism.

8. The emotion recognition and response system based on deep learning according to claim 1, characterized in that, The personalized adaptation engine includes a user profile building unit, a feedback learning unit, and a model fine-tuning unit. The user profile building unit integrates users' historical emotional response patterns, preferred response types, and interaction satisfaction scores to construct a dynamic user profile vector. The feedback learning unit collects users' explicit and implicit behavioral feedback to the system in real time and calculates response effect evaluation indicators. The model fine-tuning unit incrementally updates the fusion weights of the emotional state modeling module and the matching thresholds of the response strategy generation module based on the evaluation indicators and using an online learning mechanism.

9. The emotion recognition and response system based on deep learning according to claim 1, characterized in that, The system operates within a hierarchical decision-making architecture, comprising a perception layer, a cognition layer, and an execution layer; the perception layer is responsible for the real-time acquisition and feature extraction of multimodal data. The cognitive layer is responsible for emotional state modeling and contextual understanding, and for completing strategy reasoning; the execution layer is responsible for converting the generated response action sequence into specific interactive behaviors and outputting them through the human-computer interaction interface; the layers communicate with each other with low latency through standardized data interfaces to ensure the real-time and coherent nature of the system response.

10. A deep learning-based emotion recognition and response system according to claim 4, characterized in that, The confidence of the dominant modality is evaluated by calculating the entropy value of the probability distribution output by each emotion recognition subnetwork or the maximum probability value output by softmax; the gating mechanism takes the confidence score and data quality index of each modality as input, and outputs the weighting coefficients of each modality in the final fusion through a feedforward neural network, with the sum of the weighting coefficients being 1.

Images