Emotion recognition and response method and system based on AI intelligent customer service

Abstract

The invention relates to an emotion recognition and response method based on AI intelligent customer service, and the method is characterized in that the method comprises the steps: collecting a current session of a user, and carrying out the preprocessing of the current session, and obtaining text data and voice data; encoding the text data and the voice data through a pre-trained hierarchical multi-modal Transform model, dynamically allocating weights of a text encoding result and a voice feature encoding result and performing multi-modal weighted fusion according to a topic and historical emotion intensity of a current session by utilizing a context awareness gating attention mechanism, and generating an emotion state result; and based on the emotional state result, the user portrait vector and the current session state, outputting target response content and response action through a personalized response strategy network. According to the method, the emotion perception flexibility and robustness in a complex interaction scene are improved, and the real-time response capability in a high-concurrency scene is ensured.

Description

Technical Field

[0001] This application relates to the field of intelligent customer service, and in particular to an AI-based intelligent customer service method, system, computer device, and computer-readable storage medium for emotion recognition and response. Background Technology

[0002] Web applications and artificial intelligence technologies are increasingly being used in customer service, especially in after-sales service, pre-sales consultation, and complaint handling. Traditional intelligent customer service systems are mostly based on rule matching or single-modal text analysis technology, responding to user inquiries through keyword matching or simple semantic understanding.

[0003] In related technologies, intelligent customer service systems generally suffer from limited and lagging emotion recognition capabilities. Existing solutions primarily rely on textual information for sentiment analysis, neglecting the rich emotional information contained in acoustic features such as tone of voice, speech rate, and volume. For example, the same phrase "okay" might be interpreted as neutral or agreeable in text alone, but its actual meaning can be drastically different when combined with a tone of helplessness or anger. This modality deficiency makes it difficult for the system to capture the user's true emotional state, especially in complex scenarios such as sarcasm, impatience, or suppressed anger, resulting in a high rate of misjudgment.

[0004] Furthermore, related technologies typically treat each round of dialogue as an independent event, lacking the ability to perceive long-term context and historical interaction habits. When generating responses, standardized pre-set scripts are often used, failing to dynamically adjust based on the user's personalized profile (such as loyalty, historical complaint records) and current emotional intensity. This "one-size-fits-all" mechanical response easily exacerbates user dissatisfaction, leading to escalated complaints, while also increasing the frequency and workload of human agents.

[0005] There is currently no effective solution to the problem of low accuracy in emotion recognition for Web-based intelligent customer service technologies. Summary of the Invention

[0006] This application provides an AI-based intelligent customer service emotion recognition and response method, system, computer device, and computer-readable storage medium to at least solve the problem of low accuracy in emotion recognition for Web-based intelligent customer service in related technologies.

[0007] In a first aspect, embodiments of this application provide an emotion recognition and response method based on AI-powered intelligent customer service, the method comprising: Collect the user's current session and preprocess the current session to obtain text data and voice data; The text data and speech data are encoded by a pre-trained hierarchical multimodal Transformer model. Using a context-aware gated attention mechanism, the weights of the text encoding results and speech feature encoding results are dynamically allocated according to the topic of the current conversation and the intensity of historical emotions, and multimodal weighted fusion is performed to generate emotional state results. Based on the emotional state results, user profile vectors, and current session state, the target response content and response actions are output through a personalized response strategy network.

[0008] In some embodiments, encoding the text data and the speech data using a pre-trained hierarchical multimodal Transformer model includes: In the single-modal feature encoding layer, a pre-trained language model is used to extract the context semantic vector of the text data in the current session, and the context semantic vector is concatenated with the shallow text features and output to the bidirectional long short-term memory network for processing to obtain the text encoding result; By combining a convolutional network with a multi-head attention mechanism, acoustic vectors are extracted from the speech data within the sliding window of the current session to obtain the speech coding result. The multi-head attention mechanism allocates attention based on the energy and pitch variations of the speech frames.

[0009] In some embodiments, a context-aware gated attention mechanism is used to dynamically assign weights to text encoding results and speech feature encoding results based on the topic of the current conversation and the intensity of historical emotions, and then perform multimodal weighted fusion to generate emotional state results, including: Based on the recent multi-round conversation summaries, business tags, and historical sentiment intensity, a conversation context vector is obtained. The conversation context vector is then concatenated with the text encoding result and the speech encoding result to obtain a stacked feature vector. The stacked feature vectors are processed by a gating network to obtain modal score vectors, and the modal score vectors are normalized to obtain text modal weights and speech modal weights. Based on the text modal weights and speech modal weights, the text encoding results and the speech encoding results are subjected to multimodal weighted fusion to obtain a feature fusion representation, and the feature fusion representation is mapped to the emotion state result.

[0010] In some embodiments, the hierarchical multimodal Transformer model is optimized using knowledge distillation techniques, wherein: The complete model trained on a general emotion dataset and a de-identified customer service scenario dataset is defined as the teacher model, and a lightweight student model with fewer parameters than the teacher model is constructed. Through knowledge distillation, the knowledge of the teacher model is transferred to the lightweight student model to obtain the hierarchical multimodal Transformer model for real-time emotion reasoning.

[0011] In some embodiments, the target response content and response action are output through a personalized response strategy network, including: Construct a triplet input containing the current emotional state, user profile vector, and current session state, wherein the user profile vector includes the user's purchase history, loyalty, and number of historical complaints, and the current session state includes currently resolved issues and currently unresolved issues; The triples are input into the personalized response strategy network for processing, and the probability distribution of the optimal response text is output. The target response content is then determined based on the probability distribution.

[0012] In some embodiments, during the process of outputting the target response content and response action through a personalized response strategy network, the method further includes: Construct a structured knowledge graph in the field of intelligent customer service, wherein the knowledge graph contains a triplet of problem-cause-solution; During the generation of the response content and response action, the knowledge graph query technology is used to locate solutions related to the current conversation topic; The solution is integrated into the target response content, and the source link of the solution is included in the target response content.

[0013] In some embodiments, after the target response content and response action are output through a personalized response strategy network, if the emotional state result does not decrease to a first preset threshold, a response escalation mechanism is triggered, specifically including: If the user's emotion category is anger or anxiety, and the emotion intensity value shows an upward trend or exceeds the second preset threshold after receiving the response content, the response strategy will be upgraded from standard soothing to expert intervention or high-privilege solution recommendation. Trigger the security audit process, record session logs, and send an alert notification to the management console.

[0014] Secondly, embodiments of this application provide an AI-based intelligent customer service emotion recognition and response system, the system comprising: a data acquisition module, a recognition module, and a response module, wherein: The acquisition module is used to acquire the user's current session and preprocess the current session to extract speech features and text features; The recognition module is used to encode the speech features and the text features through a pre-trained hierarchical multimodal Transformer model, and to dynamically allocate the weights of the text feature encoding results and the speech feature encoding results according to the topic of the current conversation and the intensity of historical emotions, and perform multimodal weighted fusion to generate the emotional state result. The response module is used to output target response content and response actions through a personalized response strategy network based on the emotional state result, user profile vector and current session state.

[0015] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0016] Fourthly, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0017] Compared to related technologies, the emotion recognition and response method based on AI-powered intelligent customer service provided in this application addresses the issue of low accuracy in single-modal emotion recognition by constructing a hierarchical multimodal Transformer model and a context-aware gated attention mechanism. This achieves deep fusion and dynamic weight allocation of dual-modal features of speech and text, effectively solving the problem of low accuracy in single-modal emotion recognition. Furthermore, by introducing conversation context and user profile vectors, a personalized response strategy network is constructed, enabling dynamic adaptation and accurate recommendation of response content and actions. This method enhances the flexibility and robustness of emotion perception in complex interaction scenarios, ensures real-time response capabilities in high-concurrency scenarios, and effectively reduces complaint risks when combined with a dynamic response escalation mechanism, significantly improving the intelligence level of customer service and user experience. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of an AI-based intelligent customer service emotion recognition and response method according to an embodiment of this application; Figure 2 This is a structural block diagram of an AI-based intelligent customer service emotion recognition and response system according to an embodiment of this application; Figure 3 This is a schematic diagram of the internal structure of a computer device according to an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0020] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0021] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0022] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0023] In related technologies, the development model of intelligent customer service typically involves simply overlaying text recognition with rule-based responses. For example, when handling user complaints, the system only triggers fixed apologies based on keywords, failing to perceive changes in the user's level of anger in their voice. This approach results in the system mechanically responding even when the user is emotionally agitated, not only failing to resolve the issue but also exacerbating the conflict. This not only reduces service quality but also forces human customer service representatives to frequently intervene in basic calming tasks that could be handled by machines, leading to high operating costs.

[0024] In view of this, embodiments of this application provide an emotion recognition and response method based on AI-powered intelligent customer service. Figure 1 This is a flowchart of an AI-based intelligent customer service emotion recognition and response method according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps: Step S101: Collect the user's current session and preprocess the current session to obtain text data and voice data.

[0025] In this embodiment, data acquisition and preprocessing are fundamental to achieving high-precision emotion recognition. The user's current conversation is accessed in real-time via a full-duplex voice channel or instant messaging interface. The raw input data is first separated and cleaned.

[0026] For the speech part, a noise reduction algorithm is adopted to remove background environmental noise, and the voice activity detection (VAD) technique is used to segment the continuous speech stream into independent speech frame segments. Subsequently, the automatic speech recognition (ASR) technique is applied to convert the speech into preliminary text, while retaining the original speech waveform for subsequent acoustic feature extraction.

[0027] For the text part (including the text directly input by the user or the text transcribed from speech), word segmentation is performed, stop words (such as function words without actual semantics like "de", "shi", etc.) are removed, and the explicit sentiment words in the text are preliminarily annotated using a pre-set sentiment dictionary, finally obtaining standardized text data and speech data. This process ensures the data quality of the input to the model and reduces the interference of noise on subsequent reasoning.

[0028] Step S102, encode the text data and speech data through a pre-trained hierarchical multi-modal Transformer model.

[0029] This step aims to extract deep features from a single modality. The model architecture adopted in this embodiment includes a single-modal feature encoding layer.

[0030] For the text data, a pre-trained language model (such as RoBERTa-wwm-extBase) is used to extract the context semantic vector of the text data in the current session. This model is pre-trained on a large-scale corpus and can capture the deep semantics of words in a specific context.

[0031] In addition, to enhance the capture of sentiment details, the context semantic vector is also concatenated with text shallow features (such as negation word statistics, sentiment dictionary matching degree,词性特征). The concatenated vector is output to a bidirectional long short-term memory network (Bi-LSTM) for temporal processing. It should be noted that the bidirectional LSTM can utilize both past and future context information simultaneously, thereby more accurately capturing the emotional dynamic changes in the sequence, and finally obtaining a text encoding result containing rich semantic and temporal information.

[0032] In addition, for the speech data, it is processed through a convolutional neural network (CNN) combined with a multi-head attention mechanism. Specifically, a sliding window (such as a 2.5-second window with a 0.5-second step size) is set in the current session, and acoustic vectors such as Mel-frequency cepstral coefficients (MFCC) are extracted within the window. Subsequently, a 4-layer one-dimensional convolutional network is used to extract high-order features and input them into the multi-head attention mechanism module.

[0033] In this process, the multi-head attention mechanism is configured to allocate attention based on changes in the energy and pitch of speech frames, focusing on frames with sudden energy changes or dramatic pitch fluctuations, as these segments typically carry the user's strongest emotional expressions (such as shouts of anger or whispers of sadness), thus achieving highly robust speech coding results.

[0034] Step S103: Using a context-aware gating attention mechanism, the weights of the text encoding results and the speech feature encoding results are dynamically allocated and multimodal weighted fusion is performed based on the topic of the current conversation and the intensity of historical emotions to generate the emotional state results.

[0035] This step aims to address the problems of fixed weights and lack of context adaptability in traditional multimodal fusion, and specifically includes the following steps: First, construct the conversation context vector. Aggregate the most recent multi-turn (e.g., 5-turn) conversation summaries, current business tags (such as "refund", "inquiry", "complaint"), and historical sentiment intensity sequences, and obtain the conversation context vector reflecting the current dialogue background through vectorization.

[0036] Next, the session context vector is concatenated with the text encoding result and speech encoding result obtained in step S102 to form a stacked feature vector.

[0037] Subsequently, the stacked feature vectors are input into a gating network. The gating network calculates the importance score of each modality in the current context, i.e., the modality score vector, through fully connected layers and activation functions. This modality score vector is then subjected to Softmax normalization to obtain the text modality weights and speech modality weights. For example, in a complaint scenario where the text content is brief and ambiguous but the tone of voice is intense, the context-aware gating attention mechanism automatically assigns higher weights to the speech modality; while in a consultation scenario where the voice is flat but the text logic is rigorous, higher weights are assigned to the text modality.

[0038] Finally, based on the calculated weights, the text encoding results and speech encoding results are weighted and summed or concatenated to obtain a feature fusion representation. This feature fusion representation is then further interactively learned by a Transformer encoder and mapped to the final emotional state result through a fully connected classification layer. In this embodiment, the result includes not only the emotional category (such as anger, anxiety, sadness, neutrality, and joy), but also quantified emotional intensity values ​​(0-1 range) and confidence intervals.

[0039] Step S104: Based on the emotional state results, user profile vectors, and current session state, output the target response content and response action through the personalized response strategy network.

[0040] This step enables the transformation from "identification" to "decision-making," ensuring the relevance of the response through a personalized response strategy network.

[0041] First, construct the triple input. The triple includes: 1) the current emotional state (output by step S103); 2) the user profile vector, which specifically covers the user's purchase history (average order value, frequency), loyalty level (normal, VIP) and number of historical complaints; 3) the current conversation state, which clearly marks the currently resolved issues, currently unresolved issues, and matters pending confirmation.

[0042] Next, the triples are input into a personalized response strategy network. This network is trained through reinforcement learning or imitation learning, aiming to maximize customer satisfaction. After processing, the network outputs a probability distribution of the optimal response text and determines the target response content based on this probability distribution. For example, in a scenario involving a VIP user in an "angry" state, the strategy network will tend to select high-priority soothing phrases and compensation solutions.

[0043] In one exemplary embodiment, the response strategies for different emotions are shown in Table 1 below:

[0044] Table 1: Intensity, Type, and Response Strategies of Different Emotions It should be noted that, in order to meet the real-time requirements of high concurrency and low latency in customer service scenarios, this embodiment uses knowledge distillation technology to optimize the model.

[0045] First, a teacher model is constructed. A full-level hierarchical multimodal Transformer model trained on a large-scale general sentiment dataset and a de-identified customer service scenario dataset (containing industry-specific terminology and emotional expressions) is defined as the teacher model. The teacher model has a large number of parameters and high accuracy, but its inference speed is relatively slow.

[0046] Secondly, construct a lightweight student model. Design a student model structure with a much smaller number of parameters than the teacher model (e.g., reduce the number of Transformer layers or the dimension of hidden layers).

[0047] Through a knowledge distillation process, the student model not only learns real labels but also mimics the soft-label output (probability distribution) of the teacher model, transferring the teacher model's "knowledge" to the student model. The result is a hierarchical multimodal Transformer model (student model) used for online real-time sentiment inference. Testing showed that the optimized model has significantly reduced inference time (e.g., <85ms) and can support thousands of concurrent requests.

[0048] Furthermore, this embodiment also incorporates knowledge graph technology during the identification process. A structured knowledge graph for the intelligent customer service domain is constructed, containing "problem-cause-solution" triples. When generating response content, knowledge graph query technology is used to locate standard solutions related to the current conversation topic, and these solutions are naturally integrated into the target response content, along with source links, to ensure the professionalism and traceability of the response.

[0049] In addition, in order to optimize the overall process, after outputting the target response content and response action through the personalized response strategy network, the process does not end with a single response, but enters a dynamic monitoring closed loop.

[0050] Continuously monitor user emotional changes after receiving a response. If the emotional intensity in the emotional state result does not decrease to the first preset threshold, meaning the user's negative emotions are not alleviated, a response escalation mechanism will be triggered. Specifically, if the user's emotion category is "anger" or "anxiety," and the emotional intensity value shows an upward trend or remains above the second preset threshold (high-risk threshold) after receiving the response content, the strategy will be automatically adjusted: the response strategy will be escalated from standard reassurance to senior customer service intervention (transfer to a human) or a high-privilege solution will be recommended (such as expedited access or special compensation application).

[0051] Simultaneously, a security audit process is triggered, recording the entire conversation log, especially interaction data during periods of heightened emotion, and sending an alert notification to the management side. If sensitive words or unusually high-risk emotions are detected, the risk control and security department will also be notified immediately for intervention, thereby effectively reducing the risk of escalating complaints and ensuring service security.

[0052] Through steps S101 to S104 above, the technical solution of this application combines a deep learning model with an expert knowledge graph, utilizing a hierarchical multimodal Transformer and a context-aware gating mechanism to achieve accurate capture and dynamic response to user emotions. This solution not only supports deep semantic alignment of text and speech but also achieves lightweight deployment through knowledge distillation. Compared to traditional technologies, this solution significantly improves customer satisfaction, problem resolution rate, and operational efficiency, effectively reducing the pressure on human agents and the operating costs for enterprises.

[0053] This embodiment also provides an AI-based intelligent customer service system for emotion recognition and response. Figure 2 This is a structural block diagram of an AI-based intelligent customer service emotion recognition and response system according to an embodiment of this application. Figure 2 As shown, the system includes a data acquisition module 20, an identification module 21, and a response module 22.

[0054] The data acquisition module 20 serves as the system's front-end entry point, used to collect the user's current session. Technically, this module interfaces with the customer service interface (using the React / Vue.js framework) through a high-performance application programming interface gateway (built using Go or Java, such as FastAPI / SpringBoot), supporting high-concurrency traffic access. The data acquisition module 201 internally integrates a data preprocessing unit to process the collected voice and text streams in real time, specifically including: Noise reduction and VAD: For speech data, noise reduction algorithms are applied to remove background noise, and effective speech segments are extracted through silence segmentation (VAD). ASR and text standardization utilize speech recognition (ASR) technology to transcribe speech into text, and combine word segmentation and stop word removal techniques to standardize and clean the text data. Furthermore, to cover long-tail and unexpected situations, this module also incorporates simulated dialogue data collected from a crowdsourcing platform. This data undergoes cross-validation and multi-label annotation (emotion category, intensity level 1-5) to provide high-quality input for subsequent model training.

[0055] The recognition module 21 is used to encode speech features and text features through a pre-trained hierarchical multimodal Transformer model, and to dynamically allocate modality weights and generate emotional state results using a context-aware gated attention mechanism.

[0056] This module is built on the TensorFlow or PyTorch framework. The model training adopts a "pre-training + fine-tuning" strategy. First, the basic structure is trained on a general dataset, and then the model is deeply fine-tuned using anonymized customer service scenario data, focusing on optimizing the recognition performance for specific business emotions (such as complaints and anxiety).

[0057] Furthermore, to meet real-time requirements, this module incorporates a knowledge distillation inference engine. It distills the massive full teacher model into a lightweight student model (e.g., 12M parameters), encapsulates it as a Docker container, and deploys it on a cloud cluster via a Kubernetes orchestration platform. Utilizing GPU-accelerated inference, the time for a single inference iteration is controlled to the millisecond level (e.g., <85ms), effectively supporting scenarios with thousands of concurrent requests. The output includes not only the emotion category but also the emotion intensity and confidence interval, providing quantitative evidence for response decisions.

[0058] The response module 23 is used to output the target response content and response action through a personalized response strategy network based on the emotional state result, user profile vector and current session state.

[0059] This module runs a personalized response policy network trained using reinforcement learning or imitation learning. The network iterates using the degree of customer emotional decline or satisfaction rating as the reward function, and outputs the optimal response probability distribution based on the input triples (emotion, profile, and conversation state).

[0060] In addition, this module integrates a knowledge graph query engine and connects to a graph database (Neo4j). When generating a response, it uses graph technology to quickly locate the "problem-cause-solution" triple, ensuring that the generated content is professional and accurate, and automatically includes reference source links.

[0061] Specifically, the module has built-in dynamic monitoring logic. If it detects that the intensity of a user's anger does not decrease after receiving reassurance (i.e., the response is ineffective), it will automatically trigger an escalation mechanism, such as recommending the intervention of a senior human agent or providing a high-level compensation plan, thereby effectively reducing the risk of escalation of complaints.

[0062] To support the efficient operation of the above modules, the system has constructed a layered data storage architecture, in which: Relational databases, using PostgreSQL or MySQL, store structured user identity information, transaction records, and human agent performance data; Non-relational databases, such as MongoDB or Cassandra, are used to store massive amounts of unstructured session recordings, text logs, and model inference logs. High-speed caching uses Redis in-memory database to cache user profiles, model weights, and real-time sentiment states of sessions, enabling low-latency data querying and state synchronization.

[0063] Finally, this system incorporates deep security enhancements in its architecture design to ensure data privacy and compliance: Encrypted transmission and storage: All API calls and data transmissions are end-to-end encrypted using the TLS 1.2+ protocol; sensitive information stored in the database (such as user IDs and payment information) is statically encrypted using the AES-256 standard.

[0064] Before data enters the model training process, user identities are de-identified and anonymized using hash algorithms or random number replacement techniques. Role-based access control (RBAC) and multi-factor authentication are implemented to strictly restrict access to sensitive data; at the same time, a Web Application Firewall (WAF) is deployed to filter malicious traffic (such as SQL injection), and all operation logs are recorded in detail for security auditing.

[0065] Finally, the system also includes a feedback optimization module, which monitors the system's actual performance (such as emotional relief rate and first-time problem resolution rate) through online A / B testing. It collects feedback data after interactions and updates the parameters of the identification and response modules through periodic retraining, enabling the system to self-evolve. Based on the implementation of this architecture, this system can significantly improve customer satisfaction (expected increase of 15%-20%) and the first-time problem resolution rate (expected increase of 20%-25%), greatly reduce the energy consumption and operating costs of human agents, and effectively identify and reduce the risk of malicious complaints, achieving a dual optimization of service quality and operational efficiency.

[0066] In addition, the system includes a feedback optimization module, which collects user feedback and emotional changes after interaction. This system combines deep learning models with expert knowledge graphs, utilizing hierarchical multimodal Transformers and context-aware gating mechanisms to achieve accurate capture and dynamic response to user emotions. This solution not only supports deep semantic alignment of text and speech but also achieves lightweight deployment through knowledge distillation. Compared to traditional technologies, this solution significantly improves customer satisfaction, problem resolution rate, and operational efficiency, effectively reducing the pressure on human agents and lowering enterprise operating costs.

[0067] In one embodiment, Figure 3 This is a schematic diagram of the internal structure of a computer device according to an embodiment of this application, such as... Figure 3 As shown, a computer device is provided, which can be a cloud server or an edge computing node. The computer device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores an operating system, computer programs, and related knowledge graph databases and user profile databases. The processor provides powerful computing capabilities to support inference operations of deep learning models; the network interface is used for high-concurrency data communication with user terminals and third-party customer service systems; and the internal memory provides a high-speed caching environment for the operation of the operating system and computer programs. When the processor executes the stored computer programs, it implements a database of emotion recognition and response methods based on AI-powered intelligent customer service.

[0068] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0069] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by hardware related to computer program instructions. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0070] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for emotion recognition and response based on AI-powered intelligent customer service, characterized in that, The method includes: Collect the user's current session and preprocess the current session to obtain text data and voice data; The text data and speech data are encoded by a pre-trained hierarchical multimodal Transformer model. Using a context-aware gated attention mechanism, the weights of the text encoding results and speech feature encoding results are dynamically allocated according to the topic of the current conversation and the intensity of historical emotions, and multimodal weighted fusion is performed to generate emotional state results. Based on the emotional state results, user profile vectors, and current session state, the target response content and response actions are output through a personalized response strategy network.

2. The method according to claim 1, characterized in that, Encoding the text data and the speech data using a pre-trained hierarchical multimodal Transformer model includes: In the single-modal feature encoding layer, a pre-trained language model is used to extract the context semantic vector of the text data in the current session, and the context semantic vector is concatenated with the shallow text features and output to the bidirectional long short-term memory network for processing to obtain the text encoding result; By combining a convolutional network with a multi-head attention mechanism, acoustic vectors are extracted from the speech data within the sliding window of the current session to obtain the speech coding result. The multi-head attention mechanism allocates attention based on the energy and pitch changes of the speech frame segments.

3. The method according to claim 2, characterized in that, By utilizing a context-aware gated attention mechanism, the weights of text encoding results and speech feature encoding results are dynamically assigned based on the topic of the current conversation and the intensity of historical emotions, and then multimodal weighted fusion is performed to generate emotional state results, including: Based on the recent multi-round conversation summaries, business tags, and historical sentiment intensity, a conversation context vector is obtained. The conversation context vector is then concatenated with the text encoding result and the speech encoding result to obtain a stacked feature vector. The stacked feature vectors are processed by a gating network to obtain modal score vectors, and the modal score vectors are normalized to obtain text modal weights and speech modal weights. Based on the text modal weights and speech modal weights, the text encoding results and the speech encoding results are subjected to multimodal weighted fusion to obtain a feature fusion representation, and the feature fusion representation is mapped to the emotion state result.

4. The method according to claim 2, characterized in that, The hierarchical multimodal Transformer model is optimized using knowledge distillation techniques, wherein: The complete model trained on a general emotion dataset and a de-identified customer service scenario dataset is defined as the teacher model, and a lightweight student model with fewer parameters than the teacher model is constructed. Through knowledge distillation, the knowledge of the teacher model is transferred to the lightweight student model to obtain the hierarchical multimodal Transformer model for real-time emotion reasoning.

5. The method according to claim 1, characterized in that, The personalized response strategy network outputs the target response content and response actions, including: Construct a triplet input containing the current emotional state, user profile vector, and current session state, wherein the user profile vector includes the user's purchase history, loyalty, and number of historical complaints, and the current session state includes currently resolved issues and currently unresolved issues; The triples are input into the personalized response strategy network for processing, and the probability distribution of the optimal response text is output. The target response content is then determined based on the probability distribution.

6. The method according to claim 5, characterized in that, In the process of outputting the target response content and response action through a personalized response strategy network, the method further includes: Construct a structured knowledge graph in the field of intelligent customer service, wherein the knowledge graph contains a triplet of problem-cause-solution; During the generation of the response content and response action, the knowledge graph query technology is used to locate solutions related to the current conversation topic; The solution is integrated into the target response content, and the source link of the solution is included in the target response content.

7. The method according to claim 1, characterized in that, After the target response content and response action are output through the personalized response strategy network, if the emotional state result does not decrease to a first preset threshold, a response escalation mechanism is triggered, specifically including: If the user's emotion category is anger or anxiety, and the emotion intensity value shows an upward trend or exceeds the second preset threshold after receiving the response content, the response strategy will be upgraded from standard soothing to expert intervention or high-privilege solution recommendation. Trigger the security audit process, record session logs, and send an alert notification to the management console.

8. An emotion recognition and response system based on AI-powered intelligent customer service, characterized in that, The system includes: a data acquisition module, an identification module, and a response module, wherein: The acquisition module is used to acquire the user's current session and preprocess the current session to extract speech features and text features; The recognition module is used to encode the speech features and the text features through a pre-trained hierarchical multimodal Transformer model, and to dynamically allocate the weights of the text feature encoding results and the speech feature encoding results according to the topic of the current conversation and the intensity of historical emotions, and perform multimodal weighted fusion to generate the emotional state result. The response module is used to output target response content and response actions through a personalized response strategy network based on the emotional state result, user profile vector and current session state.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

Images