Intelligent customer service interaction method and system based on multi-modal emotion recognition
Patent Information
- Application Number
- CN202610863572.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-06-15
- Publication Date
- 2026-09-15
Smart Images
Figure CN122760084A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to an intelligent customer service interaction method and system based on multimodal emotion recognition. Background Technology
[0002] With the application of artificial intelligence in the service sector, intelligent customer service has become an important support for enterprises to improve efficiency and reduce costs. However, existing systems have significant shortcomings and cannot meet the interactive needs of "humanization" and "efficiency." The core problem lies in the rigid interaction logic, which relies heavily on text keyword matching. For example, e-commerce customer service only triggers fixed responses based on "shipping" and "refund," failing to perceive user emotions. Emotion recognition is also simplistic, relying solely on text emotion words or facial expression recognition, which is prone to misjudgment and cannot accurately capture users' true emotions. Furthermore, the lack of dynamic interaction strategies, with a uniform process applied regardless of user emotions, leads to low problem-solving efficiency and a high rate of manual transfer. Summary of the Invention
[0003] To address the problem that existing intelligent customer service systems cannot adapt to users' diverse emotional needs, this application provides an intelligent customer service interaction method and system based on multimodal emotion recognition.
[0004] Firstly, this application provides an intelligent customer service interaction method based on multimodal emotion recognition, employing the following technical solution: An intelligent customer service interaction method based on multimodal emotion recognition includes: Real-time acquisition of multimodal data during user interaction; Extract multimodal sentiment features from the multimodal data; The multimodal emotion features are input into the multimodal emotion fusion and recognition model to obtain the user's emotion category and emotion intensity; Based on the emotion category and emotion intensity, an interaction strategy is matched from the strategy rule base; The interaction strategy is executed, and feedback information from the user is received. The feedback information is then used to iteratively optimize the emotion fusion recognition model and the strategy rule base.
[0005] By adopting the above technical solution, multimodal user data is collected, emotional features are extracted and input into the fusion recognition model to obtain the emotional category and intensity, then the corresponding interaction strategy is matched, and finally the model and strategy library are iteratively optimized using user feedback. This achieves emotional and intelligent customer service interaction, effectively solving the problems of existing customer service being unable to perceive emotions and rigid interaction, improving user experience and problem-solving efficiency, and continuously improving system adaptability through iterative optimization.
[0006] In a preferred embodiment, this application can be further configured such that: the multimodal data includes speech data, text data, and facial image data; the extraction of multimodal emotional features from the multimodal data includes: Extract speech emotion features from the speech data, the speech emotion features including fundamental frequency, short-time energy and speech rate; Extract text sentiment features from the text data, the text sentiment features including sentiment word weights and sentence strength; Emotional features are extracted from the facial image data, including eye opening and mouth corner curvature. The speech emotion features, the text emotion features, and the image emotion features constitute the multimodal emotion features.
[0007] By adopting the above technical solution, it is clear that multimodal data includes three categories: speech, text, and facial images. Specific emotional features of each type of data are extracted to form multimodal emotional features, making emotional feature extraction more targeted and comprehensive. This avoids the limitations of single-modal features, provides reliable data support for subsequent accurate emotion recognition, and improves the accuracy of emotion recognition.
[0008] In a preferred embodiment, this application can be further configured such that the structure of the multimodal emotion fusion recognition model includes: An input layer is used to receive the multimodal emotional features; An attention layer is used to assign weights to each modal feature in the multimodal emotion features to obtain weighted modal features; The fusion layer is used to concatenate and fuse the weighted features of each modality to generate a fused feature vector; A fully connected layer is used to perform a nonlinear transformation on the fused feature vector to obtain a dimensionally compressed fused feature vector; The output layer is used to output the sentiment category classification result and the sentiment intensity quantification value based on the fused feature vector after dimensionality compression.
[0009] By adopting the above technical solutions, the constructed multimodal emotion fusion recognition model achieves reasonable weighting and effective fusion of multimodal emotion features through the synergistic effect of the input layer, attention layer, fusion layer and other structures. It can accurately output the user's emotion category and emotion intensity, solve the problem of low accuracy of existing single-modal recognition, and provide accurate emotional basis for subsequent interaction strategy matching.
[0010] In a preferred embodiment, this application can be further configured such that when the illumination conditions are lower than a preset illumination threshold, the attention layer reduces the modal feature weights of the facial image data according to a preset difference.
[0011] By adopting the above technical solution, for scenarios with poor lighting conditions, an attention layer is set to dynamically adjust the modal weights of facial image data, avoiding misjudgment of image emotional features caused by lighting effects, further optimizing the recognition accuracy of the multimodal emotion fusion recognition model, and improving the model's adaptability and reliability in complex environments.
[0012] In a preferred example, this application can be further configured such that: matching interaction strategies from a policy rule base based on the emotion category and emotion intensity includes: Based on a preset priority ranking, the target emotion category with the highest priority is determined from the emotion categories; Based on the target emotion category and the emotion intensity of the target emotion category, an interaction strategy is matched from the strategy rule base; The strategy rule base stores the correspondence between combinations of different emotion categories and different emotion intensities and interaction strategies.
[0013] By adopting the above technical solution, the highest priority target emotion category is determined, and the interaction strategy is matched from the strategy rule base based on its emotion intensity. The logic and basis of strategy matching are clarified, and the precise correspondence between the interaction strategy and the user's emotional state is achieved. This solves the problem that the existing customer service interaction strategy is too simple and cannot adapt to different emotional needs, and improves the pertinence and efficiency of the interaction.
[0014] In a preferred embodiment, this application can be further configured as follows: executing the interaction strategy and receiving feedback information from the user, and iteratively optimizing the emotion fusion recognition model and the strategy rule base using the feedback information, includes: A voice response with a tone matching the emotion of the interaction strategy is synthesized, and the text response is rendered to the client in real time; After the interaction ends, a feedback pop-up window appears on the interactive interface to collect the user's feedback information on the accuracy of emotion recognition and the satisfaction with problem resolution. The feedback information includes ratings and text feedback. The feedback information is labeled as correct or incorrect, added to the training set, and the multimodal emotion fusion recognition model is retrained. The policy rule base is adjusted simultaneously.
[0015] By adopting the above technical solutions, voice and text responses that match emotions are output. User feedback is collected through feedback pop-ups and used for iterative optimization of models and strategy libraries. This not only improves the humanized experience of user interaction, but also continuously corrects emotion recognition biases, optimizes interaction strategies, achieves dynamic improvement in system performance, and reduces manual operation costs.
[0016] In a preferred embodiment, this application can be further configured such that the real-time acquisition of multimodal data during user interaction includes: Voice data is collected using streaming transmission. Text data is collected based on a no-input trigger duration mechanism, and the text data is preprocessed. Image data is acquired, face detection is performed on the image data, and keyframes containing faces are extracted as facial image data.
[0017] By adopting the above technical solutions, the specific acquisition methods for multimodal data were clarified. Voice was acquired through streaming transmission, text was acquired and preprocessed through a no-input trigger duration mechanism, and key frames of facial images were extracted through face detection. This ensured the real-time performance, accuracy, and effectiveness of multimodal data acquisition, providing a high-quality data foundation for subsequent emotion feature extraction and recognition, and guaranteeing the stable operation of the entire interaction method.
[0018] Secondly, this application provides an intelligent customer service interaction system based on multimodal emotion recognition, employing the following technical solution: An intelligent customer service interaction system based on multimodal emotion recognition includes: The multimodal data acquisition module is used to collect multimodal data during user interaction in real time. The sentiment feature extraction module is used to extract multimodal sentiment features from the multimodal data; The emotion fusion and recognition module is used to input the multimodal emotion features into the multimodal emotion fusion and recognition model to obtain the user's emotion category and emotion intensity; An interaction decision module is used to match interaction strategies from a strategy rule base based on the emotion category and emotion intensity. The interaction execution and feedback module is used to execute the interaction strategy, receive feedback information from the user, and iteratively optimize the emotion fusion recognition model and the strategy rule base using the feedback information.
[0019] In a preferred example, this application may be further configured such that the emotion categories include anger, anxiety, confusion, neutrality, satisfaction, and disappointment.
[0020] In a preferred example, this application can be further configured such that the interaction strategies in the strategy rule base include push reassurance messages, simplified verification process, manual transfer, step-by-step guidance, problem focus, routine and efficient response, thank-you messages, and related service recommendations.
[0021] Thirdly, this application provides an electronic device that adopts the following technical solution: At least one processor; Memory; At least one application, wherein the at least one application is stored in memory and configured to be executed by at least one processor, the at least one application being configured to: execute the intelligent customer service interaction method based on multimodal emotion recognition as described in any of the first aspects.
[0022] Fourthly, this application provides a computer-readable storage medium, which adopts the following technical solution: A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the intelligent customer service interaction method based on multimodal emotion recognition as described in any of the first aspects.
[0023] In summary, this application includes the following beneficial technical effects: This application collects multimodal user data, extracts emotional features, and inputs them into a fusion recognition model to obtain emotional categories and intensities. Then, it matches corresponding interaction strategies and uses user feedback to iteratively optimize the model and strategy library. This achieves emotional and intelligent customer service interaction, effectively solving the problems of existing customer service systems being unable to perceive emotions and having rigid interactions. It improves user experience and problem-solving efficiency, while continuously improving system adaptability through iterative optimization. Attached Figure Description
[0024] Figure 1 This is a flowchart illustrating an intelligent customer service interaction method based on multimodal emotion recognition provided in an embodiment of this application. Figure 2 This is a schematic diagram of the structure of an intelligent customer service interaction system based on multimodal emotion recognition provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0025] The following is in conjunction with the appendix Figure 1 To be continued Figure 3 This application will be described in further detail.
[0026] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they fall within the scope of the claims of this application.
[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0028] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.
[0029] It should be noted that all data interaction processes involved in this application have corresponding transmission protocols, including authorized data collection and use, and both parties involved in the data interaction have completed data authorization through the execution of the protocol.
[0030] This application provides an intelligent customer service interaction method based on multimodal emotion recognition, such as... Figure 1 As shown, this method specifically focuses on the intelligent customer service sub-domain, especially involving emotion recognition technology based on the fusion of three modalities: voice, text, and image, as well as dynamic interaction strategy adjustment technology driven by emotional state. It is applicable to intelligent customer service systems in multiple scenarios such as e-commerce, construction industry, and government affairs.
[0031] This application provides an embodiment of an intelligent customer service interaction method based on multimodal emotion recognition, applied to an intelligent customer service interaction system based on multimodal emotion recognition, such as... Figure 2 As shown, system 200 includes a multimodal data acquisition module 201, an emotion feature extraction module 202, an emotion fusion and recognition module 203, an interactive decision-making module 204, and an interactive execution and feedback module 205.
[0032] The method provided in this application embodiment is executed by an electronic device, which can be a server or a terminal device. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, tablet computer, laptop computer, desktop computer, etc., but is not limited to these. The terminal device and the server can be directly or indirectly connected via wired or wireless communication. This application embodiment does not impose any limitations on this connection. The method includes steps S101-S105, wherein: S101. Real-time acquisition of multimodal data during user interaction.
[0033] The multimodal data acquisition module is used to acquire multimodal data, including voice data, text data, and facial image data.
[0034] S102. Extract multimodal sentiment features from multimodal data.
[0035] Specifically, the emotion feature extraction module is used to extract speech emotion features from speech data, text emotion features from text data, and image emotion features from facial image data. Speech emotion features, text emotion features, and image emotion features constitute multimodal emotion features.
[0036] S103. Input the multimodal emotion features into the multimodal emotion fusion recognition model to obtain the user's emotion category and emotion intensity.
[0037] Specifically, the multimodal emotion fusion recognition model is deployed in the emotion fusion recognition module. The multimodal emotion fusion recognition model is a pre-trained model, including an input layer, an attention layer, a fully connected layer, and an output layer, which is used to fuse features from three modalities and output the user's real-time emotion category and quantified emotion intensity.
[0038] S104. Match interaction strategies from the strategy rule base based on emotion category and emotion intensity.
[0039] Specifically, the strategy rule base is a pre-built system stored in the interaction decision module of the electronic device. It stores the correspondence between different combinations of emotion categories and intensities and the corresponding interaction strategies. The interaction strategies in the strategy rule base include push notifications of reassurance messages, simplified verification processes, manual transfer, step-by-step guidance, focused questioning, standard and efficient responses, thank-you messages, and related service recommendations. The interaction decision module matches the corresponding interaction strategy from the strategy rule base based on the current user's emotion category and intensity.
[0040] S105. Execute the interaction strategy and receive user feedback information, and use the feedback information to iteratively optimize the emotion fusion recognition model and strategy rule base.
[0041] Specifically, the interaction execution and feedback module is used to execute interaction strategies and optimize the emotion fusion recognition model and strategy rule base. Executing interaction strategies includes outputting Chinese responses and text recovery, and user feedback information includes ratings and text feedback.
[0042] The embodiments of the present application collect user's multimodal data, extract sentiment features, input the features into a fusion recognition model to obtain sentiment categories and intensities, then match corresponding interaction strategies, and finally use user feedback to iteratively optimize the model and the strategy library. This realizes emotionalization and intelligentization of customer service interaction, effectively solves the problems that existing customer service cannot perceive emotions and has rigid interaction, improves user experience and problem-solving efficiency, and continuously improves the adaptability of the system through iterative optimization.
[0043] According to a possible implementation of the embodiment of the present application, collecting multimodal data in real time during a user interaction process comprises: Collecting voice data by adopting a streaming transmission method; Collecting text data based on a no-input trigger duration mechanism, and preprocessing the text data; Collecting image data, performing face detection on the image data, and intercepting key frames containing faces as facial image data.
[0044] In this embodiment, WebRTC technology is used to establish a real-time audio and video communication channel between the client and the electronic device, and audio collection parameters are configured, including a sampling rate of 16kHz, a bit depth of 16bit, and the number of channels set to mono. A continuous voice stream is sliced according to a time length of 200ms per segment, each voice segment is independently packaged and transmitted to the electronic device in real time in a streaming manner. Through the streaming transmission mechanism, simultaneous collection and transmission of voice data is realized, and subsequent processing can be started without waiting for the user to finish speaking.
[0045] A no-input trigger duration mechanism is established, and the trigger duration threshold is set to 3s. When it is detected that the user has no new text input for 3 consecutive seconds, text data collection is automatically triggered. The Jieba word segmentation tool is used to perform word segmentation processing on the collected text, and common stop words without emotional meaning such as "de" and "le" are deleted.
[0046] The camera of the user's device is called through the OpenCV library, and image collection parameters are configured, including a resolution of 640×480 and a frame rate of 15fps, and each frame image in the user's video stream is continuously captured. The MTCNN (Multi-task Cascaded Convolutional Networks) algorithm is used to perform face detection on each frame of image, identify the face position in the image, and output the coordinates of the face bounding box. From consecutive frames where faces are detected, frames that contain a complete face and whose facial definition meets preset conditions are selected as key frames. The key frames can be used as facial image data, or the image can be cropped according to the coordinates of the face bounding box, and a sub-image only containing the face region is intercepted as facial image data.
[0047] This application's embodiments clarify the specific acquisition method for multimodal data. By acquiring voice through streaming transmission, acquiring and preprocessing text through a no-input trigger duration mechanism, and capturing key frames of facial images through face detection, the real-time, accuracy, and effectiveness of multimodal data acquisition are ensured. This provides a high-quality data foundation for subsequent emotional feature extraction and recognition, and guarantees the stable operation of the entire interaction method.
[0048] One possible implementation of this application embodiment is that the multimodal data includes voice data, text data, and facial image data; extracting multimodal emotional features from the multimodal data includes: Extract speech emotion features from speech data, which include fundamental frequency, short-time energy, and speech rate; Extracting text sentiment features from text data, including sentiment word weights and sentence strength; Extracting image emotion features from facial image data, including eye opening and mouth corner curvature; Among them, speech emotion features, text emotion features, and image emotion features constitute multimodal emotion features.
[0049] In this embodiment, for speech data, the Librosa audio analysis library is used to process the speech signal. The YIN algorithm is used to calculate the fundamental frequency (f0) of the speech signal. The YIN algorithm finds the delay point that minimizes the difference function value by calculating the difference function of the signal. The reciprocal of this delay point is the fundamental frequency, and the fundamental frequency value of each frame of speech is output. The short-time energy of the speech signal within each frame is calculated by summing the squares of the amplitudes of each sampling point within the frame. The short-time energy reflects the loudness change of the speech and is positively correlated with the intensity of the user's emotions. Syllable boundaries or word boundaries in the speech signal are identified, and the number of syllables per unit time is counted. A threshold of 12 syllables / second is used to distinguish emotions. A speech rate exceeding 12 syllables / second is judged as a relatively fast speech rate (possibly related to anxiety or excitement), and a speech rate below this threshold is judged as normal or relatively slow speech rate. The extracted fundamental frequency, short-time energy, speech rate, and other features are concatenated to generate a 128-dimensional speech feature vector, which serves as a numerical representation of the speech emotion features.
[0050] For text data, the HowNet sentiment dictionary is invoked, and each word in the text is matched against the dictionary. Sentiment word weights are assigned based on the matching results: positive sentiment words are assigned positive weights (e.g., "anger" is assigned +5), negative sentiment words are assigned negative weights (e.g., "dissatisfaction" is assigned -4), and neutral words or words not in the dictionary have a weight of 0. The weights of all sentiment words in the text are summed to obtain the overall sentiment score. Special sentence structures are identified in the text: rhetorical questions are weighted by +2, exclamatory sentences by +3, and if the text contains multiple special sentence structures, the corresponding weights are summed and added to the sentiment word weight score to obtain the corrected sentiment intensity. The Word2Vec model is used to convert the preprocessed text into a vector representation. Word2Vec uses a neural network to map words to a high-dimensional vector space, where semantically similar words are close in distance. A 300-dimensional text feature vector is generated as a numerical representation of the text's sentiment features.
[0051] For facial image data, the facial landmark detection algorithm from the Dlib library was used to extract the coordinates of 68 facial landmarks. Based on the coordinates of 6 landmarks in the eye region among the 68 landmarks, the distance between the upper and lower eyelids (the Euclidean distance between the upper and lower eyelid landmarks) was calculated, and the iris diameter was calculated (or the distance between the left and right corners of the eyes was used as a normalization benchmark). The eye opening degree was calculated as the distance between the upper and lower eyelids divided by the iris diameter; the smaller the opening degree value, the higher the degree of eye closure. The positions of the left and right corners of the mouth were identified, and the inclination angle of the line connecting the corners of the mouth was calculated to obtain the corner curvature. A threshold of ±15% was used to distinguish emotions; a corner curvature greater than +15% was labeled as an upturned mouth, a corner curvature less than -15% was labeled as a downturned mouth, and a corner curvature between ±15% was labeled as neutral. The landmark coordinates, eye opening degree, and corner curvature features were concatenated to generate a 256-dimensional image feature vector, which serves as a numerical representation of the image's emotional features.
[0052] This application clearly defines multimodal data as including three categories: speech, text, and facial images. Specific emotional features of each type of data are extracted to form multimodal emotional features, making emotional feature extraction more targeted and comprehensive. This avoids the limitations of single-modal features, provides reliable data support for subsequent accurate emotion recognition, and improves the accuracy of emotion recognition.
[0053] One possible implementation of this application embodiment, the structure of the multimodal emotion fusion recognition model includes: The input layer is used to receive multimodal sentiment features; The attention layer is used to assign weights to each modality feature in the multimodal emotion features, resulting in weighted modality features. The fusion layer is used to concatenate and fuse the weighted features of each modality to generate a fused feature vector; Fully connected layers are used to perform non-linear transformations on the fused feature vectors to obtain dimensionally compressed fused feature vectors. The output layer is used to output the sentiment category classification result and the sentiment intensity quantification value based on the fused feature vector after dimensionality compression.
[0054] In this embodiment, the training dataset for the multimodal emotion fusion recognition model includes IEMOCAP (speech-text dataset), FER-2013 (image emotion dataset), and 100,000 self-collected customer service interaction data. The Adam optimizer is used, and the model is trained for 100 epochs, achieving a validation accuracy of 92%. After training, the model is configured in electronic devices for emotion category and intensity recognition.
[0055] Specifically, the input layer receives three types of feature vectors transmitted by the emotion feature extraction module: 128-dimensional voice emotion features, 300-dimensional text emotion features, and 256-dimensional image emotion features. The three types of feature vectors are concatenated in terms of feature dimensions to form a 684-dimensional multimodal feature vector, which serves as the input to the attention layer.
[0056] The core function of the attention layer is to dynamically determine which modality is more reliable in the current scene and assign corresponding weights. Weight allocation for typical scenarios can be pre-configured, such as a speech modality weight of 0.35, a text modality weight of 0.45, and an image modality weight of 0.2. When ambient lighting conditions are detected to be below a preset threshold (e.g., the user is in a dark environment), the image modality weight decreases from 0.20 to 0.15, while the weights of the speech and / or text modalities increase accordingly (the sum of the weights remains 1). The magnitude of this increase can be flexibly set. The original feature vectors of each modality are multiplied by their corresponding attention weights to obtain the weighted feature vectors for each modality.
[0057] The fusion layer maps 684 dimensions to 512 dimensions through a linear transformation, outputting a 512-dimensional fusion feature vector that integrates emotional information from three modalities.
[0058] The fully connected layer maps 512 dimensions to 256 dimensions using the ReLU activation function, outputting a fused feature vector with compressed dimensions.
[0059] The output layer contains a classification branch and a regression branch. The classification branch uses the Softmax function to output the sentiment category, and the selectable sentiment categories include anger, anxiety, confusion, neutrality, satisfaction, and disappointment. The regression branch uses linear intensity quantization to quantize the sentiment intensity to a range of 0-10 points.
[0060] The multimodal emotion fusion recognition model constructed in this application embodiment achieves reasonable weighting and effective fusion of multimodal emotion features through the synergistic effect of the input layer, attention layer, fusion layer and other structures. It can accurately output the user's emotion category and emotion intensity, solve the problem of low accuracy of existing single-modal recognition, and provide accurate emotional basis for subsequent interaction strategy matching.
[0061] In one possible implementation of this application, when the lighting conditions are lower than a preset lighting threshold, the attention layer reduces the modal feature weights of the facial image data according to a preset difference.
[0062] Optional, the default difference is 0.05.
[0063] This application embodiment addresses scenarios with poor lighting conditions by setting an attention layer to dynamically adjust the modal weights of facial image data, avoiding misjudgment of image emotional features caused by lighting effects, further optimizing the recognition accuracy of the multimodal emotion fusion recognition model, and improving the model's adaptability and reliability in complex environments.
[0064] One possible implementation of this application embodiment involves matching interaction strategies from a strategy rule base based on emotion category and emotion intensity, including: Based on a preset priority ranking, the target emotion category with the highest priority is determined from the emotion categories; Based on the target sentiment category and the sentiment intensity of the target sentiment category, an interaction strategy is matched from the strategy rule base; The strategy rule base stores the correspondence between combinations of different emotion categories and different emotion intensities and interaction strategies.
[0065] In this embodiment, the multimodal emotion fusion recognition model can output one or more emotion categories. When there is only one emotion category, it is directly used as the target emotion category. When there are multiple emotion categories, the one with the highest priority in the preset priority ranking is used as the target emotion category. The preset priority ranking is: anger > anxiety > confusion > neutral > satisfaction > disappointment. Emotion intensity ranges are pre-defined, such as 0-4 points, 5-7 points, and 8-10 points. An emotion category and an emotion intensity range are combined as a group, and a corresponding interaction strategy is pre-configured. The correspondence between each combination and interaction strategy is stored in the strategy rule base. Boundary intensity values (such as the 5-point anxiety / neutrality threshold) are assigned to the highest priority strategy.
[0066] Specific interaction strategies can be set based on customer service experience, and this embodiment does not impose any limitations. For example, if the emotion category is anger and the emotion intensity range is 8-10, the corresponding interaction strategy is: prioritize pushing reassuring messages (such as "We are very sorry for the inconvenience caused, we will prioritize handling your problem") + simplify the verification process (such as eliminating secondary identity verification) + automatically trigger manual transfer after 10 seconds of no response; if the emotion category is anxiety and the emotion intensity range is 5-7, the corresponding interaction strategy is: step-by-step guidance (such as "To quickly resolve your problem, you only need two steps: 1. Provide your order number; 2. Explain the problem") + problem focus (block irrelevant service recommendations); if the emotion category is neutral and the emotion intensity range is 0-4, the corresponding interaction strategy is: standard and efficient response (such as directly answering the question without unnecessary guidance); if the emotion category is satisfaction and the emotion intensity range is 8-10, the corresponding interaction strategy is: thank-you messages (such as "Thank you for your recognition") + relevant service recommendations (such as "Would you like to learn about membership benefits?").
[0067] This application's embodiments determine the highest priority target emotion category and match the interaction strategy from the strategy rule base based on its emotion intensity. This clarifies the logic and basis for strategy matching, achieves a precise correspondence between the interaction strategy and the user's emotional state, solves the problem that existing customer service interaction strategies are singular and cannot adapt to different emotional needs, and improves the pertinence and efficiency of the interaction.
[0068] One possible implementation of this application embodiment involves executing an interaction strategy, receiving user feedback, and iteratively optimizing the emotion fusion recognition model and strategy rule base using the feedback information, including: Synthesize voice responses that match the tone of the interaction strategy and emotion, and render the text responses to the client in real time; After the interaction ends, a feedback pop-up window will appear on the interactive interface to collect user feedback on the accuracy of emotion recognition and the satisfaction with problem resolution. The feedback information includes ratings and text feedback. The feedback information is labeled as correct or incorrect, added to the training set, and the multimodal emotion fusion recognition model is retrained. The policy rule base is adjusted simultaneously.
[0069] In this embodiment, the interaction strategy is received and the content of the response to be received is extracted from it. The intonation parameters of the speech synthesis are determined according to the target emotion category. The intonation parameters corresponding to each emotion category can be predetermined and stored in the electronic device. For example, anger and anxiety use a gentle and calm soothing intonation (moderate speed and soft volume), neutrality uses a standard and clear normal intonation, and satisfaction uses a light and positive pleasant intonation.
[0070] Utilizing Baidu's Text-To-Speech (TTS) technology, the system inputs text-based dialogue into the TTS engine, configures synthesis parameters, and pushes the synthesized voice response to the client for playback via the user's speaker or earpiece. The system extracts the required text-based dialogue from the interaction strategy, ensuring consistency between the text and the synthesized dialogue, and transmits the text response to the client in real-time. Text and voice responses can be presented synchronously or sequentially.
[0071] Feedback collection is triggered after the interaction ends. The termination criteria include: the user's problem has been clearly resolved (the user expresses satisfaction or does not ask further questions), the user actively exits / closes the chat window, the system transfers the user to a human agent, and the user remains unresponsive for an extended period (exceeding the preset silence time). After confirming the interaction is complete, a feedback pop-up appears on the client's interface. The user clicks the 1-5 star icon to rate the user and enters their feedback in the text input box (optional, users can leave it blank).
[0072] Regularly (e.g., weekly or monthly), new feedback data is extracted from the database. Each data point includes a user rating and the corresponding sentiment recognition result, and is labeled. Optionally, a 1-star user rating is labeled "Incorrect," a 4-5 star user rating is labeled "Correct," and a 2-3 star user rating can be configured with default rules such as ignoring or requiring manual review. For data labeled "Correct," the original input features of that session are used as positive samples, and the sentiment category output by the model is used as the label. For data labeled "Incorrect," the original input features of that session are used as negative samples. Correct labels can be manually corrected based on textual feedback, or the data can be directly used as hard examples for model optimization. The labeled feedback data is then added to the original training dataset for retraining.
[0073] We analyze the interaction strategies used in low-scoring sessions and manually adjust and optimize the strategy rule base.
[0074] This application's embodiments output voice and text responses that match emotions, collect user feedback through a feedback pop-up, and use it for iterative optimization of the model and strategy library. This not only improves the humanized experience of user interaction but also continuously corrects emotion recognition biases, optimizes interaction strategies, achieves dynamic improvement in system performance, and reduces manual operation costs.
[0075] In addition, this application supports the addition of new modalities (such as gestures and physiological signals) and emotion categories, which can quickly adapt to customer service needs in multiple scenarios such as e-commerce after-sales service, financial consultation, and government services, without the need to reconstruct the system framework.
[0076] This application provides an electronic device, such as... Figure 3 As shown, Figure 3The illustrated electronic device 300 includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in practical applications, the transceiver 304 is not limited to one type, and the structure of this electronic device 300 does not constitute a limitation on the embodiments of this application.
[0077] Processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 301 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.
[0078] Bus 302 may include a pathway for transmitting information between the aforementioned components. Bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 302 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The symbol is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0079] The memory 303 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.
[0080] The memory 303 is used to store the application code that executes the solution of this application, and its execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the content shown in the aforementioned embodiment of the intelligent customer service interaction method based on multimodal emotion recognition.
[0081] Figure 3 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0082] This application provides a computer-readable storage medium storing a computer program that, when run on a computer, enables the computer to execute the content shown in the aforementioned embodiment of the intelligent customer service interaction method based on multimodal emotion recognition.
[0083] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.
[0084] The above are only some embodiments of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. An intelligent customer service interaction method based on multimodal emotion recognition, characterized in that, include: Real-time acquisition of multimodal data during user interaction; Extract multimodal sentiment features from the multimodal data; The multimodal emotion features are input into the multimodal emotion fusion and recognition model to obtain the user's emotion category and emotion intensity; Based on the emotion category and emotion intensity, an interaction strategy is matched from the strategy rule base; The interaction strategy is executed, and feedback information from the user is received. The feedback information is then used to iteratively optimize the emotion fusion recognition model and the strategy rule base.
2. The intelligent customer service interaction method based on multimodal emotion recognition according to claim 1, characterized in that, The multimodal data includes speech data, text data, and facial image data; the extraction of multimodal emotional features from the multimodal data includes: Extract speech emotion features from the speech data, the speech emotion features including fundamental frequency, short-time energy and speech rate; Extract text sentiment features from the text data, the text sentiment features including sentiment word weights and sentence strength; Image emotion features are extracted from the facial image data, including eye opening and mouth corner curvature. The speech emotion features, the text emotion features, and the image emotion features constitute the multimodal emotion features.
3. The intelligent customer service interaction method based on multimodal emotion recognition according to claim 1, characterized in that, The structure of the multimodal emotion fusion recognition model includes: An input layer is used to receive the multimodal emotional features; An attention layer is used to assign weights to each modal feature in the multimodal emotion features to obtain weighted modal features; The fusion layer is used to concatenate and fuse the weighted features of each modality to generate a fused feature vector; A fully connected layer is used to perform a nonlinear transformation on the fused feature vector to obtain a dimensionally compressed fused feature vector; The output layer is used to output the sentiment category classification result and the sentiment intensity quantification value based on the fused feature vector after dimensionality compression.
4. The intelligent customer service interaction method based on multimodal emotion recognition according to claim 3, characterized in that, When the lighting conditions are lower than a preset lighting threshold, the attention layer reduces the modal feature weights of the facial image data according to a preset difference.
5. The intelligent customer service interaction method based on multimodal emotion recognition according to claim 1, characterized in that, The step of matching interaction strategies from the strategy rule base based on the emotion category and emotion intensity includes: Based on a preset priority ranking, the target emotion category with the highest priority is determined from the emotion categories; Based on the target emotion category and the emotion intensity of the target emotion category, an interaction strategy is matched from the strategy rule base; The strategy rule base stores the correspondence between combinations of different emotion categories and different emotion intensities and interaction strategies.
6. The intelligent customer service interaction method based on multimodal emotion recognition according to claim 1, characterized in that, The step of executing the interaction strategy and receiving feedback from the user, and using the feedback to iteratively optimize the emotion fusion recognition model and the strategy rule base, includes: A voice response with a tone matching the emotion of the interaction strategy is synthesized, and the text response is rendered to the client in real time; After the interaction ends, a feedback pop-up window will appear on the interactive interface to collect the user's feedback information on the accuracy of emotion recognition and the satisfaction with problem resolution. The feedback information includes ratings and text feedback. The feedback information is labeled as correct or incorrect, added to the training set, and the multimodal emotion fusion recognition model is retrained. The policy rule base is adjusted simultaneously.
7. The intelligent customer service interaction method based on multimodal emotion recognition according to claim 1, characterized in that, The real-time acquisition of multimodal data during user interaction includes: Voice data is collected using streaming transmission. Text data is collected based on a no-input trigger duration mechanism, and the text data is preprocessed. Image data is acquired, face detection is performed on the image data, and keyframes containing faces are extracted as facial image data.
8. An intelligent customer service interaction system based on multimodal emotion recognition, characterized in that, include: The multimodal data acquisition module is used to collect multimodal data during user interaction in real time. The sentiment feature extraction module is used to extract multimodal sentiment features from the multimodal data; The emotion fusion and recognition module is used to input the multimodal emotion features into the multimodal emotion fusion and recognition model to obtain the user's emotion category and emotion intensity; An interaction decision module is used to match interaction strategies from a strategy rule base based on the emotion category and emotion intensity. The interaction execution and feedback module is used to execute the interaction strategy, receive feedback information from the user, and iteratively optimize the emotion fusion recognition model and the strategy rule base using the feedback information.
9. The intelligent customer service interaction system based on multimodal emotion recognition according to claim 8, characterized in that, The emotion categories include anger, anxiety, confusion, neutrality, satisfaction, and disappointment.
10. The intelligent customer service interaction system based on multimodal emotion recognition according to claim 8, characterized in that, The interaction strategies in the strategy rule base include push reassurance messages, simplified verification process, manual transfer, step-by-step guidance, problem focus, standard and efficient response, thank-you messages, and related service recommendations.