Intelligent customer service interaction method and system

Abstract

The invention discloses an intelligent customer service interaction method and system, and relates to the technical field of intelligent customer service interaction, and the method comprises the steps: obtaining a user request; calculating a comprehensive emotion score through multi-modal emotion analysis based on a user request; according to the comprehensive emotion score, classifying the user emotion state, according to a classification result, initializing a dialogue context, and constructing a dialogue context graph; dynamically adjusting the response mode of the intelligent customer service according to the emotional state and the user question in the dialogue context graph, and collecting the feedback information of the user; and dynamically adjusting the dialogue strategy based on the feedback information of the user and the dialogue context. According to the method, the emotion state of the user can be comprehensively captured from multiple aspects such as texts, voices and facial expressions through multi-modal emotion analysis, so that the accuracy and robustness of emotion recognition are improved, a reliable basis is provided for subsequent dialogue management and response modes, and the experience satisfaction degree and the service quality of the user are improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent customer service interaction technology, and in particular to an intelligent customer service interaction method and system. Background Art

[0002] In recent years, with the rapid development of artificial intelligence technology, the application of intelligent customer service has become increasingly widespread in all walks of life. Intelligent customer service has achieved efficient interaction with users through natural language processing (NLP), speech recognition, computer vision and other technologies, greatly improving the quality and efficiency of customer service. Traditional intelligent customer service mainly relies on text input and answers questions through predefined rules and simple machine learning models. However, with the advancement of technology, multimodal sentiment analysis has gradually become a research hotspot. It can analyze multiple modal information such as user text, voice and facial expressions to more comprehensively understand the user's emotional state, thereby providing more personalized and accurate services. Multimodal sentiment analysis technology can not only improve user satisfaction, but also effectively reduce the workload of customer service personnel and enhance the competitiveness of enterprises.

[0003] Although the application of multimodal sentiment analysis technology in the field of intelligent customer service has made significant progress, there are still some shortcomings. First, the existing intelligent customer service mostly relies on single-modal data in emotion recognition, while ignoring the complementary role of other modal information. This single-modal emotion recognition method often cannot fully and accurately capture the user's true emotional state. Secondly, the existing intelligent customer service lacks a dynamic adjustment mechanism in dialogue management. Once the initial setting is completed, it is difficult to flexibly adjust according to the user's real-time feedback. Therefore, how to achieve more accurate sentiment analysis through multimodal sentiment analysis technology, and build a dynamically adjusted dialogue management mechanism on this basis, has become a key issue that needs to be urgently solved in current intelligent customer service. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides an intelligent customer service interaction method to solve the problems of inaccurate single-modality emotion recognition and lack of flexibility in dialogue management in the prior art.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides an intelligent customer service interaction method, which includes obtaining user requests; calculating a comprehensive sentiment score based on the user requests through multimodal sentiment analysis; classifying the user's emotional state according to the comprehensive sentiment score, initializing the dialogue context according to the classification result, and constructing a dialogue context map; dynamically adjusting the response method of the intelligent customer service according to the emotional state and user questions in the dialogue context map, and collecting user feedback information; dynamically adjusting the dialogue strategy based on the user's feedback information and the dialogue context.

[0008] Preferably, the user request includes text information input through a chat window, a voice message sent through a voice assistant, and a facial expression video sent through a camera.

[0009] Preferably, the comprehensive sentiment score is calculated based on the user request through multimodal sentiment analysis, and the specific steps are as follows:

[0010] Based on the AFINN sentiment dictionary, the text information is split into phrases, and a sentiment value is assigned to each word according to the sentiment dictionary to calculate the text sentiment score. The expression is:

[0011]

[0012] Where n is the number of phrases in the text, z i is the sentiment value of the ith phrase, E t is the text sentiment score, i is the index variable of the phrase;

[0013] Computer vision algorithms are used to extract facial features. Based on historical data and big data analysis, the range of facial feature emotion values ​​is defined. According to the range of facial feature emotion values, the facial emotion score is calculated. The expression is:

[0014]

[0015] Where p is the number of facial features, u j is the weight of the jth facial feature, E f is the facial emotion score, j is the index variable of facial features, x j is the jth facial feature value, x n is the median of all facial feature values, x max is the maximum value among all facial features, x min is the minimum value among all facial features;

[0016] The facial features include the degree of eyebrow uplift, the degree of eye opening and the angle of mouth corner uplift;

[0017] Through signal analysis, acoustic features are extracted, and the emotional range of acoustic features is defined based on historical data. According to the emotional range of acoustic features, the speech emotion score is calculated. The expression is:

[0018]

[0019] Where m is the number of acoustic features, w k is the weight of the kth acoustic feature, E v is the speech emotion score, k is the index variable of the acoustic feature, r k is the kth acoustic eigenvalue, r n is the median of all acoustic eigenvalues, r max is the maximum value among all acoustic features, r min is the minimum value among all acoustic features;

[0020] The acoustic features include pitch, volume, speaking rate and fundamental frequency;

[0021] The text sentiment score E t , facial emotion score E f and speech emotion score E v Perform nonlinear fusion and calculate the nonlinear fusion value. The expression is:

[0022]

[0023] Among them, S is the nonlinear fusion value of text emotion score, facial emotion score and voice emotion score, w1 is the weight coefficient of text emotion feature, w2 is the weight coefficient of voice emotion feature, w3 is the weight coefficient of facial emotion feature, and α is the adjustment parameter of the influence of nonlinear term on the comprehensive emotion score;

[0024] Based on the nonlinear fusion value S, the Sigmoid function is used to calculate the comprehensive sentiment score, which is expressed as:

[0025]

[0026] Among them, E is the comprehensive sentiment score, c is the median of the Sigmoid function, e is the base of the natural logarithm, and -b is the sensitivity adjustment coefficient of the sentiment score to the nonlinear fusion value S.

[0027] Preferably, the user's emotional state is classified according to the comprehensive emotional score, and the specific steps are as follows:

[0028] Based on historical user data and sentiment analysis reports, define positive threshold T1 and negative threshold T2;

[0029] According to the comprehensive emotion score E, the user's emotional state is classified;

[0030] When E ≥ T1, the current user's emotional state is considered positive;

[0031] When T2<E<T1, the former user’s emotional state is considered neutral;

[0032] When E≤T2, the current user's emotional state is considered to be negative.

[0033] Preferably, the initial dialogue context includes user ID, user initial request content, user initial emotional state, initial dialogue turn and historical dialogue record;

[0034] The specific steps of constructing the conversation context graph are as follows:

[0035] Based on the initial emotional state of the initialized user, create an emotional state node;

[0036] Based on the initialized user initial request content, create an initial request content node;

[0037] A directed edge is established between the emotional state node and the request content node to connect the emotional state with the request content using a causal relationship;

[0038] A directed graph is created based on the graph database, and the emotional state nodes, request content nodes, and the causal relationship between the emotional nodes and the request content nodes are added to the graph to build a conversation context graph.

[0039] Preferably, the method of dynamically adjusting the response mode of the intelligent customer service and collecting user feedback information according to the emotional state and user questions in the conversation context map is as follows:

[0040] Read the current emotional state node and the current request content node from the conversation context graph to obtain the user's emotional state and initial request content;

[0041] Use NLP to analyze the user's request content, extract the user's questions and analyze the question type;

[0042] Generate preliminary response content based on the user's question and the user's emotional state;

[0043] Optimize the initial intelligent customer service response content based on historical conversation records and user feedback in the conversation context graph;

[0044] Dynamically select the best intelligent customer service response method based on the user's emotional state and the type of user's question;

[0045] The types of questions include consultation, help-seeking, complaint and emotional support.

[0046] Preferably, the dialogue strategy is dynamically adjusted based on the user's feedback signal and the dialogue context, and the specific steps are as follows:

[0047] Through the optimal intelligent customer service response method, the optimized intelligent customer service response content is sent to the user, and the user's feedback information is received;

[0048] Record the user's feedback information into the conversation context, and update the user's emotional state node and historical conversation records;

[0049] Dynamically adjust the dialogue strategy based on user feedback information, updated emotional state nodes, and historical dialogue records.

[0050] In a second aspect, the present invention provides an intelligent customer service interaction system, including a user request acquisition module, an emotion scoring module, an emotion state classification module, a response mode adjustment module and a dialogue strategy adjustment module; the user request acquisition module is used to obtain user requests; the emotion scoring module is used to calculate a comprehensive emotion score based on user requests through multimodal emotion analysis; the emotion state classification module is used to classify the user's emotion state according to the comprehensive emotion score, initialize the dialogue context according to the classification result, and construct a dialogue context map; the response mode adjustment module is used to dynamically adjust the response mode of the intelligent customer service according to the emotion state and user questions in the dialogue context map, and collect user feedback information; the dialogue strategy adjustment module is used to dynamically adjust the dialogue strategy based on the user's feedback information and dialogue context.

[0051] In a third aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the computer program is executed by the processor, any step of the intelligent customer service interaction method described in the first aspect of the present invention is implemented.

[0052] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, any step of the intelligent customer service interaction method as described in the first aspect of the present invention is implemented.

[0053] The beneficial effects of the present invention are as follows: through multimodal sentiment analysis, the user's emotional state can be fully captured from multiple angles such as text, voice and facial expressions, thereby improving the accuracy and robustness of emotion recognition, providing a reliable basis for subsequent dialogue management and response methods, and improving user experience satisfaction and service quality; by dynamically adjusting the dialogue strategy steps, the dialogue strategy can be optimized in real time according to the user's feedback and dialogue context during each round of dialogue, ensuring the coherence and consistency of the dialogue, enhancing adaptability and flexibility, improving user satisfaction, reducing customer service costs, and enhancing the competitiveness of the enterprise. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0055] Figure 1 This is a flow chart of the intelligent customer service interaction method in Example 1.

[0056] Figure 2 Schematic diagram of the intelligent customer service interaction system in Example 1. DETAILED DESCRIPTION

[0057] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings.

[0058] Example 1, reference Figure 1 and Figure 2 , which is the first embodiment of the present invention, and provides an intelligent customer service interaction method, comprising the following steps:

[0059] S1: Get user request.

[0060] S1.1: User requests include text messages entered through the chat window, voice messages sent through the voice assistant, and facial expression videos sent through the camera.

[0061] It should be noted that the purpose of collecting the above data is to achieve more comprehensive and accurate emotion recognition through multimodal sentiment analysis technology, comprehensively processing text information, voice messages and facial expression videos, thereby providing users with a more personalized and accurate intelligent customer service interaction experience.

[0062] S2: Based on user requests, the comprehensive sentiment score is calculated through multimodal sentiment analysis.

[0063] S2.1: Based on the AFINN sentiment dictionary, the text information is split into phrases, and a sentiment value is assigned to each word according to the sentiment dictionary. The text sentiment score is calculated as follows:

[0064]

[0065] Where n is the number of phrases in the text, z i is the sentiment value of the ith phrase, E t is the text sentiment score, i is the index variable of the phrase;

[0066] Furthermore, first, natural language processing technology is used to split the text information into multiple phrases; then, a corresponding sentiment value is assigned to each word according to the AFINN sentiment dictionary, where positive sentiment values ​​are positive and negative sentiment values ​​are negative; finally, the average of the sentiment values ​​of all phrases is calculated to obtain the final text sentiment score.

[0067] It should be noted that the reason for AFINN is that it provides a widely recognized sentiment lexicon that can assign sentiment values ​​to words, which is suitable for text sentiment analysis and helps to accurately assess the user's emotional state.

[0068] S2.2: Use computer vision algorithms to extract facial features, define the range of facial feature emotion values ​​based on historical data and big data analysis, and calculate the facial emotion score based on the range of facial feature emotion values. The expression is:

[0069]

[0070] Where p is the number of facial features, u j is the weight of the jth facial feature, E f is the facial emotion score, j is the index variable of facial features, x j is the jth facial feature value, x n is the median of all facial feature values, x max is the maximum value among all facial features, x min is the minimum value among all facial features;

[0071] Facial features include the degree of eyebrow rise, eye opening, and angle of mouth corners;

[0072] Furthermore, facial features are extracted using computer vision algorithms, specifically using facial detection and feature extraction techniques (such as algorithms in OpenCV or Dlib libraries) to identify and extract key points of the face (such as eyes, mouth, eyebrows, etc.), and using machine learning models (such as convolutional neural networks (CNNs)) to analyze the changing patterns of these features, define the range of emotional values ​​of facial features based on historical data and big data analysis, and then calculate the facial emotion score.

[0073] Define the range of emotional values ​​for facial features as follows:

[0074] The emotional range of facial features distinguishes different emotional states of users by setting thresholds. By analyzing multiple facial features, such as the degree of eyebrow rise, eye opening, and mouth corner rise, the emotional threshold of each feature is defined in combination with historical data. By comparing the current values ​​of these features with the preset thresholds, the user's emotional state can be classified as positive, neutral, or negative. This threshold-based sentiment analysis helps the customer service system better understand user emotions and adjust the dialogue strategy.

[0075] For example, the upward threshold value X1 and the downward threshold value X2 of the mouth corner position are defined based on historical data.

[0076] Positive emotion: The upward angle of the mouth corners is greater than or equal to X1, which is manifested as a smile.

[0077] Neutral emotion: The position of the mouth corners is between X2 and X1, and the mouth corners remain straight or slightly upturned.

[0078] Negative emotion: The drooping angle of the mouth corners is less than or equal to X2, showing a frown or bored expression.

[0079] S2.3: Extract acoustic features through signal analysis, define the emotional range of acoustic features based on historical data, and calculate the speech emotion score based on the emotional range of acoustic features. The expression is:

[0080]

[0081] Where m is the number of acoustic features, w k is the weight of the kth acoustic feature, E v is the speech emotion score, k is the index variable of the acoustic feature, r k is the kth acoustic eigenvalue, r n is the median of all acoustic eigenvalues, r max is the maximum value among all acoustic features, r min is the minimum value among all acoustic features;

[0082] Acoustic features include pitch, volume, speech rate, and fundamental frequency;

[0083] Specifically: first, use audio processing tools (such as Librosa) to preprocess the voice signal and extract acoustic features such as pitch, volume, speaking rate and fundamental frequency; then, based on historical data and big data analysis, define the emotional value range of acoustic features; finally, based on the defined emotional value range, calculate the voice emotion score to evaluate the user's emotional state.

[0084] Define the emotional value range of acoustic features as follows:

[0085] The emotional value range of acoustic features distinguishes different emotional states of users by setting thresholds. By analyzing multiple acoustic features, such as pitch, volume, speech rate, and fundamental frequency, and combining historical data, the emotional threshold of each feature is defined. By comparing the current values ​​of these features with the preset thresholds, the user's emotional state can be classified as positive, neutral, or negative. This threshold-based emotional analysis helps intelligent customer service better understand the user's voice emotions and adjust the dialogue strategy.

[0086] Example: Define the positive emotion threshold Y1 and negative emotion threshold Y2 for speech rate based on historical data.

[0087] Positive emotion: speaking speed is greater than or equal to Y1, which is characterized by fast and fluent speaking speed.

[0088] Neutral emotion: The speaking speed is between Y2 and Y1, which is normal and moderate.

[0089] Negative emotion: speaking speed is less than or equal to Y2, manifested as slow, hesitant or delayed speaking speed.

[0090] S2.4: Score the text sentiment E t , facial emotion score E f and speech emotion score E v Perform nonlinear fusion and calculate the nonlinear fusion value. The expression is:

[0091]

[0092] Among them, S is the nonlinear fusion value of text emotion score, facial emotion score and voice emotion score, w1 is the weight coefficient of text emotion feature, w2 is the weight coefficient of voice emotion feature, w3 is the weight coefficient of facial emotion feature, and α is the adjustment parameter of the influence of nonlinear term on the comprehensive emotion score;

[0093] It should be noted that by nonlinearly fusing text emotion scores, facial emotion scores, and voice emotion scores and comprehensively considering emotional information of multiple modalities, the accuracy and comprehensiveness of emotion recognition are improved, thereby more accurately evaluating the user's emotional state and providing a more personalized and effective interactive experience for intelligent customer service.

[0094] S2.5: Based on the nonlinear fusion value S, the Sigmoid function is used to calculate the comprehensive sentiment score, which is expressed as:

[0095]

[0096] Among them, E is the comprehensive sentiment score, S is the nonlinear fusion value, c is the median of the Sigmoid function, e is the base of the natural logarithm, and -b is the sensitivity adjustment coefficient of the sentiment score to the nonlinear fusion value S.

[0097] It should be noted that mapping the nonlinear fusion value S to the range of 0 to 1 makes the comprehensive sentiment score E more standardized and easier to interpret. Through the nonlinear transformation of the Sigmoid function, extreme values ​​can be smoothly processed to avoid large fluctuations in the comprehensive score due to outliers in individual modes, thereby improving the stability and reliability of the score. In addition, the standardized sentiment score E can more easily classify emotional states, such as positive, neutral, and negative, thereby providing more accurate emotion recognition and response strategies for intelligent customer service.

[0098] S3: Classify the user's emotional state according to the comprehensive emotional score, initialize the conversation context based on the classification results, and build a conversation context graph.

[0099] S3.1: Based on historical user data and sentiment analysis reports, define positive threshold T1 and negative threshold T2;

[0100] Furthermore, first, a large amount of historical user data is collected and sorted, including the user's text, voice and facial expression data and their corresponding emotional tags; then, these data are processed through the sentiment analysis algorithm to generate the sentiment score distribution; then, according to the distribution of the sentiment score and the statistical results in the sentiment analysis report, the dividing point between positive and negative emotions is determined, and set as the positive threshold T1 and the negative threshold T2 respectively. This process is based on the sentiment score distribution in the historical data and the statistical results in the sentiment analysis report, ensuring that the threshold setting can accurately reflect the user's emotional state, thereby providing a reliable reference standard for the classification of emotional states.

[0101] S3.2: Classify the user's emotional state according to the comprehensive emotional score E;

[0102] When E ≥ T1, the current user's emotional state is considered positive;

[0103] This means that the user's comprehensive emotion score exceeds the positive threshold T1, indicating that the user is in a positive, satisfied, or happy state.

[0104] When T2<E<T1, the former user’s emotional state is considered neutral;

[0105] This means that the user's comprehensive emotion score is between the positive threshold T1 and the negative threshold T2, indicating that the user's emotional state is relatively stable, neither particularly positive nor particularly negative.

[0106] When E≤T2, the current user's emotional state is considered to be negative.

[0107] This means that the user's comprehensive sentiment score is lower than the negative threshold T2, indicating that the user is in a negative, dissatisfied or unhappy state.

[0108] S3.3: Initial conversation context includes user ID, user's initial request content, user's initial emotional state, initial conversation turn and historical conversation records;

[0109] Specifically, first, record the user's unique identifier (user ID) to ensure that each user's identity is traceable; then, obtain and record the user's initial request content, including text information, voice messages, and facial expression videos; then, determine the user's initial emotional state based on the comprehensive emotional score E and classify it as positive, neutral, or negative; then, set the initial conversation round to 1, indicating that this is the starting point of the conversation; finally, initialize the historical conversation record to empty, and prepare to record subsequent conversation content and user feedback. This process ensures the integrity and coherence of the conversation context, provides the necessary initial information for intelligent customer service, and supports subsequent multi-round conversation management and emotional state tracking.

[0110] S3.4: Construct a conversation context graph. The specific steps are as follows:

[0111] S3.4.1: Create an emotional state node based on the initial emotional state of the initialized user;

[0112] The user's current emotional state is recorded and represented in the conversation context graph, providing a basis for subsequent emotion change tracking and conversation management.

[0113] S3.4.2: creating an initial request content node based on the initialized user initial request content;

[0114] The user's initial request information is recorded and represented in the conversation context graph, providing a basis for subsequent conversation understanding and response.

[0115] S3.4.3: Create a directed edge between the emotional state node and the request content node to link the emotional state with the request content using a causal relationship;

[0116] Specifically, first, find the emotional state node and the initial request content node from the conversation context graph; then, establish a directed edge between the two nodes, from the request content node to the emotional state node, indicating that the user's request content leads to the current emotional state. The establishment of this directed edge not only records the user's emotional state, but also clarifies the causal relationship between the emotional state and the request content, that is, the user's request content is the cause of their current emotional state. This process helps intelligent customer service better understand the user's emotional changes and provide strong support for subsequent conversation management and emotional response.

[0117] S3.4.4: Create a directed graph based on the graph database, add emotional state nodes, request content nodes, and the causal relationship between emotional nodes and request content nodes to the graph, and build a conversation context graph.

[0118] It should be noted that, first, an empty directed graph is created using a graph database (such as Neo4j); then, the emotional state node and the request content node are added to the graph as vertices; then, a directed edge is established between the two nodes, indicating that the request content leads to the current emotional state; finally, the relationship between these nodes and edges is stored in the graph database to form a complete conversation context graph, providing structured data support for subsequent multi-round conversation management and emotional state tracking.

[0119] S4: Dynamically adjust the response method of intelligent customer service based on the emotional state and user questions in the conversation context map, and collect user feedback information.

[0120] S4.1: Read the current emotional state node and the current request content node from the conversation context graph to obtain the user's emotional state and initial request content;

[0121] S4.2: Use NLP to analyze the user's request content, extract the user's questions and analyze the question type;

[0122] First, the user's request content is preprocessed, including removing stop words, punctuation marks and numbers, and performing stemming or word form restoration to ensure the neatness and standardization of the text; then, natural language understanding (NLU) technology is used to extract the user's specific questions and intentions through named entity recognition (NER) and intent recognition (Intent Recognition); finally, based on the extracted questions and intentions, the user's questions are classified into consultation, help, complaint or emotional support categories, providing clear guidance for subsequent intelligent customer service responses.

[0123] S4.3: Generate preliminary response content based on the user's question and the user's emotional state;

[0124] Specifically, first, NLP technology is used to analyze the user's question type, such as identifying that the user's question is of the consulting type, "I want to know what the weather will be like tomorrow?"; then, combined with the user's emotional state, such as positive (happy), positive and friendly response content is generated, such as: "Hello! It's a pleasure to serve you. Tomorrow's weather forecast shows sunny weather and moderate temperatures, which is very suitable for outdoor activities. I hope you have a nice day!" This can not only accurately answer the user's question, but also meet the user's current emotional state, thereby improving the user experience.

[0125] S4.4: Optimize the initial intelligent customer service response content based on historical conversation records and user feedback in the conversation context graph;

[0126] Specifically, first, read the historical conversation records from the conversation context map to understand the user's previous interactions; then, analyze the user's feedback information to identify the user's specific needs and preferences; finally, combine the historical conversation records and user feedback to adjust and supplement the initially generated response content to make it more accurate and in line with user expectations. For example, if the initial response is "Hello! Tomorrow's weather forecast shows sunny, moderate temperature, very suitable for outdoor activities." But the user feedback hopes to get more detailed information, such as temperature range and wind conditions, the optimized response can be: "Hello! Tomorrow's weather forecast shows sunny, daytime maximum temperature is about 25℃, nighttime minimum temperature is about 15℃, wind speed is about level 3, very suitable for outdoor activities. I hope this information is helpful to you!" This can better meet the needs of users and improve user experience.

[0127] S4.5: Dynamically select the optimal intelligent customer service response method based on the user's emotional state and the type of user's question;

[0128] Dynamically select the best intelligent customer service response method, as follows:

[0129] The emotional state is positive:

[0130] Consultation: Choose text response to provide detailed information and answers.

[0131] Help-seeking: Choose text response and provide specific solutions and steps.

[0132] Complaints: Choose a text response to express your apology and provide a solution.

[0133] Emotional Support: Select voice responses to provide words of warmth and comfort.

[0134] Emotional state is neutral:

[0135] Consultation: Choose text response to provide concise information and answers.

[0136] Help-seeking: Choose text response and provide basic solutions and steps.

[0137] Complaints: Choose a text response to express your apology and provide a basic solution.

[0138] Emotional support: Choose a text response to provide brief comfort and encouragement.

[0139] Negative emotional state:

[0140] Consultation: Choose text response to provide detailed information and answers and express concern.

[0141] Help-seeking: Select voice response, provide specific solutions and steps, and express concern.

[0142] Complaints: Choose voice response, express sincere apologies and provide detailed solutions.

[0143] Emotional Support: Select voice responses that provide words of warmth and comfort and express concern.

[0144] The types of questions include consultation, help-seeking, complaint, and emotional support.

[0145] S5: Dynamically adjust the dialogue strategy based on user feedback and dialogue context.

[0146] S5.1: Send the optimized intelligent customer service response content to the user through the optimal intelligent customer service response method, and receive the user's feedback information;

[0147] Feedback information includes user satisfaction ratings, specific improvement suggestions, and emotional responses to the intelligent customer service response. For example, a user may reply: "Thank you for your help, but can you provide more detailed logistics information? I'm a little anxious right now." This indicates that the user is generally satisfied with the initial response, and has made improvement suggestions that require more detailed information, while expressing current anxiety.

[0148] S5.2: Record the user's feedback information into the conversation context, and update the user's emotional state node and historical conversation record;

[0149] Furthermore, first, satisfaction evaluation, improvement suggestions and emotional reactions are extracted from user feedback; then, these feedback information are recorded in the conversation context graph, and the emotional state node is updated to reflect the user's current emotional changes; finally, the feedback information and responses are added to the historical conversation record to ensure the integrity and coherence of the conversation context.

[0150] S5.3: Dynamically adjust the dialogue strategy based on user feedback information, updated emotional state nodes, and historical dialogue records.

[0151] Specifically: First, read the user feedback information, such as user feedback: "Thank you for your help, but can you provide more detailed logistics information? I am a little anxious now."; then, based on the user feedback, update the emotional state node from "neutral" to "negative" to reflect the user's anxious mood; then, record the user's feedback information and previous responses in the historical conversation record to ensure the integrity and coherence of the conversation context; finally, dynamically adjust the conversation strategy to generate new response content.

[0152] For example: "I understand your anxiety very well, and I will provide you with more detailed logistics information. Your order is shipped by "XX" Express, the logistics order number is "xxxxxxx", and it is expected to be delivered tomorrow afternoon. You can check the real-time logistics information through this link: [link]. If you have any other questions or need further help, please feel free to let me know and we will resolve it for you as soon as possible. I hope this information can help you wait with peace of mind." This ensures that the response not only meets the user's emotional needs, but also effectively solves the problem and improves user satisfaction.

[0153] The present embodiment also provides an intelligent customer service interaction system, including: a user request acquisition module, an emotion scoring module, an emotion state classification module, a response mode adjustment module and a dialogue strategy adjustment module; the user request acquisition module is used to obtain user requests; the emotion scoring module is used to calculate a comprehensive emotion score based on user requests through multimodal emotion analysis; the emotion state classification module is used to classify the user's emotion state according to the comprehensive emotion score, initialize the dialogue context according to the classification result, and construct a dialogue context map; the response mode adjustment module is used to dynamically adjust the response mode of the intelligent customer service according to the emotion state and user questions in the dialogue context map, and collect user feedback information; the dialogue strategy adjustment module is used to dynamically adjust the dialogue strategy based on the user's feedback information and the dialogue context.

[0154] This embodiment also provides a computer device suitable for the case of an intelligent customer service interaction method, including: a memory and a processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the intelligent customer service interaction method proposed in the above embodiment.

[0155] The computer device may be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a key, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0156] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the method for realizing intelligent customer service interaction as proposed in the above embodiment is implemented; the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (Static Random Access Memory, referred to as SRAM), electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, referred to as EEPROM), erasable programmable read-only memory (Erasable Programmable Read Only Memory, referred to as EPROM), programmable read-only memory (Programmable Red-Only Memory, referred to as PROM), read-only memory (Read-Only Memory, referred to as ROM), magnetic storage, flash memory, disk or optical disk.

[0157] In summary, the present invention can: through multimodal sentiment analysis, comprehensively capture the user's emotional state from multiple angles such as text, voice and facial expressions, improve the accuracy and robustness of emotion recognition, provide a reliable foundation for subsequent dialogue management and response methods, and improve user experience satisfaction and service quality; through dynamic adjustment of dialogue strategy steps, it can optimize the dialogue strategy in real time according to user feedback and dialogue context during each round of dialogue, ensure the coherence and consistency of the dialogue, enhance adaptability and flexibility, improve user satisfaction, reduce customer service costs, and enhance the competitiveness of the enterprise.

[0158] Example 2, referring to Table 1, is the second example of the present invention. To further verify the technical solution of the present invention, experimental simulation data of the intelligent customer service interaction method are provided.

[0159] In order to verify the application effect of the multimodal sentiment analysis technology of the present invention in intelligent customer service, the following experiment was conducted. The test subjects included 20 users, who input text messages through the chat window, send voice messages through the voice assistant, and send facial expression videos through the camera to simulate user requests in real scenarios.

[0160] First, in order to verify the application effect of multimodal sentiment analysis technology in intelligent customer service, an experiment was conducted. The test subjects included 20 users who entered text messages through the chat window, sent voice messages through the voice assistant, and sent facial expression videos through the camera to simulate user requests in real scenarios.

[0161] Secondly, based on user requests, a comprehensive sentiment score is calculated using multimodal sentiment analysis technology. The specific steps include: sentiment analysis of text information to calculate text sentiment scores; facial features are extracted to calculate facial sentiment scores; acoustic features are extracted to calculate speech sentiment scores. Then, these three sentiment scores are combined to calculate a comprehensive sentiment score.

[0162] Next, based on the comprehensive emotion score, the user's emotional state is classified and the conversation context is initialized. Specifically, the user's emotional state is classified as positive, neutral, or negative. Then, the user's unique identifier, initial request content, initial emotional state, initial conversation turn, and historical conversation records are recorded to build a conversation context graph to support subsequent multi-turn conversation management and emotional state tracking.

[0163] Finally, based on the emotional state and user questions in the conversation context map, the response method of the intelligent customer service is dynamically adjusted, and user feedback information is collected to update the conversation context map and dynamically adjust the conversation strategy to improve user satisfaction and interactive experience.

[0164] The details are shown in Table 1 below:

[0165] Table 1 Multimodal sentiment analysis experimental data table

[0166]

[0167]

[0168] Through the analysis of the data in the above table, it can be clearly seen that the application of the multimodal sentiment analysis technology of the present invention in intelligent customer service has significant advantages and innovations. For example, the comprehensive sentiment score of user 1 is 0.75, which is classified as positive, and the initial response satisfaction is 7.1, which eventually increased to 8.9; the comprehensive sentiment score of user 2 is 0.40, which is classified as negative, and the initial response satisfaction is 4.2, which eventually increased to 7.3. These data show that the present invention can more comprehensively and accurately evaluate the user's emotional state by combining the emotional information of the three modalities of text, face and voice, and significantly improve the user's satisfaction and interactive experience by dynamically adjusting the dialogue strategy.

[0169] The intelligent customer service interaction method of the present invention can not only more comprehensively and accurately evaluate the user's emotional state, but also dynamically adjust the response strategy according to the user's emotional changes, significantly improving the user's satisfaction and interactive experience. This reflects the innovation and practicality of the present invention in multimodal sentiment analysis technology, which can effectively improve the performance of intelligent customer service and provide more personalized and accurate services.

[0170] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. An intelligent customer service interaction method, characterized in that: include, Get user request; Based on user requests, calculate the comprehensive sentiment score through multimodal sentiment analysis; According to the comprehensive emotion score, the user's emotional state is classified, and based on the classification results, the conversation context is initialized and the conversation context graph is constructed; Dynamically adjust the intelligent customer service response method based on the emotional state and user questions in the conversation context map and collect user feedback information; Dynamically adjust the dialogue strategy based on user feedback and dialogue context.

2. The intelligent customer service interaction method according to claim 1, characterized in that: The user request includes text information entered through the chat window, voice messages sent through the voice assistant, and facial expression videos sent through the camera.

3. The intelligent customer service interaction method according to claim 2, characterized in that: Based on the user request, the comprehensive sentiment score is calculated through multimodal sentiment analysis. The specific steps are as follows: Based on the AFINN sentiment dictionary, the text information is split into phrases, and a sentiment value is assigned to each word according to the sentiment dictionary to calculate the text sentiment score. The expression is: Where n is the number of phrases in the text, z i is the sentiment value of the ith phrase, E t is the text sentiment score, i is the index variable of the phrase; Computer vision algorithms are used to extract facial features. Based on historical data and big data analysis, the range of facial feature emotion values ​​is defined. According to the range of facial feature emotion values, the facial emotion score is calculated. The expression is: Where p is the number of facial features, u j is the weight of the jth facial feature, E f is the facial emotion score, j is the index variable of facial features, x j is the jth facial feature value, x n is the median of all facial feature values, x max is the maximum value among all facial features, x min is the minimum value among all facial features; The facial features include the degree of eyebrow uplift, the degree of eye opening and the angle of mouth corner uplift; Through signal analysis, acoustic features are extracted, and the emotional range of acoustic features is defined based on historical data. According to the emotional range of acoustic features, the speech emotion score is calculated. The expression is: Where m is the number of acoustic features, w k is the weight of the kth acoustic feature, E v is the speech emotion score, k is the index variable of the acoustic feature, r k is the kth acoustic eigenvalue, r n is the median of all acoustic eigenvalues, r max is the maximum value among all acoustic features, r min is the minimum value among all acoustic features; The acoustic features include pitch, volume, speaking rate and fundamental frequency; The text sentiment score E t , facial emotion score E f and speech emotion score E v Perform nonlinear fusion and calculate the nonlinear fusion value. The expression is: Among them, S is the nonlinear fusion value of text emotion score, facial emotion score and voice emotion score, w1 is the weight coefficient of text emotion feature, w2 is the weight coefficient of voice emotion feature, w3 is the weight coefficient of facial emotion feature, and α is the adjustment parameter of the influence of nonlinear term on the comprehensive emotion score; Based on the nonlinear fusion value S, the Sigmoid function is used to calculate the comprehensive sentiment score, which is expressed as: Among them, E is the comprehensive sentiment score, c is the median of the Sigmoid function, e is the base of the natural logarithm, and -b is the sensitivity adjustment coefficient of the sentiment score to the nonlinear fusion value S.

4. The intelligent customer service interaction method according to claim 3, characterized in that: The user's emotional state is classified according to the comprehensive emotional score. The specific steps are as follows: Based on historical user data and sentiment analysis reports, define positive threshold T1 and negative threshold T2; According to the comprehensive emotion score E, the user's emotional state is classified; When E ≥ T1, the current user's emotional state is considered positive; When T2<E<T1, the former user’s emotional state is considered neutral; When E≤T2, the current user's emotional state is considered to be negative.

5. The intelligent customer service interaction method according to claim 1, characterized in that: The initial dialogue context includes the user ID, the user's initial request content, the user's initial emotional state, the initial dialogue turn and the historical dialogue record; The specific steps of constructing the conversation context graph are as follows: Based on the initial emotional state of the initialized user, create an emotional state node; Based on the initialized user initial request content, create an initial request content node; A directed edge is established between the emotional state node and the request content node to connect the emotional state with the request content using a causal relationship; A directed graph is created based on the graph database, and the emotional state nodes, request content nodes, and the causal relationship between the emotional nodes and the request content nodes are added to the graph to build a conversation context graph.

6. The intelligent customer service interaction method according to claim 1, characterized in that: According to the emotional state and user questions in the conversation context map, the response method of the intelligent customer service is dynamically adjusted, and user feedback information is collected. The specific steps are as follows: Read the current emotional state node and the current request content node from the conversation context graph to obtain the user's emotional state and initial request content; Use NLP to analyze the user's request content, extract the user's questions and analyze the question type; Generate preliminary response content based on the user's question and the user's emotional state; Optimize the initial intelligent customer service response content based on historical conversation records and user feedback in the conversation context graph; Dynamically select the best intelligent customer service response method based on the user's emotional state and the type of user's question; The types of questions include consultation, help-seeking, complaint and emotional support.

7. The intelligent customer service interaction method according to claim 1, characterized in that: The specific steps of dynamically adjusting the dialogue strategy based on the user's feedback signal and dialogue context are as follows: Through the optimal intelligent customer service response method, the optimized intelligent customer service response content is sent to the user, and the user's feedback information is received; Record the user's feedback information into the conversation context, and update the user's emotional state node and historical conversation records; Dynamically adjust the dialogue strategy based on user feedback information, updated emotional state nodes, and historical dialogue records.

8. An intelligent customer service interaction system, based on the intelligent customer service interaction method according to any one of claims 1 to 7, characterized in that: It includes user request acquisition module, emotion scoring module, emotion state classification module, response mode adjustment module and dialogue strategy adjustment module; A user request acquisition module, used to acquire user requests; The sentiment scoring module is used to calculate the comprehensive sentiment score based on user requests through multimodal sentiment analysis; The emotional state classification module is used to classify the user's emotional state according to the comprehensive emotional score, initialize the conversation context according to the classification results, and construct the conversation context map; The response mode adjustment module is used to dynamically adjust the response mode of the intelligent customer service according to the emotional state and user questions in the conversation context map, and collect user feedback information; The dialogue strategy adjustment module is used to dynamically adjust the dialogue strategy based on user feedback information and dialogue context.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the intelligent customer service interaction method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the intelligent customer service interaction method described in any one of claims 1 to 7 are implemented.

Images