Intelligent customer service interaction method and system

Abstract

The application discloses an intelligent customer service interaction method and system, and relates to the technical field of intelligent customer service interaction.The method comprises the following steps: obtaining a user request; calculating a comprehensive sentiment score through multi-modal sentiment analysis based on the user request; classifying the user's emotional state according to the comprehensive sentiment score, initializing the dialogue context, and constructing a dialogue context graph according to the classification result; dynamically adjusting the response mode of the intelligent customer service according to the emotional state and the user's question in the dialogue context graph, and collecting the feedback information of the user; and dynamically adjusting the dialogue strategy based on the feedback information of the user and the dialogue context.The application can comprehensively capture the emotional state of the user from multiple angles such as text, voice and facial expression through multi-modal sentiment analysis, improves the accuracy and robustness of emotion recognition, provides a reliable basis for subsequent dialogue management and response mode, and improves the experience satisfaction and service quality of the user.

Description

Technical Field

[0001] This invention relates to the field of intelligent customer service interaction technology, and in particular to an intelligent customer service interaction method and system. Background Technology

[0002] In recent years, with the rapid development of artificial intelligence technology, intelligent customer service has been increasingly widely applied across various industries. Intelligent customer service utilizes technologies such as Natural Language Processing (NLP), speech recognition, and computer vision to achieve efficient interaction with users, significantly improving the quality and efficiency of customer service. Traditional intelligent customer service primarily relies on text input, answering questions through predefined rules and simple machine learning models. However, with technological advancements, multimodal sentiment analysis has gradually become a research hotspot. It can analyze multiple modalities of information, including text, speech, and facial expressions, to more comprehensively understand a user's emotional state, thereby providing more personalized and accurate services. Multimodal sentiment analysis technology not only improves user satisfaction but also effectively reduces the workload of customer service personnel, enhancing corporate competitiveness.

[0003] Despite significant progress in the application of multimodal sentiment analysis technology in intelligent customer service, some shortcomings remain. First, existing intelligent customer service systems largely rely on data from a single modality for sentiment recognition, neglecting the complementary role of information from other modalities. This single-modal sentiment recognition method often fails to comprehensively and accurately capture the user's true emotional state. Second, existing intelligent customer service systems lack dynamic adjustment mechanisms in dialogue management; once the initial settings are completed, it is difficult to flexibly adjust based on real-time user feedback. Therefore, how to achieve more accurate sentiment analysis through multimodal sentiment analysis technology and, based on this, build a dynamically adjustable dialogue management mechanism has become a key issue that urgently needs to be addressed in current intelligent customer service systems. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides an intelligent customer service interaction method to solve the problems of inaccurate single-modal emotion recognition and lack of flexibility in dialogue management in the prior art.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0007] In a first aspect, the present invention provides an intelligent customer service interaction method, comprising: acquiring a user request; calculating a comprehensive sentiment score based on the user request through multimodal sentiment analysis; classifying the user's emotional state according to the comprehensive sentiment score, initializing a dialogue context based on the classification results, and constructing a dialogue context graph; dynamically adjusting the intelligent customer service response method according to the sentiment state and user questions in the dialogue context graph, and collecting user feedback information; and dynamically adjusting the dialogue strategy based on the user feedback information and the dialogue context.

[0008] Preferably, the user request includes text information entered through a chat window, voice messages sent through a voice assistant, and facial expression videos sent through a camera.

[0009] Preferably, the step of calculating a comprehensive sentiment score based on user requests through multimodal sentiment analysis includes the following specific steps:

[0010] Based on the AFINN sentiment dictionary, the text information is broken down into phrases, and each word is assigned a sentiment value according to the sentiment dictionary to calculate the text sentiment score. The expression is as follows:

[0011]

[0012] Where n is the number of phrases in the text, z i E is the sentiment value of the i-th phrase. t is the text sentiment score, and i is the phrase index variable;

[0013] Facial features are extracted using computer vision algorithms. Based on historical data and big data analysis, a range of facial feature emotion values ​​is defined. A facial emotion score is then calculated according to this range, expressed as:

[0014]

[0015] Where p is the number of facial features, u j E is the weight of the j-th facial feature. f It is a facial emotion score, where j is an index variable of facial features, and x is a facial emotion score. j It is the j-th facial feature value, x n It is the median of all facial feature values, x max It is the maximum value among all facial features, x min It is the minimum value among all facial features;

[0016] The facial features include the degree of eyebrow elevation, eye opening and closing, and the angle of mouth corner elevation.

[0017] Acoustic features are extracted using signal analysis. The emotional range of these acoustic features is defined based on historical data. A speech emotion score is then calculated based on this emotional range, expressed as:

[0018]

[0019] Where m is the number of acoustic features, w k It is the weight of the k-th acoustic feature, E v It is a speech emotion score, k is an index variable of acoustic features, and r is an index variable of acoustic features. k It is the k-th acoustic eigenvalue, r n It is the median of all acoustic eigenvalues, r max It is the maximum value among all acoustic features, r min It is the minimum value among all acoustic features;

[0020] The acoustic features include pitch, volume, speech rate, and fundamental frequency;

[0021] Text sentiment score E t Facial emotion score E f And voice emotion score E v Perform nonlinear fusion and calculate the nonlinear fusion value, expressed as:

[0022]

[0023] Where S is the nonlinear fusion value of text sentiment score, facial sentiment score and voice sentiment score, w1 is the weight coefficient of text sentiment feature, w2 is the weight coefficient of voice sentiment feature, w3 is the weight coefficient of facial sentiment feature, and α is the adjustment parameter for the degree of influence of nonlinear term on comprehensive sentiment score.

[0024] Based on the nonlinear fusion value S, the Sigmoid function is used to calculate the comprehensive sentiment score, expressed as follows:

[0025]

[0026] Where E is the overall sentiment score, c is the median of the sigmoid function, e is the base of the natural logarithm, and -b is the sensitivity modulator of the sentiment score to the nonlinear fusion value S.

[0027] Preferably, the step of classifying users' emotional states based on comprehensive emotional scores involves the following specific steps:

[0028] Based on historical user data and sentiment analysis reports, a positive threshold T1 and a negative threshold T2 are defined.

[0029] Users' emotional states are categorized based on their overall sentiment score E.

[0030] When E≥T1, the current user's emotional state is considered positive;

[0031] When T2 < E < T1, the previous user's emotional state is considered neutral.

[0032] When E≤T2, the current user's emotional state is considered negative.

[0033] Preferably, the initial dialogue context includes user ID, user's initial request content, user's initial emotional state, initial dialogue round, and historical dialogue records;

[0034] The specific steps for constructing the dialogue context graph are as follows:

[0035] Create an emotional state node based on the user's initial emotional state after initialization;

[0036] Create an initial request content node based on the initialized user initial request content;

[0037] Establish a directed edge between the emotional state node and the request content node, and link the emotional state and the request content using a causal relationship;

[0038] A directed graph is created based on a graph database, and emotion state nodes, request content nodes, and causal relationships between emotion nodes and request content nodes are added to the graph to construct a dialogue context graph.

[0039] Preferably, the steps for dynamically adjusting the intelligent customer service response based on the emotional state and user questions in the dialogue context graph, and collecting user feedback information, are as follows:

[0040] Read the current sentiment state node and the current request content node from the dialogue context graph to obtain the user's sentiment state and initial request content;

[0041] Use NLP to analyze user request content, extract user questions, and analyze question types;

[0042] Based on the user's question and emotional state, generate an initial response.

[0043] Based on historical dialogue records and user feedback in the dialogue context graph, the initial responses of the intelligent customer service are optimized.

[0044] The system dynamically selects the optimal response method for intelligent customer service based on the user's emotional state and the type of question.

[0045] The types of questions include consultation, help-seeking, complaint, and emotional support.

[0046] Preferably, the step of dynamically adjusting the dialogue strategy based on user feedback signals and dialogue context includes the following specific steps:

[0047] By using the optimal intelligent customer service response method, the optimized intelligent customer service response content is sent to the user, and the user's feedback information is received.

[0048] Record user feedback information into the dialogue context and update the user's emotional state node and historical dialogue records;

[0049] The dialogue strategy is dynamically adjusted based on user feedback, updated emotional state nodes, and historical dialogue records.

[0050] Secondly, this invention provides an intelligent customer service interaction system, comprising: a user request acquisition module, an emotion scoring module, an emotion state classification module, a response method adjustment module, and a dialogue strategy adjustment module; the user request acquisition module is used to acquire user requests; the emotion scoring module is used to calculate a comprehensive emotion score based on the user request through multimodal emotion analysis; the emotion state classification module is used to classify the user's emotion state according to the comprehensive emotion score, initialize the dialogue context according to the classification results, and construct a dialogue context graph; the response method adjustment module is used to dynamically adjust the intelligent customer service's response method according to the emotion state and user questions in the dialogue context graph, and collect user feedback information; the dialogue strategy adjustment module is used to dynamically adjust the dialogue strategy based on user feedback information and the dialogue context.

[0051] Thirdly, the present invention provides a computer device including a memory and a processor, wherein the memory stores a computer program, wherein the computer program, when executed by the processor, implements any step of the intelligent customer service interaction method as described in the first aspect of the present invention.

[0052] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein: when the computer program is executed by a processor, it implements any step of the intelligent customer service interaction method as described in the first aspect of the present invention.

[0053] The beneficial effects of this invention are as follows: Through multimodal sentiment analysis, it can comprehensively capture the user's emotional state from multiple perspectives such as text, voice, and facial expressions, improving the accuracy and robustness of sentiment recognition, providing a reliable foundation for subsequent dialogue management and response methods, and enhancing user experience satisfaction and service quality; by dynamically adjusting dialogue strategy steps, it can optimize the dialogue strategy in real time based on user feedback and dialogue context during each round of dialogue, ensuring the coherence and consistency of the dialogue, enhancing adaptability and flexibility, improving user satisfaction, reducing customer service costs, and enhancing the competitiveness of enterprises. Attached Figure Description

[0054] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0055] Figure 1 This is a flowchart of the intelligent customer service interaction method in Example 1.

[0056] Figure 2 This is a schematic diagram of the intelligent customer service interaction system in Example 1. Detailed Implementation

[0057] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0058] Example 1, referring to Figure 1 and Figure 2 This is the first embodiment of the present invention, which provides an intelligent customer service interaction method, including the following steps:

[0059] S1: Obtain user requests.

[0060] S1.1: User requests include text messages entered through a chat window, voice messages sent through a voice assistant, and facial expression videos sent through a camera.

[0061] It should be noted that the purpose of collecting the above data is to use multimodal sentiment analysis technology to comprehensively process text information, voice messages, and facial expression videos to achieve more comprehensive and accurate sentiment recognition, thereby providing users with a more personalized and precise intelligent customer service interaction experience.

[0062] S2: Based on user requests, calculate a comprehensive sentiment score through multimodal sentiment analysis.

[0063] S2.1: Based on the AFINN sentiment dictionary, the text information is broken down into phrases, and each word is assigned a sentiment value according to the sentiment dictionary to calculate the text sentiment score. The expression is as follows:

[0064]

[0065] Where n is the number of phrases in the text, z i E is the sentiment value of the i-th phrase. t is the text sentiment score, and i is the phrase index variable;

[0066] Furthermore, firstly, natural language processing techniques are used to break down the text information into multiple phrases; then, each word is assigned a corresponding sentiment value according to the AFINN sentiment dictionary, where positive sentiment values ​​are positive and negative sentiment values ​​are negative; finally, the average sentiment value of all phrases is calculated to obtain the final text sentiment score.

[0067] It should be noted that the reason for using AFINN is that it provides a widely recognized sentiment dictionary that can assign sentiment values ​​to words, making it suitable for text sentiment analysis and helping to accurately assess the emotional state of users.

[0068] S2.2: Facial features are extracted using computer vision algorithms. Based on historical data and big data analysis, a range of facial feature emotion values ​​is defined. A facial emotion score is calculated based on this range, expressed as:

[0069]

[0070] Where p is the number of facial features, u j E is the weight of the j-th facial feature. f It is a facial emotion score, where j is an index variable of facial features, and x is a facial emotion score. j It is the j-th facial feature value, x n It is the median of all facial feature values, x max It is the maximum value among all facial features, x min It is the minimum value among all facial features;

[0071] Facial features include the degree of eyebrow elevation, eye opening and closing, and the angle of mouth elevation;

[0072] Furthermore, using computer vision algorithms to extract facial features specifically refers to using facial detection and feature extraction techniques (such as algorithms in the OpenCV or Dlib libraries) to identify and extract key points of the face (such as eyes, mouth, eyebrows, etc.), and using machine learning models (such as convolutional neural networks CNN) to analyze the changing patterns of these features. Based on historical data and big data analysis, the range of emotional values ​​of facial features is defined, and then facial emotion scores are calculated.

[0073] Define the emotional value range for facial features as follows:

[0074] The emotional range of facial features is differentiated by setting thresholds to distinguish different emotional states of users. By analyzing multiple facial features, such as eyebrow elevation, eye opening and closing, and mouth corner upward angle, and combining historical data, an emotional threshold for each feature is defined. By comparing the current values ​​of these features with the preset thresholds, the user's emotional state can be classified as positive, neutral, or negative. This threshold-based sentiment analysis helps customer service systems better understand user emotions and thus adjust their dialogue strategies.

[0075] For example, the upward threshold X1 and downward threshold X2 of the corner of the mouth are defined based on historical data.

[0076] Positive emotion: The corners of the mouth turn up by more than or equal to X1, which is a smile.

[0077] Neutral emotion: The corners of the mouth are positioned between X2 and X1, and the corners of the mouth remain straight or slightly upturned.

[0078] Negative emotions: The corners of the mouth droop less than or equal to twice the normal range, showing a frown or annoyance.

[0079] S2.3: Using signal analysis, acoustic features are extracted. The emotional range of these acoustic features is defined based on historical data. The speech emotion score is then calculated according to the emotional range of the acoustic features, expressed as:

[0080]

[0081] Where m is the number of acoustic features, w k It is the weight of the k-th acoustic feature, E v It is a speech emotion score, k is an index variable of acoustic features, and r is an index variable of acoustic features. k It is the k-th acoustic eigenvalue, r n It is the median of all acoustic eigenvalues, r max It is the maximum value among all acoustic features, r min It is the minimum value among all acoustic features;

[0082] Acoustic characteristics include pitch, volume, speech rate, and fundamental frequency;

[0083] Specifically, the process involves: first, using audio processing tools (such as Librosa) to preprocess the speech signal and extract acoustic features such as pitch, volume, speech rate, and fundamental frequency; then, defining the emotional value range of the acoustic features based on historical data and big data analysis; and finally, calculating the speech emotion score according to the defined emotional value range to assess the user's emotional state.

[0084] Define the emotional value range of acoustic features as follows:

[0085] The emotional value range of acoustic features distinguishes different emotional states of users by setting thresholds. By analyzing multiple acoustic features, such as pitch, volume, speech rate, and fundamental frequency, and combining historical data, an emotional threshold for each feature is defined. By comparing the current values ​​of these features with the preset thresholds, the user's emotional state can be classified as positive, neutral, or negative. This threshold-based sentiment analysis helps intelligent customer service better understand the user's voice emotions, thereby adjusting dialogue strategies.

[0086] Example: Define positive sentiment threshold Y1 and negative sentiment threshold Y2 based on historical data for speech rate.

[0087] Positive Emotion: Speech rate greater than or equal to Y1, characterized by a fast and fluent speech rate.

[0088] Neutral emotional state: Speech rate is between Y2 and Y1, normal and moderate.

[0089] Negative emotions: Speech rate less than or equal to Y2, manifested as slow, hesitant, or procrastinating speech.

[0090] S2.4: Evaluate the text sentiment score. t Facial emotion score E f And voice emotion score E v Perform nonlinear fusion and calculate the nonlinear fusion value, expressed as:

[0091]

[0092] Where S is the nonlinear fusion value of text sentiment score, facial sentiment score and voice sentiment score, w1 is the weight coefficient of text sentiment feature, w2 is the weight coefficient of voice sentiment feature, w3 is the weight coefficient of facial sentiment feature, and α is the adjustment parameter for the degree of influence of nonlinear term on comprehensive sentiment score.

[0093] It should be noted that by non-linearly fusing text sentiment scoring, facial sentiment scoring, and voice sentiment scoring, and comprehensively considering sentiment information from multiple modalities, the accuracy and comprehensiveness of sentiment recognition are improved, thereby more accurately assessing the user's emotional state and providing a more personalized and effective interactive experience for intelligent customer service.

[0094] S2.5: Based on the non-linear fusion value S, the Sigmoid function is used to calculate the comprehensive sentiment score, expressed as follows:

[0095]

[0096] Where E is the overall sentiment score, S is the non-linear fusion value, c is the median of the sigmoid function, e is the base of the natural logarithm, and -b is the sensitivity moderating factor of the sentiment score to the non-linear fusion value S.

[0097] It should be noted that mapping the nonlinear fusion value S to the range of 0 to 1 makes the comprehensive sentiment score E more standardized and easier to interpret. The nonlinear transformation of the sigmoid function smoothly handles extreme values, avoiding large fluctuations in the comprehensive score due to outliers in individual modalities, thus improving the stability and reliability of the score. Furthermore, the standardized sentiment score E makes it easier to classify sentiment states, such as positive, neutral, and negative, thereby providing more accurate sentiment recognition and response strategies for intelligent customer service.

[0098] S3: Based on the comprehensive sentiment score, classify the user's sentiment state, initialize the dialogue context based on the classification results, and construct a dialogue context graph.

[0099] S3.1: Based on historical user data and sentiment analysis reports, define a positive threshold T1 and a negative threshold T2;

[0100] Furthermore, firstly, a large amount of historical user data is collected and organized, including users' text, voice, and facial expression data and their corresponding sentiment tags. Then, this data is processed using sentiment analysis algorithms to generate a sentiment score distribution. Next, based on the sentiment score distribution and the statistical results in the sentiment analysis report, the dividing points between positive and negative sentiment are determined and set as a positive threshold T1 and a negative threshold T2, respectively. This process is based on the sentiment score distribution in historical data and the statistical results in the sentiment analysis report, ensuring that the threshold settings accurately reflect the user's emotional state, thus providing a reliable reference standard for sentiment state classification.

[0101] S3.2: Classify the user's emotional state based on the comprehensive emotional score E;

[0102] When E≥T1, the current user's emotional state is considered positive;

[0103] This means that the user's overall emotional score exceeds the positive threshold T1, indicating that the user is in a positive, satisfied, or happy state.

[0104] When T2 < E < T1, the previous user's emotional state is considered neutral.

[0105] This means that the user's overall sentiment score is between the positive threshold T1 and the negative threshold T2, indicating that the user's emotional state is relatively stable, neither particularly positive nor particularly negative.

[0106] When E≤T2, the current user's emotional state is considered negative.

[0107] This means that if a user's overall emotional score is below the negative threshold T2, it indicates that the user is in a negative, dissatisfied, or unpleasant state.

[0108] S3.3: Initialize the dialogue context, including user ID, user's initial request content, user's initial emotional state, initial dialogue round, and historical dialogue records;

[0109] Specifically, the process is as follows: First, a user's unique identifier (user ID) is recorded to ensure the traceability of each user's identity. Then, the user's initial request content is acquired and recorded, including text messages, voice messages, and facial expression videos. Next, the user's initial emotional state is determined based on a comprehensive sentiment score (E), categorized as positive, neutral, or negative. Subsequently, the initial dialogue round is set to 1, indicating the start of the conversation. Finally, the historical dialogue record is initialized to empty, preparing for the recording of subsequent dialogue content and user feedback. This process ensures the integrity and coherence of the dialogue context, providing necessary initial information for intelligent customer service and supporting subsequent multi-turn dialogue management and sentiment state tracking.

[0110] S3.4: Construct a dialogue context graph. The specific steps are as follows:

[0111] S3.4.1: Create an emotion state node based on the user's initial emotion state after initialization;

[0112] Recording and representing a user's current emotional state in a dialogue context graph provides a foundation for subsequent emotional change tracking and dialogue management.

[0113] S3.4.2: Create an initial request content node based on the initialized user initial request content;

[0114] Record and represent the user's initial request information in the dialogue context graph to provide a foundation for subsequent dialogue understanding and response.

[0115] S3.4.3: Establish a directed edge between the sentiment state node and the request content node to link the sentiment state and the request content using a causal relationship;

[0116] Specifically, the process involves several steps: First, identifying the sentiment state node and the initial request content node from the dialogue context graph. Then, establishing a directed edge between these two nodes, pointing from the request content node to the sentiment state node, signifies that the user's request content led to the current sentiment state. This directed edge not only records the user's sentiment state but also clarifies the causal relationship between the sentiment state and the request content—that is, the user's request content is the cause of their current sentiment state. This process helps the intelligent customer service system better understand user emotional changes, providing strong support for subsequent dialogue management and sentiment response.

[0117] S3.4.4: Create a directed graph based on a graph database, add sentiment state nodes, request content nodes, and causal relationships between sentiment nodes and request content nodes to the graph, and construct a dialogue context graph.

[0118] It should be noted that, firstly, an empty directed graph is created using a graph database (such as Neo4j); then, sentiment state nodes and request content nodes are added to the graph as vertices; next, a directed edge is established between these two nodes, indicating that the request content caused the current sentiment state; finally, the relationships between these nodes and edges are stored in the graph database to form a complete dialogue context graph, providing structured data support for subsequent multi-turn dialogue management and sentiment state tracking.

[0119] S4: Based on the emotional state and user questions in the dialogue context graph, dynamically adjust the response method of the intelligent customer service and collect user feedback information.

[0120] S4.1: Read the current sentiment state node and the current request content node from the dialogue context graph to obtain the user's sentiment state and initial request content;

[0121] S4.2: Use NLP to analyze the user's request content, extract the user's question, and analyze the question type;

[0122] First, the user's request content is preprocessed, including removing stop words, punctuation marks, and numbers, and performing stemming or lemmatization to ensure the text is clean and standardized. Then, Natural Language Understanding (NLU) technology is used to extract the user's specific questions and intentions through Named Entity Recognition (NER) and Intent Recognition. Finally, based on the extracted questions and intentions, the user's questions are categorized into consultation, assistance, complaint, or emotional support categories, providing clear guidance for subsequent intelligent customer service responses.

[0123] S4.3: Generate preliminary response content based on the user's question and emotional state;

[0124] Specifically, firstly, NLP techniques are used to analyze the type of user's question, such as identifying a query like "What will the weather be like tomorrow?"; then, combined with the user's emotional state, such as positive (happy), positive and friendly responses are generated, such as: "Hello! It's a pleasure to help you. The weather forecast for tomorrow shows sunny skies and moderate temperatures, perfect for outdoor activities. Hope you have a wonderful day!" This approach accurately answers the user's question while also aligning with their current emotional state, thus enhancing the user experience.

[0125] S4.4: Optimize the initial intelligent customer service response content based on historical dialogue records and user feedback in the dialogue context graph;

[0126] Specifically, the process involves several steps. First, historical dialogue records are retrieved from the dialogue context graph to understand the user's previous interactions. Then, user feedback is analyzed to identify specific needs and preferences. Finally, the initial response is adjusted and supplemented based on the historical dialogue records and user feedback to make it more accurate and aligned with user expectations. For example, if the initial response is, "Hello! The weather forecast for tomorrow shows sunny skies and moderate temperatures, perfect for outdoor activities," but the user requests more detailed information, such as temperature range and wind conditions, the optimized response could be: "Hello! The weather forecast for tomorrow shows sunny skies with a daytime high of approximately 25°C and a nighttime low of approximately 15°C, with a wind force of around level 3, perfect for outdoor activities. We hope this information is helpful!" This approach better meets the user's needs and enhances the user experience.

[0127] S4.5: Dynamically select the optimal response method for intelligent customer service based on the user's emotional state and the type of the user's question;

[0128] The system dynamically selects the optimal response method for the intelligent customer service, as detailed below:

[0129] The emotional state is positive.

[0130] For inquiries: Choose to respond in text format, providing detailed information and answers.

[0131] Request for help: Choose to respond in text format, providing specific solutions and steps.

[0132] For complaints: Choose to respond in text, express your apology, and provide a solution.

[0133] Emotional support: Choose voice response to provide warm and comforting words.

[0134] The emotional state is neutral.

[0135] For inquiries: Choose to respond with text, providing concise information and answers.

[0136] Request for help: Choose to respond in text format, providing basic solutions and steps.

[0137] For complaints: Choose to respond in text, express your apology, and provide a basic solution.

[0138] Emotional support: Choose a text response to offer brief words of comfort and encouragement.

[0139] The emotional state is negative:

[0140] For inquiries: Choose to respond in text, providing detailed information and answers, and expressing concern.

[0141] For requests for help: Choose to respond with a voice message, providing specific solutions and steps, and expressing concern.

[0142] For complaints: Choose to respond with a voice message, express sincere apologies, and provide a detailed solution.

[0143] Emotional support: Choose a voice response to provide warm and comforting words and express care.

[0144] The types of questions include consultation, help-seeking, complaint, and emotional support.

[0145] S5: Dynamically adjust dialogue strategies based on user feedback and dialogue context.

[0146] S5.1: Send the optimized intelligent customer service response to the user through the best intelligent customer service response method, and receive the user's feedback information;

[0147] Feedback includes user satisfaction ratings for the intelligent customer service response, specific improvement suggestions, and emotional reactions. For example, a user might reply, "Thank you for your help, but could you provide more detailed logistics information? I'm feeling a bit anxious right now." This indicates that the user is only moderately satisfied with the initial response, has suggested improvements requiring more detailed information, and is expressing current anxiety.

[0148] S5.2: Record user feedback information into the dialogue context and update the user's emotional state node and historical dialogue records;

[0149] Furthermore, firstly, satisfaction ratings, improvement suggestions, and emotional responses are extracted from user feedback; then, this feedback information is recorded in the dialogue context graph, and the emotional state nodes are updated to reflect the user's current emotional changes; finally, the feedback information and responses are added to the historical dialogue record to ensure the integrity and coherence of the dialogue context.

[0150] S5.3: Dynamically adjust dialogue strategies based on user feedback, updated emotional state nodes, and historical dialogue records.

[0151] Specifically, the process involves: First, reading user feedback, such as, "Thank you for your help, but could you provide more detailed logistics information? I'm feeling a bit anxious right now."; then, updating the emotional state node from "neutral" to "negative" based on the user feedback to reflect the user's anxiety; next, recording the user's feedback and previous responses in the historical dialogue record to ensure the integrity and coherence of the dialogue context; and finally, dynamically adjusting the dialogue strategy to generate new response content.

[0152] For example: "I understand your anxiety. I will provide you with more detailed logistics information. Your order is being shipped by 'XX' courier, the tracking number is 'xxxxxxx,' and it is expected to arrive tomorrow afternoon. You can check the real-time logistics information through this link: [link]. If you have any other questions or need further assistance, please feel free to let me know, and we will resolve it for you as soon as possible. I hope this information helps you wait with peace of mind." This ensures that the response not only meets the user's emotional needs but also effectively solves the problem, improving user satisfaction.

[0153] This embodiment also provides an intelligent customer service interaction system, including: a user request acquisition module, an emotion scoring module, an emotion state classification module, a response method adjustment module, and a dialogue strategy adjustment module; the user request acquisition module is used to acquire user requests; the emotion scoring module is used to calculate a comprehensive emotion score based on user requests through multimodal emotion analysis; the emotion state classification module is used to classify user emotion states according to the comprehensive emotion score, initialize the dialogue context according to the classification results, and construct a dialogue context graph; the response method adjustment module is used to dynamically adjust the intelligent customer service response method according to the emotion state and user questions in the dialogue context graph, and collect user feedback information; the dialogue strategy adjustment module is used to dynamically adjust the dialogue strategy based on user feedback information and the dialogue context.

[0154] This embodiment also provides a computer device applicable to the intelligent customer service interaction method, including: a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions to implement the intelligent customer service interaction method proposed in the above embodiment.

[0155] The computer device can be a terminal, comprising a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad on the computer device's casing, or an external keyboard, touchpad, or mouse.

[0156] This embodiment also provides a storage medium storing a computer program, which, when executed by a processor, implements the intelligent customer service interaction method proposed in the above embodiments. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0157] In summary, this invention, through multimodal sentiment analysis, comprehensively captures users' emotional states from multiple perspectives, including text, voice, and facial expressions, improving the accuracy and robustness of sentiment recognition. This provides a reliable foundation for subsequent dialogue management and response methods, enhancing user experience satisfaction and service quality. Furthermore, by dynamically adjusting dialogue strategy steps, it optimizes dialogue strategies in real time based on user feedback and dialogue context during each round of conversation, ensuring dialogue coherence and consistency, enhancing adaptability and flexibility, improving user satisfaction, reducing customer service costs, and boosting enterprise competitiveness.

[0158] Example 2, referring to Table 1, is the second embodiment of the present invention. To further verify the technical solution of the present invention, experimental simulation data of the intelligent customer service interaction method are given.

[0159] To verify the application effect of the multimodal sentiment analysis technology of this invention in intelligent customer service, the following experiment was conducted. The experiment included 20 users who simulated user requests in real-world scenarios by entering text information in the chat window, sending voice messages through a voice assistant, and sending facial expression videos through a camera.

[0160] First, to verify the effectiveness of multimodal sentiment analysis technology in intelligent customer service, an experiment was conducted. The experiment involved 20 users who simulated real-world user requests by entering text messages in the chat window, sending voice messages through a voice assistant, and sending facial expression videos via camera.

[0161] Secondly, based on user requests, a comprehensive sentiment score is calculated using multimodal sentiment analysis technology. Specific steps include: performing sentiment analysis on the text information to calculate the text sentiment score; extracting facial features to calculate the facial sentiment score; and extracting acoustic features to calculate the speech sentiment score. Then, these three sentiment scores are fused to calculate the comprehensive sentiment score.

[0162] Next, based on the comprehensive sentiment score, the user's sentiment state is categorized, and the dialogue context is initialized. Specifically, the user's sentiment state is categorized as positive, neutral, or negative. Then, the user's unique identifier, initial request content, initial sentiment state, initial dialogue turn, and historical dialogue records are recorded to construct a dialogue context graph, providing support for subsequent multi-turn dialogue management and sentiment state tracking.

[0163] Finally, based on the emotional state and user questions in the dialogue context graph, the response method of the intelligent customer service is dynamically adjusted, and user feedback is collected to update the dialogue context graph and dynamically adjust the dialogue strategy to improve user satisfaction and interaction experience.

[0164] The details are shown in Table 1 below:

[0165] Table 1. Experimental Data of Multimodal Emotion Analysis

[0166]

[0167]

[0168] Analysis of the data in the table above clearly demonstrates the significant advantages and innovations of the multimodal sentiment analysis technology of this invention in intelligent customer service. For example, User 1's overall sentiment score was 0.75, categorized as positive, with an initial response satisfaction score of 7.1, which ultimately improved to 8.9; User 2's overall sentiment score was 0.40, categorized as negative, with an initial response satisfaction score of 4.2, which ultimately improved to 7.3. These data indicate that by combining emotional information from text, facial, and voice modalities, this invention can more comprehensively and accurately assess users' emotional states and significantly improve user satisfaction and interactive experience by dynamically adjusting dialogue strategies.

[0169] The intelligent customer service interaction method of this invention can not only more comprehensively and accurately assess the user's emotional state, but also dynamically adjust response strategies according to changes in the user's emotions, significantly improving user satisfaction and interaction experience. This demonstrates the innovation and practicality of this invention in multimodal sentiment analysis technology, effectively improving the performance of intelligent customer service and providing more personalized and precise services.

[0170] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An intelligent customer service interaction method, characterized in that: include, Get user request; Based on user requests, a comprehensive sentiment score is calculated using multimodal sentiment analysis. The specific steps for calculating this comprehensive sentiment score using multimodal sentiment analysis are as follows: Based on the AFINN sentiment dictionary, the text information is broken down into phrases, and each word is assigned a sentiment value according to the sentiment dictionary to calculate the text sentiment score. The expression is as follows: ； Where n is the number of phrases in the text. It is the sentiment value of the i-th phrase. is the text sentiment score, and i is the phrase index variable; Facial features are extracted using computer vision algorithms. Based on historical data and big data analysis, a range of facial feature emotion values ​​is defined. A facial emotion score is then calculated according to this range, expressed as: ； in, It is the number of facial features. It is the first The weight of each facial feature, It is a facial emotion score. It is an index variable for facial features. It is the j-th facial feature value. It is the median of all facial feature values. It is the maximum value among all facial features. It is the minimum value among all facial features; The facial features include the degree of eyebrow elevation, eye opening and closing, and the angle of mouth corner elevation. Acoustic features are extracted using signal analysis. The emotional range of these acoustic features is defined based on historical data. A speech emotion score is then calculated based on this emotional range, expressed as: ； in, It is the number of acoustic features. It is the weight of the k-th acoustic feature. It is a speech emotion score, and k is an index variable of acoustic features. It is the k-th acoustic eigenvalue. It is the median of all acoustic characteristic values. It is the maximum value among all acoustic features. It is the minimum value among all acoustic features; The acoustic features include pitch, volume, speech rate, and fundamental frequency; Text sentiment scoring Facial Emotion Score And voice emotion score Perform nonlinear fusion and calculate the nonlinear fusion value, expressed as: ； in, It is a non-linear fusion value of text sentiment score, facial sentiment score, and voice sentiment score. These are the weighting coefficients for the text's sentiment features. These are the weighting coefficients for the emotional features of speech. These are the weighting coefficients for facial emotional features. It is a moderating parameter for the degree of influence of nonlinear terms on the overall sentiment score; Based on nonlinear fusion value The Sigmoid function is used to calculate the overall sentiment score, and the expression is: ； in, It is a comprehensive sentiment score. It is the median of the sigmoid function. It is the base of the natural logarithm. It is the sensitivity moderating coefficient of the sentiment score to the nonlinear fusion value S; Based on the comprehensive sentiment score, the user's sentiment state is classified. Based on the classification results, the dialogue context is initialized and a dialogue context graph is constructed. Based on the emotional state and user questions in the dialogue context graph, the intelligent customer service dynamically adjusts its response method and collects user feedback information. The dialogue strategy is dynamically adjusted based on user feedback and the context of the conversation.

2. The intelligent customer service interaction method as described in claim 1, characterized in that: The user requests include text messages entered through a chat window, voice messages sent through a voice assistant, and facial expression videos sent through a camera.

3. The intelligent customer service interaction method as described in claim 1, characterized in that, The process of classifying users' emotional states based on comprehensive sentiment scores involves the following steps. Based on historical user data and sentiment analysis reports, a positive threshold T1 and a negative threshold T2 are defined. Based on comprehensive sentiment score To classify users' emotional states; when If the value is greater than or equal to T1, then the current user's emotional state is considered positive. When T2 < If the value is less than T1, then the previous user's emotional state is considered neutral. when If the value is less than or equal to T2, then the current user's emotional state is considered negative.

4. The intelligent customer service interaction method as described in claim 1, characterized in that: The initial dialogue context includes user ID, user's initial request content, user's initial emotional state, initial dialogue round, and historical dialogue records; The specific steps for constructing the dialogue context graph are as follows: Create an emotional state node based on the user's initial emotional state after initialization; Create an initial request content node based on the initialized user initial request content; Establish a directed edge between the emotional state node and the request content node, and link the emotional state and the request content using a causal relationship; A directed graph is created based on a graph database, and emotion state nodes, request content nodes, and causal relationships between emotion nodes and request content nodes are added to the graph to construct a dialogue context graph.

5. The intelligent customer service interaction method as described in claim 1, characterized in that: The process of dynamically adjusting the intelligent customer service response based on the emotional state and user questions in the dialogue context graph, and collecting user feedback information, involves the following specific steps: Read the current sentiment state node and the current request content node from the dialogue context graph to obtain the user's sentiment state and initial request content; Use NLP to analyze user request content, extract user questions, and analyze question types; Based on the user's question and emotional state, generate an initial response. Based on historical dialogue records and user feedback in the dialogue context graph, the initial responses of the intelligent customer service are optimized. The system dynamically selects the optimal response method for intelligent customer service based on the user's emotional state and the type of question. The types of questions include consultation, help-seeking, complaint, and emotional support.

6. The intelligent customer service interaction method as described in claim 1, characterized in that: The specific steps for dynamically adjusting the dialogue strategy based on user feedback signals and dialogue context are as follows. By using the optimal intelligent customer service response method, the optimized intelligent customer service response content is sent to the user, and the user's feedback information is received. Record user feedback information into the dialogue context and update the user's emotional state node and historical dialogue records; The dialogue strategy is dynamically adjusted based on user feedback, updated emotional state nodes, and historical dialogue records.

7. An intelligent customer service interaction system, based on the intelligent customer service interaction method according to any one of claims 1 to 6, characterized in that: This includes a user request acquisition module, an emotion scoring module, an emotion state classification module, a response method adjustment module, and a dialogue strategy adjustment module; The user request acquisition module is used to acquire user requests; The sentiment scoring module is used to calculate a comprehensive sentiment score based on user requests through multimodal sentiment analysis; The emotion state classification module is used to classify the user's emotion state based on the comprehensive emotion score, initialize the dialogue context based on the classification results, and construct the dialogue context graph. The response method adjustment module is used to dynamically adjust the response method of the intelligent customer service based on the emotional state and user questions in the dialogue context graph, and to collect user feedback information. The dialogue strategy adjustment module is used to dynamically adjust the dialogue strategy based on user feedback and dialogue context.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the intelligent customer service interaction method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the intelligent customer service interaction method according to any one of claims 1 to 6.

Images