Online customer service user emotion recognition and prompt system based on semantic emotion recognition

Abstract

The invention discloses an online customer service user emotion recognition and prompt system based on semantic emotion recognition, which is characterized in that the system is realized by adopting a deep learning model, and the deep learning model comprises a word segmentation module; a word embedding module; an emotion perception module; and an output end. According to the method, the contents of multiple rounds of dialogues can be contacted, the emotion of the user can be accurately identified, and clear and practical feedback can be provided for an artificial customer service in real time so as to optimize the interactive behavior between the customer service and the customer.

Description

Technical Field

[0001] The present invention relates to an emotion recognition system based on deep learning, which focuses on providing real-time emotion insights for manual customer service. The present invention utilizes advanced natural language processing technology, combined with contextual understanding, to identify subtle emotional changes in user messages, and instantly presents them to customer service personnel through an intuitive interface, helping customer service personnel to formulate response strategies. Background Art

[0002] The online customer service industry is undergoing a transformation driven by AI, especially the application of emotional intelligence, which aims to improve customer satisfaction and loyalty. However, human customer service is still the key to handling complex issues and building deep customer relationships. To assist human customer service, there have been initial attempts based on text and voice analysis to identify customer emotions, but these tools often have limited accuracy, are not easy to capture changes in user emotions, and are difficult to integrate into actual workflows.

[0003] Online customer service assistance systems currently on the market typically use basic natural language processing (NLP) techniques and sentiment dictionaries to try to infer emotional states from user messages. These systems may label positive, negative, or neutral tones, but often do not work well when dealing with ambiguous or complex emotions, and have difficulty applying this information instantly in actual customer service conversations. Specifically, existing online customer service assistance systems have the following shortcomings:

[0004] 1. The sentiment dictionary needs to be constantly updated and maintained: Rule-based sentiment recognition tools maintain a large sentiment dictionary, assign sentiment types to each possible word, traverse the sentiment type distribution of words in a sentence, and give the sentiment type of the entire sentence. If new possible sentiment-related words appear, the sentiment dictionary needs to be updated and maintained;

[0005] 2. Complex recognition rules need to be maintained: In addition to maintaining the sentiment dictionary, in order to perform a certain degree of context perception and logical reasoning, it is also necessary to maintain a degree adverb dictionary and a negation dictionary, and formulate a large number of reasoning rules based on them. If there are words or sentence structures that cannot be correctly recognized by the rules, new rules need to be added;

[0006] 3. Difficulty in handling complex emotions or subtle changes in tone: Even if rule-based emotion recognition maintains a large number of sentiment words and recognition rules, when handling complex emotions or subtle changes in tone, existing systems often have difficulty accurately identifying them because the rules are difficult to make fine-grained distinctions and judgments. Some emotion recognition methods based on machine learning can often only give the emotion type of a single paragraph of text, and cannot combine the context to analyze the emotional changes of each person in a multi-person conversation in real time to help customer service formulate effective customer service strategies.

[0007] At the same time, traditional emotion recognition algorithms have the following disadvantages:

[0008] 1. The emotional dictionary needs to be constantly updated and maintained: The emotional dictionary is the basis of traditional emotion recognition algorithms. It presets a set of correspondences between words and emotional polarity (positive, negative, neutral). However, the richness and flexibility of the Chinese language, especially the rapid development of Internet language, has led to the emergence of new words and new expressions. For example, Internet buzzwords such as "gei li" and "lan shou xiang gu" do not have obvious emotional polarity and need to be included in the dictionary in a timely manner. If the emotion recognition system fails to recognize these words, the recognition results will be distorted.

[0009] 2. Complex recognition rules need to be maintained: Sentiment recognition rules usually involve the following types: Sentiment vocabulary matching rules that determine the sentiment tendency of the text based on the positive or negative score of the vocabulary when a word in the sentiment dictionary appears in the text; Negation word rules that may reverse the sentiment tendency if there is a negative word (such as "no" or "no") before the sentiment word; Degree adverb rules, where degree adverbs (such as "very" or "extremely") can enhance or weaken the intensity of the subsequent sentiment words; for example, the sentiment intensity expressed by "very like" is greater than that of "like". In order to more accurately identify sentiment, it is necessary to give the intensity of the sentiment impact of each sentiment-related word on the sentence, and configure a large number of rules to infer the final sentiment score of the sentence; if you want to deal with complex emotions or context-dependent scenarios, you need to configure more complex reasoning rules;

[0010] 3. Difficulty in handling complex emotions or subtle changes in tone: Emotional expressions in the real world are often mixed, and a single emotional dictionary and simple rules can hardly capture this complexity and nuance. For example, a sentence may express sadness and hope at the same time, or a positive word with an ironic meaning may actually convey a negative emotion. Or, during the course of speaking, the speaker may change the modal particles or symbols used previously, which often implies a change in the speaker's emotions. Traditional algorithms often have difficulty distinguishing these complex emotions and subtle changes in tone, resulting in inaccurate recognition results. Summary of the invention

[0011] The purpose of the present invention is to address the deficiencies of the prior art in assisting human customer service to identify and respond to user emotions in real time, especially when dealing with complex emotions and rapidly changing conversation environments.

[0012] In order to achieve the above object, the technical solution of the present invention is to provide an online customer service user emotion recognition and prompting system based on semantic emotion recognition, characterized in that it is implemented by using a deep learning model, and the deep learning model includes:

[0013] The word segmentation module is used to extract words from the raw text corresponding to the entire conversation;

[0014] The word embedding module is used to convert the words extracted by the word segmentation module into word vectors;

[0015] The emotion perception module is used to perceive and remember the conversation context, and further includes a situation context feature extraction unit and a speaker context feature extraction unit. The situation context feature extraction unit and the speaker context feature extraction unit use the same neural network structure:

[0016] The emotion perception module converts all word vectors belonging to the same sentence into text features of the same dimension through the convolution layer and the maximum pooling layer, and obtains the sentence text feature u of the current i-th sentence in the entire conversation. i , then the entire conversation is an ordered list of sentence text features in the conversation, represented by U = [u1,u2,…,u N ]; At the same time, the sentiment perception module maintains the sentence text features u i Mapping to speaker number and the mapping of the global sequence number to the local sequence number of the statement φ(i,k), then the sentence text feature u i By speaker p k Issued and is the speaker The jth sentence said,

[0017] The sentence text feature u i and As the input of the scenario context feature extraction unit, the scenario context feature extraction unit outputs and in, and are the sentence text features u in the entire conversation i Corresponding situational context text output features and situational context memory features;

[0018] The sentence text feature u i and As the input of the speaker context feature extraction unit, the speaker context feature extraction unit outputs and in, and are speakers p k The jth sentence u i Corresponding speaker context text output features and speaker context memory features;

[0019] The scene context features output by the scene context feature extraction unit and the speaker context features output by the speaker context feature extraction unit are concatenated into a vector, and after two layers of full connection and one layer of Dropout, a deep feature combining the scene context and the speaker context is obtained as the output of the emotion perception module;

[0020] The output end is used to convert the deep features into a sentiment score for each person and each sentence.

[0021] Preferably, the neural network is an LSTM network, a Transformer network or a GRU network.

[0022] Preferably, the sentiment score ranges from [-1, 1], where a negative number represents a negative sentiment and a positive number represents a positive sentiment. A larger absolute value of the score indicates a stronger sentiment.

[0023] Preferably, the output terminal comprises:

[0024] Multi-layer fully connected layer: The deep features are obtained through multi-layer fully connected layers and are consistent with the sentence text features u i The corresponding 1D output x i ;

[0025] Output activation function, used to convert the 1-dimensional output x i Mapped to an output between -1 and 1, as shown below:

[0026]

[0027] Preferably, a visualization module is also included, which is used to display the sentiment score output by the deep learning model in real time to the customer service system.

[0028] Preferably, when the deep learning model is trained, for each sentence u i Predicted sentiment score And the true sentiment score y of this sentence i Calculate the loss function as shown below:

[0029]

[0030] Where N is the number of samples;

[0031] After obtaining the loss value RMSE, the parameters of the deep learning model are updated using a small batch gradient descent algorithm so that the predicted value gradually approaches the true value, thereby obtaining the final model parameters.

[0032] Preferably, the deep learning model is integrated into the workflow of online customer service, deployed on multiple servers with GPUs, and uses Python's Web service framework to provide a calling interface to the outside world, and performs load balancing to disperse the pressure; when the service starts, the deep learning model is loaded into the video memory, and only inference operations are performed when the service is called, and the recognition results are returned to the customer service system in real time for display and monitoring.

[0033] Preferably, a predetermined number of intermediate features that need to be saved for continuous network reasoning and analysis are generated during a session, and these intermediate features are directly saved in the server memory.

[0034] Preferably, the session ID, content, speaker, and sentiment score of each conversation are written into the Kafka message queue, and the Kafka consumer service asynchronously writes these data into the database. The monitoring service monitors whether there is abnormal data in the database and issues monitoring alarms.

[0035] The present invention can not only link the content of multiple rounds of conversations and accurately identify the user's emotions, but also provide clear and practical feedback to the manual customer service in real time to optimize the interaction between the customer service and the customer. Compared with the existing technical solutions, the innovation of the present invention lies in:

[0036] 1. Proposed an emotion recognition model that can connect to the context of multi-person conversations:

[0037] a) Abstract the emotional reasoning problem in the conversation into the change of neuron state in the neural network model;

[0038] b) Extract the situation context state and speaker context state respectively, and identify each person’s emotion in the current conversation situation;

[0039] 2. Give a normalized sentiment score that can effectively measure the emotional changes of each speaker in a multi-person conversation:

[0040] a) Intuitively reflect the emotional changes of each speaker in the conversation;

[0041] b) Used by the system to monitor the emotional state of all conversations:

[0042] 3. Integrate the proposed emotion recognition model into the online customer service workflow:

[0043] a) To speed up model inference, the model is deployed on a machine with a GPU and FastAPI is used to provide external services;

[0044] b) To improve service throughput and prevent single machine failure, services are deployed on multiple machines to balance load and distribute pressure;

[0045] c) When the service is started, the model is loaded into the video memory. When the service is called, only inference operations need to be performed, and the recognition results are returned to the customer service system in real time for display and monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The system structure of the present invention is illustrated;

[0047] Figure 2 The sentiment analysis network cluster is illustrated;

[0048] Figure 3 A curve showing the continuous changes in customer sentiment. DETAILED DESCRIPTION

[0049] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0050] like Figure 1 As shown, an embodiment of the present invention discloses an online customer service user emotion recognition and prompting system based on semantic emotion recognition. The system is implemented based on a deep learning model that can extract context-related emotion scores of each sentence in a conversation in real time, and mainly includes: a word segmentation module for extracting words from sentences, a word embedding module for mapping discrete words to a continuous word vector space, an emotion perception module for perceiving and memorizing conversation contexts, an output end composed of a fully connected layer and a Tanh activation function, and a visualization module for displaying the model output in real time in the customer service system.

[0051] The word segmentation module divides continuous text into independent words, preparing for subsequent word embedding and sentiment analysis. Word segmentation is the first step in natural language processing, which is particularly important for Chinese because Chinese does not have clear delimiters between words. Word segmentation tools such as jieba, THULAC or LTP can identify word boundaries in text and convert text into word sequences for subsequent processing. The accuracy of word segmentation directly affects the performance of subsequent steps, so it is very important to choose appropriate word segmentation tools and parameters. Jieba word segmentation is a very popular open source word segmentation tool that supports precise mode, full mode and search engine mode, which can be flexibly selected according to different needs. The word segmentation module in the disclosed system of the embodiment of the present invention uses jieba word segmentation by default, and can also be switched to other tools through configuration.

[0052] The word embedding module converts the discrete words obtained by the word segmentation module into continuous vector representations to capture the semantic and grammatical information of the words. Word embedding maps each word to a high-dimensional vector space. These vectors can not only represent the semantics of the word, but also reflect the similarity and correlation between words. Word embedding methods such as Word2Vec, GloVe, and BERT can generate high-quality word vectors through learning from large-scale corpora. When using deep learning models for natural language processing, discrete word segmentation numbers are usually converted into continuous word vector representations. In deep learning models, word embeddings in sentences are used as input layers and converted into text features through pooling and convolution layers, which can provide semantically rich feature representations for the model.

[0053] The emotion perception module is used to distinguish the context at the situation level and the speaker level. In an embodiment of the present invention, the emotion perception module is composed of a bidirectional LSTM for extracting the situation context. In order to build a neural network model that can handle complex emotions and subtle changes in tone in multi-person conversation scenarios, it is first necessary to convert the entire conversation into a vector representation and distinguish the speakers in the conversation while retaining the order of the conversation. In addition, it is necessary to design a series of intermediate features to capture and remember the emotional changes in the conversation, and finally obtain the predicted value of the emotional score. In order to model the emotion recognition problem, the following concepts need to be defined:

[0054] 1. Text features: The vector representation of the sentences in the conversation obtained by the word embedding module retains the conversation order and sentences. The entire conversation is an ordered list of text features of the sentences in the conversation, expressed as:

[0055] U=[u1,u2,…,u N ]

[0056] At the same time, the maintenance statement u i Mapping to speaker number and the mapping of statement global serial numbers to local serial numbers φ(i,k), then statement u i By the speaker Issued and is the speaker Said sentence.

[0057] 2. Situational context features: Situational context features are divided into situational context text output features and situational context memory features, which correspond to the output value and hidden state in LSTM respectively. The situational context text output features are used to calculate the sentiment score of the current conversation, and the situational context memory features are used to save the conversation context and calculate the situational context text output features and situational context memory features of the next conversation. Both features are calculated by LSTM, and the calculation method is as follows:

[0058]

[0059] Among them, LSTM s It is an LSTM network used to extract situational context features in the emotion perception module. The initial value of the context memory feature is a zero vector. and are the i-th sentence u in the whole conversation. i Corresponding situational context text output features and situational context memory features.

[0060] 3. Speaker context features: Similar to the situational context features, speaker context features are also divided into speaker context text output features and speaker context memory features. The difference is that speaker context features are extracted using a different LSTM network from situational context features, and for each speaker, the speaker context features of the speaker are calculated separately. The calculation method is:

[0061]

[0062] And φ(i,k)=j

[0063] in, and The speaker p k The jth sentence u i The corresponding speaker context text output features and speaker context memory features. The LSTM network saves the speaker context memory features of each speaker separately When mapping When the speaker p k Speaker context memory features and the statement u i As input, a new speaker context feature is obtained. LSTM network for extracting speaker context features p , in terms of model structure and LSTM network LSTM s Same, u i is input to LSTM at the same time p and LSTM s But according to u i Input to LSTM according to different speakers p The memory features in LSTM are different. s The output is the contextual features of the entire conversation scenario, LSTM p The output is the contextual features of different speakers. After training, LSTM p It can distinguish different speakers. LSTM sIt can take into account the context of the entire conversation scenario while calculating the sentiment score of a single speaker.

[0064] LSTM s Output context features and LSTM p The output speaker context features are concatenated into a vector and passed through two layers of full connection + Dropout layers to obtain deep features that combine the situation context and speaker context. The deep features are input to the output end, which converts the situation context and speaker context into sentiment scores for each person and each sentence. The purpose of the model is to distinguish the sentiment tendency of each sentence, such as negative or positive. In order to distinguish the emotional changes in the conversation in a more fine-grained manner, it is necessary to map the deep features into continuous sentiment scores instead of discrete labels as the output of the model. The sentiment score ranges from [-1,1]. Negative numbers represent negative emotions, and positive numbers represent positive emotions. The larger the absolute value of the score, the stronger the emotion. In order to obtain the sentiment score in the above range, tanh is selected as the output activation function at the output end of the model. The formula is:

[0065]

[0066] Among them, x i is the input sentence u i The deep features obtained when passing through multiple layers of fully connected layers get a 1-dimensional output. This function f(x i ) Each sentence u i Get any real number x i Mapped to an output between -1 and 1 to obtain the final predicted sentiment score.

[0067] The visualization module displays the predicted sentiment scores output by the model in real time in the customer service system to help customer service make decisions and monitor all conversations in the system in real time.

[0068] When training the model, for each sentence u i Predicted sentiment score All need to match the real emotional score of this sentence i To calculate the loss function, the present invention uses the root mean square error, and the calculation formula is:

[0069]

[0070] After obtaining the loss value, the mini-batch gradient descent algorithm is used to update the parameters of the entire model so that the predicted value gradually approaches the true value and the final model parameters are obtained.

[0071] In many years of online customer service work, we have accumulated a large amount of customer service conversation data. After desensitizing this data, we can get a large amount of unlabeled corpus data. This data is handed over to sentiment experts for analysis, and sentiment labels are added to each sentence. Five options are provided for labeling: very positive, positive, neutral, negative, and very negative. The corresponding sentiment scores are 1.0, 0.5, 0.0, -0.5, and -1.0, which can be used for model training. Although only five options are provided for labeling, because the output of the model is a continuous value with a value of [-1.0, 1.0], the final score is a continuous value as close to the label as possible, which can reflect the continuous and subtle changes in the emotions of each sentence of the customer service and the customer during the conversation.

[0072] Deploy the trained model to multiple servers with GPUs and use FastAPI to provide an HTTP calling interface, which consists of Figure 2 The sentiment analysis network cluster shown in the figure. During a conversation, a small number of intermediate features (historical text features, context memory features) that need to be saved are generated for continuous reasoning and analysis of the network. In order to improve efficiency, these features are directly saved in the server memory. Therefore, each request for the same conversation is required to call the same server for sentiment analysis. To this end, the session ID is placed in the URL path parameter of the request, and URL_HASH is used as the load balancing strategy. The load balancing server will evenly and directionally distribute all requests to each server in the cluster according to different session IDs.

[0073] In addition to immediately returning the sentiment score of each sentence to the client for front-end display and prompts, the system also writes information such as the session ID, content, speaker, sentiment score, etc. into the Kafka message queue. The Kafka consumer service will asynchronously write this data to the database. The monitoring service monitors whether there is abnormal data in the database and issues monitoring alarms.

[0074] During the customer service conversation, you can see the curve of the customer's emotions changing continuously, such as Figure 3 shown.

[0075] The emotional score line will show a variety of changing trends. If the emotional trend is rising, it means that the customer service strategy is appropriate. If the emotional score is on a downward trend, it means that the customer service may need to change the response strategy at this time. In addition to the two trends of rising and falling, there may also be trends such as little change and violent fluctuations. When the customer's emotional curve is in different trends, the customer service can adopt different response strategies. For conversations with good trends, they can be used as excellent cases for internal learning and analysis by the customer service. For controversial score results, the customer service can make an appeal. The results of the appeal are judged and re-labeled by emotional experts and included in the backup training data. When the backup training data accumulates to a certain amount, the model will be re-trained together with the previous training data, and the model parameters will be updated continuously through continuous iteration.

Claims

1. An online customer service user emotion recognition and prompting system based on semantic emotion recognition, characterized in that: The deep learning model is used for implementation, and the deep learning model includes: The word segmentation module is used to extract words from the raw text corresponding to the entire conversation; The word embedding module is used to convert the words extracted by the word segmentation module into word vectors; The emotion perception module is used to perceive and remember the conversation context, and further includes a situation context feature extraction unit and a speaker context feature extraction unit. The situation context feature extraction unit and the speaker context feature extraction unit use the same neural network structure: The emotion perception module converts all word vectors belonging to the same sentence into text features of the same dimension through the convolution layer and the maximum pooling layer, and obtains the sentence text feature u of the current i-th sentence in the entire conversation. i , then the entire conversation is an ordered list of sentence text features in the conversation, represented by U = [u1,u2,…,u N ]; At the same time, the sentiment perception module maintains the sentence text features u i Mapping to speaker number and the mapping of the global sequence number to the local sequence number of the sentence φ(i,k), then the sentence text feature u i By speaker p k Issued and is the speaker The jth sentence said, The sentence text feature u i and As the input of the scenario context feature extraction unit, the scenario context feature extraction unit outputs and in, and are the sentence text features u in the entire conversation i Corresponding situational context text output features and situational context memory features; The sentence text feature u i and As the input of the speaker context feature extraction unit, the speaker context feature extraction unit outputs and in, and are speakers p k The jth sentence u i Corresponding speaker context text output features and speaker context memory features; The scene context features output by the scene context feature extraction unit and the speaker context features output by the speaker context feature extraction unit are concatenated into a vector, and after two layers of full connection and one layer of Dropout, a deep feature combining the scene context and the speaker context is obtained as the output of the emotion perception module; The output end is used to convert the deep features into a sentiment score for each person and each sentence.

2. The online customer service user emotion recognition and prompting system based on semantic emotion recognition as claimed in claim 1, characterized in that: The neural network is an LSTM network, a Transformer network or a GRU network.

3. The online customer service user emotion recognition and prompting system based on semantic emotion recognition as claimed in claim 1, characterized in that: The value range of the sentiment score is [-1, 1], negative numbers represent negative sentiment, positive numbers represent positive sentiment, and the larger the absolute value of the score, the stronger the sentiment.

4. The online customer service user emotion recognition and prompting system based on semantic emotion recognition as claimed in claim 1, characterized in that: The output terminal comprises: Multi-layer fully connected layer: The deep features are obtained through multi-layer fully connected layers and are consistent with the sentence text features u i The corresponding 1D output x i ; Output activation function, used to convert the 1-dimensional output x i Mapped to an output between -1 and 1, as shown below:

5. The online customer service user emotion recognition and prompting system based on semantic emotion recognition as claimed in claim 1, characterized in that: It also includes a visualization module for displaying the sentiment score output by the deep learning model in real time in the customer service system.

6. The online customer service user emotion recognition and prompting system based on semantic emotion recognition as claimed in claim 1, characterized in that: When the deep learning model is trained, for each sentence u i Predicted sentiment score And the true sentiment score y of this sentence i Calculate the loss function as shown below: Where N is the number of samples; After obtaining the loss value RMSE, the parameters of the deep learning model are updated using a small batch gradient descent algorithm so that the predicted value gradually approaches the true value, thereby obtaining the final model parameters.

7. The online customer service user emotion recognition and prompting system based on semantic emotion recognition as claimed in claim 1, characterized in that: The deep learning model is integrated into the workflow of online customer service, deployed on multiple servers with GPUs, and a Python Web service framework is used to provide a calling interface to the outside world, and load balancing is performed to disperse the pressure. When the service is started, the deep learning model is loaded into the video memory, and only inference operations are performed when the service is called, and the recognition results are returned to the customer service system in real time for display and monitoring.

8. The online customer service user emotion recognition and prompting system based on semantic emotion recognition as claimed in claim 7, characterized in that: During a session, a predetermined number of intermediate features are generated that need to be saved for continuous network reasoning and analysis, and these intermediate features are directly saved in the server memory.

9. The online customer service user emotion recognition and prompting system based on semantic emotion recognition as claimed in claim 7, characterized in that: The conversation ID, content, speaker, and sentiment score of each conversation are written into the Kafka message queue. The Kafka consumer service asynchronously writes this data into the database. The monitoring service monitors whether there is abnormal data in the database and issues monitoring alarms.

Images