Emotion recognition method, device and electronic device based on customer service communication data

Abstract

The present invention belongs to the field of data information processing technology, and provides an emotion recognition method, device, electronic device and recording medium based on customer service communication data, the method comprising: collecting communication data between customer service and users, and performing preliminary emotion recognition on the communication data; using an emotion recognition model to perform first emotion recognition on the communication data based on the preliminary emotion recognition result, and judging whether the first emotion recognition result meets the standard, and if so, saving the emotion recognition model and putting it online; after the emotion recognition model is put online, it is iterated through self-learning, and the iterated emotion recognition model is used to perform second emotion recognition. The present invention completes the construction of the emotion recognition model through preliminary emotion recognition and first emotion recognition, and after going online, a large amount of online communication data is used to iterate and upgrade the emotion recognition model through self-learning, so that extreme emotions can be more accurately recognized.

Description

Technical Field

[0001] The present invention belongs to the technical field of data information processing, and is particularly suitable for data information processing in online services. More specifically, it relates to an emotion recognition method, device and electronic equipment based on customer service communication data. Background Art

[0002] The customer service center is the main bridge for communication between enterprises and users, and the main channel for improving user satisfaction. In the past, the customer service center was mainly based on manual customer service, with professional customer service personnel providing services to users.

[0003] With the development of computer information processing technology, more and more customer service centers have begun to use voice robots to serve users, alleviating the problem of long waiting time for manual customer service. In the process of communicating with users, it is necessary to identify users' emotions in order to provide a better user experience.

[0004] However, the voice robots currently in use can only recognize the user's general emotions. For example, thanks include polite thanks, general thanks, and sincere thanks. The voice robot can recognize that the user is expressing thanks, but it cannot determine whether the user is polite or sincere, and cannot adjust the strategy according to the situation, resulting in a very poor user experience.

[0005] In addition, the amount of data on extreme emotions in the communication data center is very small, resulting in few samples, which makes it difficult to train voice robots. This makes it more difficult for voice robots to recognize extreme emotions. Summary of the invention

[0006] 1. Technical issues to be resolved

[0007] The present invention aims to solve the problem that the voice robots used in existing customer service centers cannot recognize extreme emotions.

[0008] (II) Technical solution

[0009] In order to solve the above technical problems, one aspect of the present invention provides an emotion recognition method based on customer service communication data, comprising:

[0010] S1. Collect communication data between customer service and users, and perform preliminary emotion recognition on the communication data;

[0011] S2, using the emotion recognition model to perform a first emotion recognition on the communication data based on the preliminary emotion recognition result, and determining whether the first emotion recognition result meets the standard, and if so, saving and putting the emotion recognition model online;

[0012] S3. After the emotion recognition model is put online, it is iterated through self-learning, and the iterated emotion recognition model is used to perform a second emotion recognition.

[0013] According to a preferred embodiment of the present invention, performing preliminary emotion recognition on the communication data further comprises:

[0014] S101, purifying the communication data using sample-free learning;

[0015] S102, manually performing preliminary screening on the purified communication data.

[0016] According to a preferred embodiment of the present invention, performing first emotion recognition on the communication data based on the preliminary emotion recognition result using the emotion recognition model further comprises:

[0017] S201, manually labeling the preliminary emotion recognition results;

[0018] S202, creating training data using the annotated preliminary emotion recognition results;

[0019] S203, training the emotion recognition model using the training data;

[0020] S204: Use the trained emotion recognition model to perform a first emotion recognition on the communication data.

[0021] According to a preferred embodiment of the present invention, determining whether the first emotion recognition result meets the standard further includes:

[0022] A first recall rate threshold and a first accuracy rate threshold are preset to determine whether the first emotion recognition result satisfies both the first recall rate threshold and the first accuracy rate threshold. If so, the first emotion recognition result meets the criteria.

[0023] According to a preferred embodiment of the present invention, if the first emotion recognition result still does not meet the first recall rate threshold and the first accuracy rate threshold, the labeling standard is adjusted and S201 to S204 are re-executed.

[0024] According to a preferred embodiment of the present invention, the emotion recognition model is iterated through self-learning after being put online, further comprising:

[0025] S301, collecting online communication data between customer service and users after the emotion recognition model is launched;

[0026] S302, using the emotion recognition model to perform preliminary emotion recognition on the online communication data;

[0027] S303: Training the emotion recognition model based on the preliminary emotion recognition result.

[0028] According to a preferred embodiment of the present invention, training the emotion recognition model based on the preliminary emotion recognition result further comprises:

[0029] Set the positive threshold and hard negative threshold;

[0030] The confidence data in the preliminary emotion recognition results that meets the positive example threshold are added to the training data as training positive examples;

[0031] The confidence data in the preliminary emotion recognition results that meet the hard negative example threshold are added to the training data center as hard negative examples;

[0032] The emotion recognition model is trained using training data with supplementary training positive examples and hard negative examples.

[0033] According to a preferred embodiment of the present invention, a second accuracy threshold and a second recall threshold are preset to determine whether the emotion recognition result satisfies both the second accuracy threshold and the second recall threshold. If yes, the current emotion recognition model is retained; otherwise, S301 to S303 are re-executed.

[0034] According to a preferred embodiment of the present invention, a threshold of the number of iterations is preset. If the preliminary emotion recognition result still does not meet the second accuracy threshold or the second recall threshold after the threshold of the number of iterations is reached, the labeling standard is adjusted and steps S2 to S3 are re-executed.

[0035] A second aspect of the present invention provides an emotion recognition device based on customer service communication data, comprising:

[0036] A preliminary emotion recognition module is used to collect communication data between customer service and users, and perform preliminary emotion recognition on the communication data;

[0037] A model building module, used to use an emotion recognition model to perform a first emotion recognition on the communication data based on the preliminary emotion recognition result, and determine whether the first emotion recognition result meets the standard. If so, the emotion recognition model is saved and put online;

[0038] The model iteration module iterates the emotion recognition model through self-learning after it is launched, and uses the iterated emotion recognition model to perform a second emotion recognition.

[0039] According to a preferred embodiment of the present invention, performing preliminary emotion recognition on the communication data further comprises:

[0040] Purifying the communication data using sample-free learning;

[0041] The purified communication data is initially screened manually.

[0042] According to a preferred embodiment of the present invention, performing first emotion recognition on the communication data based on the preliminary emotion recognition result using the emotion recognition model further comprises:

[0043] The preliminary emotion recognition results are manually annotated;

[0044] Create training data using the annotated preliminary emotion recognition results;

[0045] Using the training data to train the emotion recognition model;

[0046] The trained emotion recognition model is used to perform a first emotion recognition on the communication data.

[0047] According to a preferred embodiment of the present invention, determining whether the first emotion recognition result meets the standard further includes:

[0048] A first recall rate threshold and a first accuracy rate threshold are preset to determine whether the first emotion recognition result satisfies both the first recall rate threshold and the first accuracy rate threshold. If so, the first emotion recognition result meets the criteria.

[0049] According to a preferred embodiment of the present invention, if the first emotion recognition result still does not meet the first recall rate threshold and the first accuracy rate threshold, the labeling standard is adjusted to re-execute the first emotion recognition on the communication data based on the preliminary emotion recognition result using the emotion recognition model.

[0050] According to a preferred embodiment of the present invention, the emotion recognition model is iterated through self-learning after being put online, further comprising:

[0051] Collect online communication data between customer service and users after the emotion recognition model is launched;

[0052] Using the emotion recognition model to perform preliminary emotion recognition on the online communication data;

[0053] The emotion recognition model is trained based on the preliminary emotion recognition result.

[0054] According to a preferred embodiment of the present invention, training the emotion recognition model based on the preliminary emotion recognition result further comprises:

[0055] Set the positive threshold and hard negative threshold;

[0056] The confidence data in the preliminary emotion recognition results that meets the positive example threshold are added to the training data as training positive examples;

[0057] The confidence data in the preliminary emotion recognition results that meet the hard negative example threshold are added to the training data center as hard negative examples;

[0058] The emotion recognition model is trained using training data with supplementary training positive examples and hard negative examples.

[0059] According to a preferred embodiment of the present invention, a second accuracy threshold and a second recall threshold are preset to determine whether the emotion recognition result satisfies both the second accuracy threshold and the second recall threshold. If so, the current emotion recognition model is retained; otherwise, the emotion recognition model is retrained based on the preliminary emotion recognition result.

[0060] According to a preferred embodiment of the present invention, a threshold number of iterations is preset. If the preliminary emotion recognition result still does not meet the second accuracy threshold or does not meet the second recall threshold after the threshold number of iterations is reached, the labeling standard is adjusted to re-execute the first emotion recognition on the communication data based on the preliminary emotion recognition result using the emotion recognition model.

[0061] A third aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory is used to store a computer executable program, and when the computer program is executed by the processor, the processor executes the described method.

[0062] A fourth aspect of the present invention further proposes a computer-readable medium storing a computer-executable program, wherein when the computer-executable program is executed, the method described is implemented.

[0063] (III) Beneficial effects

[0064] The present invention completes the construction of the emotion recognition model through preliminary emotion recognition and the first emotion recognition. After going online, a large amount of online communication data is used to iterate and upgrade the emotion recognition model through self-learning, so that extreme emotions can be recognized more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 It is a flowchart of an emotion recognition method based on customer service communication data according to an embodiment of the present invention;

[0066] Figure 2 It is a schematic diagram of the structure of an emotion recognition device based on customer service communication data according to an embodiment of the present invention;

[0067] Figure 3 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

[0068] Figure 4 is a schematic diagram of a computer-readable recording medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.

[0070] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to the actual situation without departing from the main purpose of the present invention.

[0071] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processing unit devices and / or microcontroller devices.

[0072] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0073] In order to solve the above technical problems, the present invention proposes an emotion recognition method based on customer service communication data. The method flow chart is as follows: Figure 1 As shown, including:

[0074] S1. Collect communication data between customer service and users, and perform preliminary emotion recognition on the communication data.

[0075] In this embodiment, the content to be identified is extreme emotions. Usually, the user's emotions develop over a process during a conversation. For example, anger usually transitions from dissatisfaction to anger, and finally to anger. In voice robot customer service, the user's emotions are usually identified. When the user's emotions are identified as anger, the preset dialogue strategy is used to appease and regulate the user's emotions. However, there are very few cases where the user directly enters an angry state. At this time, if the general dialogue strategy for anger is still used to appease, it is likely that the user's emotions will not be appeased.

[0076] Since extreme emotions appear very rarely in the conversation data between customer service and users, it is difficult to capture them as training data to train the model.

[0077] Therefore, in the present embodiment, the communication data between the customer service and the user are first collected and summarized, and then the communication data between the customer service and the user are preliminarily screened, that is, preliminary emotion recognition is performed.

[0078] Further on the basis of the above technical solution, performing preliminary emotion recognition on the communication data further includes:

[0079] S101, purifying the communication data using sample-free learning;

[0080] S102, manually performing preliminary screening on the purified communication data.

[0081] Sample-free learning is an algorithm that uses a model trained on natural language inference tasks and known labels to infer unlabeled data. In this implementation, a trained model is used to set the label content, construct a sentence with blanks, fill the label content into the blanks of the constructed sentence, and judge the confidence of the sentence filled with labels and the unlabeled data to make inferences.

[0082] The following example illustrates sample-free learning. The sample-free learning model has been completed and trained.

[0083] Unlabeled data: As a parent, I have learned a lot from this, thank you.

[0084] Labels: A: User dissatisfaction; B: User satisfaction.

[0085] Construct a sentence: This sentence is [__].

[0086] Fill labels A and B into the constructed sentences respectively to determine the similarity between the unlabeled data and the two sentences. The output result of the sample-free learning model is: user A is dissatisfied with a confidence level of 6.2%; user B is satisfied with a confidence level of 93.8%.

[0087] Therefore, after the judgment of the sample-free learning model, the sentence is labeled with label B. Through sample-free learning, a lot of useless data can be screened out, greatly increasing the data concentration of extreme emotional information in communication data.

[0088] Then, we use manual screening to preliminarily screen the purified communication data and select sentences or information that express extreme emotions.

[0089] Although manual indexing is accurate, it is relatively inefficient and cannot be used to index all communication data. Generally, the distribution density of information expressing extreme emotions in 10,000 communication data is very close to the actual concentration of extreme emotions in the communication data. Therefore, in this embodiment, manual indexing is used to index 10,000 communication data.

[0090] S2. Use the emotion recognition model to perform a first emotion recognition on the communication data based on the preliminary emotion recognition result, and determine whether the first emotion recognition result meets the standard. If so, save the emotion recognition model and put it online.

[0091] In this embodiment, the results of preliminary emotion recognition are used as training data, and the emotion recognition model is preliminarily trained using the training data to determine whether the emotion recognition results meet the standards. If they meet the standards, it means that the emotion recognition model meets the basic requirements and can be put online and added to the existing voice robot.

[0092] Further on the basis of the above technical solution, using the emotion recognition model to perform the first emotion recognition on the communication data based on the preliminary emotion recognition result further comprises:

[0093] S201, manually labeling the preliminary emotion recognition results.

[0094] In this embodiment, the communication data is purified through sample-free learning, and relatively accurate information expressing extreme emotions is initially screened out through manual labeling. Then, this information is specifically labeled through manual annotation to give accurate labels.

[0095] S202: Create training data using the annotated preliminary emotion recognition results.

[0096] In this embodiment, the emotion recognition model is a bidirectional long-short term memory network model. The bidirectional long-short term memory network model is a type of recurrent neural network, which can better consider the words before and after the sentence, such as "I don't think the weather is good today", where "no" is a limitation on the following "good weather", indicating the negation of good weather. The use of the bidirectional long-short term memory network model can better capture the dependency relationship over a longer distance, and the bidirectional long-short term memory network model can also take into account the limitation of the following words on the previous words, such as "Today's weather is too cold", where "not" is a modification and limitation of "cold".

[0097] Then the manually annotated data in S201 is divided into two parts, one of which is used as a training set and the other is used as a verification set.

[0098] S203: Use the training data to train the emotion recognition model.

[0099] In this embodiment, the training set is first used to train the bidirectional long-short term memory network model until the bidirectional long-short term memory network model converges or has achieved the expected effect. Then, the verification set is used to verify the bidirectional long-short term memory network model. If it passes the verification, the training of the bidirectional long-short term memory network model is completed.

[0100] S204: Use the trained emotion recognition model to perform a first emotion recognition on the communication data.

[0101] In this embodiment, after the emotion recognition model training is completed, the first emotion recognition is performed on the unlabeled communication data to give the confidence of the label. For example, "I am a little busy now, the time is not suitable, please record it", after the emotion recognition model calculation, the confidence of the user dissatisfaction of label A is 53.32%, and the confidence of the user satisfaction of label B is 46.68%; and "The price of this course is really favorable, I will sign up for it" after the emotion recognition model calculation, the confidence of the user dissatisfaction of label A is 11.04%, and the confidence of the user satisfaction of label B is 88.96%.

[0102] Further on the basis of the above technical solution, judging whether the first emotion recognition result meets the standard further includes:

[0103] A first recall rate threshold and a first accuracy rate threshold are preset to determine whether the first emotion recognition result satisfies both the first recall rate threshold and the first accuracy rate threshold. If so, the first emotion recognition result meets the criteria.

[0104] In this embodiment, the criteria for evaluating whether the first emotion recognition result meets the standard are the recall rate and the precision rate.

[0105] TP (Ture Positive) judges the positive as a positive number, the judgment is correct, and it is judged to be positive, that is, the positive prediction is positive;

[0106] FN (False Negative) is the number of positive numbers that are mistakenly judged as negative.

[0107] FP (False Positive) is the number of times a negative number is mistakenly judged as positive.

[0108] TN (True Negative) judges the negative as a negative number, the judgment is correct, and it is judged as negative, that is, the negative is judged as negative.

[0109] The recall rate is relative to the sample, that is, how many positive samples in the sample are predicted correctly. There are TP of them. All positive samples have two destinations, one is judged as positive, and the other is wrongly judged as negative. Therefore, there are TP+FN in total. Therefore, the recall rate is

[0110] The accuracy is relative to the prediction result. It indicates how many of the samples predicted as positive are correct. Then there are two possible sources of samples predicted as positive. One is to predict positive as positive, which has TP, and the other is to misjudge negative as positive, which has FP. Therefore, the accuracy is

[0111] Recall rate and precision rate are the most basic indicators to measure the performance of emotion recognition models. Since it is impossible for both indicators to reach 100% at the same time, the specific values ​​of the two indicators will be adjusted according to different tasks.

[0112] Further on the basis of the above technical solution, if the first emotion recognition result still does not meet the first recall rate threshold and the first accuracy rate threshold, the labeling standard is adjusted and S201 to S204 are re-executed.

[0113] In this embodiment, if the result of the first emotion recognition does not meet the standard, there may be some problems with the labeling standard that need to be adjusted. After the adjustment, the emotion recognition model is used again to perform the first emotion recognition on the communication data based on the preliminary emotion recognition result. If it meets the standard, the parameters of the current emotion recognition model can be saved and applied online. Otherwise, the labeling standard continues to be adjusted, and steps S201-S204 are repeated until the first recall rate threshold and the first accuracy threshold are met.

[0114] S3. After the emotion recognition model is put online, it is iterated through self-learning, and the iterated emotion recognition model is used to perform a second emotion recognition.

[0115] In this embodiment, due to the convenience and ease of online communication, users can communicate with customer service personnel at the customer service center anytime and anywhere. The amount of online communication data is much larger than the previously accumulated communication data, and the scenarios encountered will also be diverse. The model may not be able to meet the online communication data. Therefore, after the emotion recognition model is launched, it is iteratively upgraded through self-study.

[0116] Further on the basis of the above technical solution, the emotion recognition model is iterated through self-learning after it is online, further comprising:

[0117] S301, collecting online communication data between customer service and users after the emotion recognition model is launched.

[0118] S302, using the emotion recognition model to perform preliminary emotion recognition on the online communication data;

[0119] S303: Training the emotion recognition model based on the preliminary emotion recognition result.

[0120] Further on the basis of the above technical solution, training the emotion recognition model based on the preliminary emotion recognition result further includes:

[0121] Set the positive threshold and hard negative threshold;

[0122] The confidence data in the preliminary emotion recognition results that meets the positive example threshold are added to the training data as training positive examples;

[0123] The confidence data in the preliminary emotion recognition results that meet the hard negative example threshold are added to the training data center as hard negative examples;

[0124] The emotion recognition model is trained using training data with supplementary training positive examples and hard negative examples.

[0125] In this embodiment, the positive example threshold is usually set in the range of 80%-95%, which means that sentences or information expressing extreme emotions can be accurately identified.

[0126] Hard negatives refer to data that is relatively difficult for the emotion recognition model to distinguish. For example, among sentences expressing gratitude, some sentences express polite gratitude, some express general gratitude, and some express sincere gratitude. Among them, sentences expressing sincere gratitude are the targets of extreme emotion recognition, that is, they are positive examples, while sentences expressing polite gratitude and general gratitude are also gratitude, but they are not extreme emotions. Such models are relatively difficult to distinguish, so they are hard negatives. The hard negative threshold is usually an interval value of the confidence level, with the upper limit usually set at 40%-70% and the lower limit set at 20%-40%. Different hard negative intervals are set for different tasks.

[0127] The following examples illustrate positive examples and hard negative examples.

[0128] Labels include A: user is dissatisfied, B: user is satisfied.

[0129] The positive threshold is set to a B label confidence level greater than 90%, and the hard negative threshold is set to a B label confidence level between 30% and 55%.

[0130] "As a parent, I have learned a lot from this. Thank you." After calculation by the emotion recognition model, the confidence of label B is 93.8%, which meets the positive example threshold and is a positive example. This sentence expresses extreme emotions and is added to the training data.

[0131] "That's all for today, thank you." After calculation by the emotion recognition model, the confidence of label B is 53.32%, which meets the hard negative threshold and is a hard negative example. Although this sentence expresses gratitude, it is a polite thank you and does not reach the level of extreme emotion. It is a negative example in the training data.

[0132] Use the current emotion recognition model as the parent model, obtain the second emotion recognition result by performing the second emotion recognition on the online communication data, select positive examples and hard negative examples as training models to train the parent model, and adjust the model parameters to obtain the child model. Then use the obtained child model as the second parent model to perform the second emotion recognition on the online communication data, and repeat the above steps to iteratively upgrade the emotion recognition model. The online communication data is constantly accumulated, and the next online communication data includes the previous online communication data and the newly collected communication data.

[0133] On the basis of the above technical solution, a second accuracy threshold and a second recall threshold are further preset to determine whether the emotion recognition result satisfies the second accuracy threshold and the second recall threshold at the same time. If yes, the current emotion recognition model is retained, otherwise S301 to S303 are re-executed.

[0134] In this embodiment, the same indicators for evaluating the second emotion recognition result are still the recall rate and the precision rate.

[0135] Further on the basis of the above technical solution, a threshold of the number of iterations is preset. If the preliminary emotion recognition result still does not meet the second accuracy threshold or the second recall threshold after the threshold of the number of iterations is reached, the labeling standard is adjusted and S201 is re-executed.

[0136] In this embodiment, if after multiple iterations, the index of the second emotion recognition result still does not meet the set second accuracy threshold and second recall threshold after reaching the preset number of iterations, it is likely that the distribution of extreme emotions in the first 10,000 communication data marked is significantly different from the distribution of extreme emotions in the actual communication data, so S2 to S3 are re-executed to re-mark the 10,000 communication data so that the distribution of extreme emotions is closer to the actual situation.

[0137] Figure 2 A data processing device 200 for mining user needs in an embodiment of the present invention includes:

[0138] The preliminary emotion recognition module 201 is used to collect communication data between customer service and users, and perform preliminary emotion recognition on the communication data.

[0139] In this embodiment, the content to be identified is extreme emotions. Usually, the user's emotions develop over a process during a conversation. For example, anger usually transitions from dissatisfaction to anger, and finally to anger. In voice robot customer service, the user's emotions are usually identified. When the user's emotions are identified as anger, the preset dialogue strategy is used to appease and regulate the user's emotions. However, there are very few cases where the user directly enters an angry state. At this time, if the general dialogue strategy for anger is still used to appease, it is likely that the user's emotions will not be appeased.

[0140] Since extreme emotions appear very rarely in the conversation data between customer service and users, it is difficult to capture them as training data to train the model.

[0141] Therefore, in the present embodiment, the communication data between the customer service and the user are first collected and summarized, and then the communication data between the customer service and the user are preliminarily screened, that is, preliminary emotion recognition is performed.

[0142] Further on the basis of the above technical solution, performing preliminary emotion recognition on the communication data further includes:

[0143] S101, purifying the communication data using sample-free learning;

[0144] S102, manually performing preliminary screening on the purified communication data.

[0145] Sample-free learning is an algorithm that uses a model trained on natural language inference tasks and known labels to infer unlabeled data. In this implementation, a trained model is used to set the label content, construct a sentence with blanks, fill the label content into the blanks of the constructed sentence, and judge the confidence of the sentence filled with labels and the unlabeled data to make inferences.

[0146] The following example illustrates sample-free learning. The sample-free learning model has been completed and trained.

[0147] Unlabeled data: As a parent, I have learned a lot from this, thank you.

[0148] Labels: A: User dissatisfaction; B: User satisfaction.

[0149] Construct a sentence: This sentence is [__].

[0150] Fill labels A and B into the constructed sentences respectively to determine the similarity between the unlabeled data and the two sentences. The output result of the sample-free learning model is: user A is dissatisfied with a confidence level of 6.2%; user B is satisfied with a confidence level of 93.8%.

[0151] Therefore, after the judgment of the sample-free learning model, the sentence is labeled with label B. Through sample-free learning, a lot of useless data can be screened out, greatly increasing the data concentration of extreme emotional information in communication data.

[0152] Then, we use manual screening to preliminarily screen the purified communication data and select sentences or information that express extreme emotions.

[0153] Although manual indexing is accurate, it is relatively inefficient and cannot be used to index all communication data. Generally, the distribution density of information expressing extreme emotions in 10,000 communication data is very close to the actual concentration of extreme emotions in the communication data. Therefore, in this embodiment, manual indexing is used to index 10,000 communication data.

[0154] The model building module 202 is used to use the emotion recognition model to perform the first emotion recognition on the communication data based on the preliminary emotion recognition result, and determine whether the first emotion recognition result meets the standard. If so, the emotion recognition model is saved and put online.

[0155] In this embodiment, the results of preliminary emotion recognition are used as training data, and the emotion recognition model is preliminarily trained using the training data to determine whether the emotion recognition results meet the standards. If they meet the standards, it means that the emotion recognition model meets the basic requirements and can be put online and added to the existing voice robot.

[0156] Further on the basis of the above technical solution, using the emotion recognition model to perform the first emotion recognition on the communication data based on the preliminary emotion recognition result further comprises:

[0157] S201, manually labeling the preliminary emotion recognition results.

[0158] In this embodiment, the communication data is purified through sample-free learning, and relatively accurate information expressing extreme emotions is initially screened out through manual labeling. Then, this information is specifically labeled through manual annotation to give accurate labels.

[0159] S202: Create training data using the annotated preliminary emotion recognition results.

[0160] In this embodiment, the emotion recognition model is a bidirectional long-short term memory network model. The bidirectional long-short term memory network model is a type of recurrent neural network, which can better consider the words before and after the sentence, such as "I don't think the weather is good today", where "no" is a limitation on the following "good weather", indicating the negation of good weather. The use of the bidirectional long-short term memory network model can better capture the dependency relationship over a longer distance, and the bidirectional long-short term memory network model can also take into account the limitation of the following words on the previous words, such as "Today's weather is too cold", where "not" is a modification and limitation of "cold".

[0161] Then the manually annotated data in S201 is divided into two parts, one of which is used as a training set and the other is used as a verification set.

[0162] S203: Use the training data to train the emotion recognition model.

[0163] In this embodiment, the training set is first used to train the bidirectional long-short term memory network model until the bidirectional long-short term memory network model converges or has achieved the expected effect. Then, the verification set is used to verify the bidirectional long-short term memory network model. If it passes the verification, the training of the bidirectional long-short term memory network model is completed.

[0164] S204: Use the trained emotion recognition model to perform a first emotion recognition on the communication data.

[0165] In this embodiment, after the emotion recognition model training is completed, the first emotion recognition is performed on the unlabeled communication data to give the confidence of the label. For example, "I am a little busy now, the time is not suitable, please record it", after the emotion recognition model calculation, the confidence of the user dissatisfaction of label A is 53.32%, and the confidence of the user satisfaction of label B is 46.68%; and "The price of this course is really favorable, I will sign up for it" after the emotion recognition model calculation, the confidence of the user dissatisfaction of label A is 11.04%, and the confidence of the user satisfaction of label B is 88.96%.

[0166] Further on the basis of the above technical solution, judging whether the first emotion recognition result meets the standard further includes:

[0167] A first recall rate threshold and a first accuracy rate threshold are preset to determine whether the first emotion recognition result satisfies both the first recall rate threshold and the first accuracy rate threshold. If so, the first emotion recognition result meets the criteria.

[0168] In this embodiment, the criteria for evaluating whether the first emotion recognition result meets the standard are the recall rate and the precision rate.

[0169] TP (Ture Positive) judges the positive as a positive number, the judgment is correct, and it is judged to be positive, that is, the positive prediction is positive;

[0170] FN (False Negative) is the number of positive numbers that are mistakenly judged as negative.

[0171] FP (False Positive) is the number of times a negative number is mistakenly judged as positive.

[0172] TN (True Negative) judges the negative as a negative number, the judgment is correct, and it is judged as negative, that is, the negative is judged as negative.

[0173] The recall rate is relative to the sample, that is, how many positive samples in the sample are predicted correctly. There are TP of them. All positive samples have two destinations, one is judged as positive, and the other is wrongly judged as negative. Therefore, there are TP+FN in total. Therefore, the recall rate is

[0174] The accuracy is relative to the prediction result. It indicates how many of the samples predicted as positive are correct. Then there are two possible sources of samples predicted as positive. One is to predict positive as positive, which has TP, and the other is to misjudge negative as positive, which has FP. Therefore, the accuracy is

[0175] Recall rate and precision rate are the most basic indicators to measure the performance of emotion recognition models. Since it is impossible for both indicators to reach 100% at the same time, the specific values ​​of the two indicators will be adjusted according to different tasks.

[0176] Further on the basis of the above technical solution, if the first emotion recognition result still does not meet the first recall rate threshold and the first accuracy rate threshold, the labeling standard is adjusted and S201 to S204 are re-executed.

[0177] In this embodiment, if the result of the first emotion recognition does not meet the standard, there may be some problems with the labeling standard that need to be adjusted. After the adjustment, the emotion recognition model is used again to perform the first emotion recognition on the communication data based on the preliminary emotion recognition result. If it meets the standard, the parameters of the current emotion recognition model can be saved and applied online. Otherwise, the labeling standard continues to be adjusted, and steps S201-S204 are repeated until the first recall rate threshold and the first accuracy threshold are met.

[0178] The model iteration module 203 iterates the emotion recognition model through self-learning after it is put online, and uses the iterated emotion recognition model to perform second emotion recognition.

[0179] In this embodiment, due to the convenience and ease of online communication, users can communicate with customer service personnel at the customer service center anytime and anywhere. The amount of online communication data is much larger than the previously accumulated communication data, and the scenarios encountered will also be diverse. The model may not be able to meet the online communication data. Therefore, after the emotion recognition model is launched, it is iteratively upgraded through self-study.

[0180] Further on the basis of the above technical solution, the emotion recognition model is iterated through self-learning after it is online, further comprising:

[0181] S301, collecting online communication data between customer service and users after the emotion recognition model is launched.

[0182] S302, using the emotion recognition model to perform preliminary emotion recognition on the online communication data;

[0183] S303: Training the emotion recognition model based on the preliminary emotion recognition result.

[0184] Further on the basis of the above technical solution, training the emotion recognition model based on the preliminary emotion recognition result further includes:

[0185] Set the positive threshold and hard negative threshold;

[0186] The confidence data in the preliminary emotion recognition results that meets the positive example threshold are added to the training data as training positive examples;

[0187] The confidence data in the preliminary emotion recognition results that meet the hard negative example threshold are added to the training data center as hard negative examples;

[0188] The emotion recognition model is trained using training data with supplementary training positive examples and hard negative examples.

[0189] In this embodiment, the positive example threshold is usually set in the range of 80%-95%, which means that sentences or information expressing extreme emotions can be accurately identified.

[0190] Hard negatives refer to data that is relatively difficult for the emotion recognition model to distinguish. For example, among sentences expressing gratitude, some sentences express polite gratitude, some express general gratitude, and some express sincere gratitude. Among them, sentences expressing sincere gratitude are the targets of extreme emotion recognition, that is, they are positive examples, while sentences expressing polite gratitude and general gratitude are also gratitude, but they are not extreme emotions. Such models are relatively difficult to distinguish, so they are hard negatives. The hard negative threshold is usually an interval value of the confidence level, with the upper limit usually set at 40%-70% and the lower limit set at 20%-40%. Different hard negative intervals are set for different tasks.

[0191] The following examples illustrate positive examples and hard negative examples.

[0192] Labels include A: user is dissatisfied, B: user is satisfied.

[0193] The positive threshold is set to the confidence level of label A greater than 90%, and the hard negative threshold is set to the confidence level of label A between 30% and 55%.

[0194] "As a parent, I have learned a lot from this. Thank you." After calculation by the emotion recognition model, the confidence of label B is 93.8%, which meets the positive example threshold and is a positive example. This sentence expresses extreme emotions and is added to the training data.

[0195] "That's all for today, thank you." After calculation by the emotion recognition model, the confidence of label B is 53.32%, which meets the hard negative threshold and is a hard negative example. Although this sentence expresses gratitude, it is a polite thank you and does not reach the level of extreme emotion. It is a negative example in the training data.

[0196] Use the current emotion recognition model as the parent model, obtain the second emotion recognition result by performing the second emotion recognition on the online communication data, select positive examples and hard negative examples as training models to train the parent model, and adjust the model parameters to obtain the child model. Then use the obtained child model as the second parent model to perform the second emotion recognition on the online communication data, and repeat the above steps to iteratively upgrade the emotion recognition model. The online communication data is constantly accumulated, and the next online communication data includes the previous online communication data and the newly collected communication data.

[0197] On the basis of the above technical solution, a second accuracy threshold and a second recall threshold are further preset to determine whether the emotion recognition result satisfies the second accuracy threshold and the second recall threshold at the same time. If yes, the current emotion recognition model is retained, otherwise S301 to S303 are re-executed.

[0198] In this embodiment, the same indicators for evaluating the second emotion recognition result are still the recall rate and the precision rate.

[0199] Further on the basis of the above technical solution, a threshold of the number of iterations is preset. If the preliminary emotion recognition result still does not meet the second accuracy threshold or the second recall threshold after the threshold of the number of iterations is reached, the labeling standard is adjusted and S201 is re-executed.

[0200] In this embodiment, if after multiple iterations, the index of the second emotion recognition result still does not meet the set second accuracy threshold and second recall threshold after reaching the preset number of iterations, it is likely that the distribution of extreme emotions in the first 10,000 communication data marked is significantly different from the distribution of extreme emotions in the actual communication data, so S2 to S3 are re-executed to re-mark the 10,000 communication data so that the distribution of extreme emotions is closer to the actual situation.

[0201] Figure 3 It is a structural schematic diagram of an electronic device of an embodiment of the present invention, the electronic device includes a processor and a memory, the memory is used to store a computer executable program, when the computer program is executed by the processor, the processor executes a vehicle intelligent power-assisted propulsion method based on rotation angle monitoring.

[0202] like Figure 3 As shown, the electronic device is presented in the form of a general computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0203] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least part of the steps in the method.

[0204] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0205] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with an external device. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0206] It should be understood that Figure 3The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0207] Figure 4 Schematic diagram of a computer readable recording medium according to an embodiment of the present invention. Figure 4 As shown, a computer executable program is stored in a computer-readable recording medium, and when the computer executable program is executed, the vehicle intelligent power-assisted pushing method based on rotation angle monitoring of the present invention is implemented. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate or transmit a program for use by or in combination with an instruction execution system, device or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0208] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0209] Through the above description of the implementation mode, it is easy for those skilled in the art to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by a vehicle comprising at least a part of the above system or components. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. on the locomotive side. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware, such as some method steps executed by the computer program can be executed on the locomotive side, and another part can be executed in a mobile terminal or a smart helmet, etc. For computer software, the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), or it can be distributed and stored on the network, as long as it enables the electronic device to execute the method according to the present invention.

[0210] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the present invention is not inherently related to any specific computer, virtual device or electronic device, and various general devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for identifying extreme emotions based on customer service communication data, It is characterized in that Methods include: S1. Collect communication data between customer service and users, purify the communication data using a sample-free learning model, and manually screen the purified communication data to perform preliminary emotion recognition on the communication data, where the emotion is an extreme emotion; S2, using the trained emotion recognition model to perform a first emotion recognition on the communication data based on the preliminary emotion recognition result, and judging whether the first emotion recognition result meets the standard, wherein the standard for meeting the standard is the recall rate and the accuracy rate; if the standard is met, the emotion recognition model is saved and put online; S3. After the emotion recognition model is put online, it is iterated through self-learning, and the iterated emotion recognition model is used to perform the second emotion recognition; when iterating through self-learning, the current emotion recognition model is used as the parent model, and the second emotion recognition result is obtained by performing the second emotion recognition on the online communication data, and positive examples and hard negative examples are selected from the parent model to train the parent model and adjust the model parameters to obtain the child model, and the child model is used as the second parent model to perform the second emotion recognition on the online communication data and repeat the above process to iteratively upgrade the emotion recognition model, wherein the latter online communication data includes the previous online communication data and the newly collected communication data; and, a preset iteration number threshold is set. If the second emotion recognition result does not meet the preset second accuracy threshold or does not meet the preset second recall threshold after reaching the iteration number threshold, the labeling standard is adjusted and S2 to S3 are re-executed to make the distribution of extreme emotions approach the actual situation.

2. The method according to claim 1, It is characterized in that Using the trained emotion recognition model to perform first emotion recognition on the communication data based on the preliminary emotion recognition result further includes: S201, manually labeling the preliminary emotion recognition results; S202, creating training data using the annotated preliminary emotion recognition results, using a portion of the training data as a training set and another portion of the training data as a verification set; S203, training the emotion recognition model using the training data; S204: Use the trained emotion recognition model to perform a first emotion recognition on the communication data.

3. The method according to claim 2, It is characterized in that Determining whether the first emotion recognition result meets the standard further includes: A first recall rate threshold and a first accuracy rate threshold are preset to determine whether the first emotion recognition result satisfies both the first recall rate threshold and the first accuracy rate threshold. If so, the first emotion recognition result meets the criteria.

4. The method according to claim 3, It is characterized in that If the first emotion recognition result still does not meet the first recall rate threshold and the first accuracy rate threshold, the labeling standard is adjusted and S201 to S204 are re-executed until the first recall rate threshold and the first accuracy rate threshold are met.

5. The method according to claim 1, It is characterized in that Determining whether the first emotion recognition result meets the standard further includes: A first recall rate threshold and a first accuracy rate threshold are preset to determine whether the first emotion recognition result satisfies both the first recall rate threshold and the first accuracy rate threshold. If so, the first emotion recognition result meets the criteria.

6. The method according to any one of claims 1 to 5, It is characterized in that After the emotion recognition model is launched, iteration through self-learning further includes: S301, collecting online communication data between customer service and users after the emotion recognition model is launched; S302, using the emotion recognition model to perform preliminary emotion recognition on the online communication data; S303: Training the emotion recognition model based on the preliminary emotion recognition result.

7. The method according to claim 6, It is characterized in that Training the emotion recognition model based on the preliminary emotion recognition result further includes: Set the positive threshold and hard negative threshold; The confidence data in the preliminary emotion recognition results that meets the positive example threshold are added to the training data as training positive examples; The confidence data in the preliminary emotion recognition results that meet the hard negative example threshold are added to the training data center as hard negative examples; The emotion recognition model is trained using training data with supplementary training positive examples and hard negative examples.

8. The method according to claim 7, It is characterized in that Also includes: It is determined whether the second emotion recognition result satisfies both the second accuracy threshold and the second recall threshold. If yes, the current emotion recognition model is retained; otherwise, S301 to S303 are re-executed.

9. An extreme emotion recognition device based on customer service communication data, It is characterized in that The device includes: A preliminary emotion recognition module is used to collect communication data between customer service and users, purify the communication data using a sample-free learning model, and manually screen the purified communication data to perform preliminary emotion recognition on the communication data, where the emotion is an extreme emotion; A model building module, used to use the trained emotion recognition model to perform first emotion recognition on the communication data based on the preliminary emotion recognition result, and determine whether the first emotion recognition result meets the standard, wherein the standard for meeting the standard is the recall rate and the accuracy rate; if the standard is met, the emotion recognition model is saved and put online; A model iteration module, wherein the emotion recognition model is iterated through self-learning after being put online, and the iterated emotion recognition model is used to perform a second emotion recognition; when iterating through self-learning, the current emotion recognition model is used as the parent model, and a second emotion recognition result is obtained by performing a second emotion recognition on the online communication data, and positive examples and hard negative examples are selected from the parent model to train the parent model and adjust the model parameters to obtain a child model, and the child model is used as the second parent model to perform a second emotion recognition on the online communication data and the aforementioned process is repeated to iterate and upgrade the emotion recognition model, wherein the latter online communication data includes the previous online communication data and the newly collected communication data; and, a preset iteration number threshold is set, and if the second emotion recognition result does not meet the preset second accuracy threshold or the preset second recall threshold after reaching the iteration number threshold, the labeling standard is adjusted and S2 to S3 are re-executed to make the distribution of extreme emotions approach the actual situation.

10. The device according to claim 9, It is characterized in that The model building module further includes: using the trained emotion recognition model to perform first emotion recognition on the communication data based on the preliminary emotion recognition result: The preliminary emotion recognition results are manually annotated; Use the annotated preliminary emotion recognition results to create training data, with part of the training data used as a training set and the other part of the training data used as a validation set; Using the training data to train the emotion recognition model; The trained emotion recognition model is used to perform a first emotion recognition on the communication data.

11. The device according to claim 10, It is characterized in that In the model building module, judging whether the first emotion recognition result meets the standard further includes: A first recall rate threshold and a first accuracy rate threshold are preset to determine whether the first emotion recognition result satisfies both the first recall rate threshold and the first accuracy rate threshold. If so, the first emotion recognition result meets the criteria.

12. The device according to claim 11, It is characterized in that If the first emotion recognition result still does not meet the first recall rate threshold and the first accuracy rate threshold, the labeling standard is adjusted to re-execute the processing further including performing the first emotion recognition on the communication data based on the preliminary emotion recognition result using the emotion recognition model until it meets the first recall rate threshold and the first accuracy threshold.

13. The device according to claim 9, It is characterized in that In the model building module, judging whether the first emotion recognition result meets the standard further includes: A first recall rate threshold and a first accuracy rate threshold are preset to determine whether the first emotion recognition result satisfies both the first recall rate threshold and the first accuracy rate threshold. If so, the first emotion recognition result meets the criteria.

14. The device according to any one of claims 9 to 13, It is characterized in that After the emotion recognition model in the model iteration module is launched, iterating through self-learning further includes: Collect online communication data between customer service and users after the emotion recognition model is launched; Using the emotion recognition model to perform preliminary emotion recognition on the online communication data; The emotion recognition model is trained based on the preliminary emotion recognition result.

15. The device according to claim 14, It is characterized in that Training the emotion recognition model based on the preliminary emotion recognition result further includes: Set the positive threshold and hard negative threshold; The confidence data in the preliminary emotion recognition results that meets the positive example threshold are added to the training data as training positive examples; The confidence data in the preliminary emotion recognition results that meet the hard negative example threshold are added to the training data center as hard negative examples; The emotion recognition model is trained using training data with supplementary training positive examples and hard negative examples.

16. The device according to claim 15, It is characterized in that Also includes: It is determined whether the emotion recognition result satisfies the second accuracy threshold and the second recall threshold at the same time. If yes, the current emotion recognition model is retained. Otherwise, the processing further including iterating through self-learning after the emotion recognition model is put online is re-executed.

17. An electronic device comprising a processor and a memory, wherein the memory is used to store a computer executable program, Features: When the computer executable program is executed by the processor, the processor performs the method according to any one of claims 1 to 8.

18. A computer readable medium storing a computer executable program, It is characterized in that When the computer executable program is executed, the method according to any one of claims 1 to 8 is implemented.

Images