Customer service response evaluation method and system based on large language model

Abstract

The present invention discloses a customer service response evaluation method and system based on a large language model, and relates to the technical field of customer service response. The method and system obtain customer service dialogue data, assign sentiment scores through a sentiment dictionary and a sentiment tag library, and obtain customer service sentiment text; obtain several groups of different sentiment threshold combinations, and stratify the customer service sentiment text to obtain the customer's emotion level; obtain sentiment matching data and customer sentiment change data; construct a sentiment threshold evaluation model based on machine learning; construct an effect display model, and perform data analysis on the effect display model to obtain an optimized sentiment threshold combination, and apply it to customer service response evaluation. The present application solves the problem that the threshold of sentiment analysis is set based on experience, which leads to inaccurate analysis results and unstable service effects, and ultimately affects customer satisfaction and service quality.

Description

Technical Field

[0001] The present invention relates to the technical field of customer service response, and more specifically, to a customer service response evaluation method and system based on a large language model. Background Art

[0002] This part refers to the core technology of the invention - the large language model. Large language models (such as GPT, BERT, etc.) are deep learning models trained with a large amount of text data, and have the ability to understand, generate, and convert natural language. In this patent, based on the large language model means that the technology relies on this advanced natural language processing (NLP) capability to solve problems.

[0003] This section describes the scenario of the invention application - customer service response. Customer service response refers to the feedback, solutions, support information, etc. provided by a company or organization to customers, usually provided when customers inquire or complain. Effective customer service response is crucial to improving customer satisfaction, loyalty and brand image. In this context, customer service response is an object that needs to be evaluated.

[0004] Evaluation refers to the quantitative or qualitative analysis and judgment of the quality, effectiveness or other relevant indicators of customer service responses. The purpose of evaluation is to help companies understand the effectiveness of customer service so that they can improve and optimize it. This can include dimensions such as timeliness, accuracy, politeness, emotional understanding, etc.

[0005] For example, the invention patent with announcement number: CN108596434A announces a fraud detection and risk assessment method, system, device and computer-readable storage medium; the method includes the following steps: collecting the original data of the client user; extracting feature data from the original data using a data processing algorithm; inputting the feature data into a pre-trained machine learning model that matches the feature data, generating a model output result, and uploading it to a server; using a risk control decision engine, combining the model output result and historical data and third-party data associated with the client user, outputting fraud detection and risk assessment results. By using the present invention, the computing power of the client device can be fully utilized to reduce the computing pressure of the server. Because the client does not need to upload the original data to the server, the present invention can also reduce the data transmission pressure between the client and the server and the risk of leakage of the user's privacy data and security information.

[0006] The above disclosed technical solutions have at least the following technical problems:

[0007] Sentiment analysis technology plays an important role in customer service evaluation based on large language models. It can not only identify and quantify the emotional state between customers and customer service, but also help companies analyze customer service quality, judge customer satisfaction and discover potential improvement opportunities;

[0008] Sentiment analysis usually sets a threshold for the intensity of sentiment to distinguish between positive, negative, or neutral sentiment. If the threshold is too high, the system may miss some low-intensity sentiments (such as slight disappointment or confusion), resulting in a lack of sensitivity in sentiment judgment. If the threshold is too low, the system will misjudge some neutral or non-emotional expressions as negative sentiments, which may easily lead to unnecessary sentiment matching processing.

[0009] The threshold of sentiment analysis in the prior art is set based on experience, which leads to inaccurate analysis results and unstable service effects, ultimately affecting customer satisfaction and service quality. In view of the above problems, the present invention proposes a solution. Summary of the invention

[0010] In order to overcome the above-mentioned defects of the prior art, an embodiment of the present invention provides a customer service response evaluation method and system based on a large language model, which sets different thresholds for sentiment analysis to solve the problem that the threshold for sentiment analysis is set based on experience, resulting in inaccurate analysis results and unstable service effects, ultimately affecting customer satisfaction and service quality.

[0011] To achieve the above object, the present invention provides the following technical solutions:

[0012] The customer service response evaluation method based on a large language model includes the following steps: obtaining customer service dialogue data, performing data preprocessing and word segmentation, and obtaining a customer service standard text; based on a deep learning sentiment vocabulary enhancement model, identifying sentiment intensity words in the customer service standard text and assigning sentiment scores through a sentiment dictionary and a sentiment tag library to obtain a customer service sentiment text; obtaining a number of different sentiment threshold combinations, and stratifying the customer service sentiment text based on preset sentiment threshold combinations to obtain the customer's sentiment level and respond accordingly; obtaining sentiment matching data and customer sentiment change data in a sentiment analysis process corresponding to each sentiment threshold combination; constructing a sentiment threshold evaluation model based on machine learning according to the sentiment matching data and the customer sentiment change data; constructing an effect display model according to the output of the sentiment threshold evaluation model and the corresponding sentiment threshold combination, and performing data analysis on the effect display model to obtain an optimized sentiment threshold combination, and applying it to customer service response evaluation.

[0013] In a preferred embodiment, the customer service conversation data is obtained, data preprocessing and word segmentation are performed, and the customer service standard text is obtained, specifically: the customer service conversation data is collected from the customer service system, and invalid characters, repeated content, and punctuation interference are removed; the customer service conversation data is anonymized to filter out any personal sensitive information contained therein; the customer service conversation data is split into words or phrases based on a word segmentation tool, and the part of speech is annotated for each word; the customer service conversation data is denoised, and the customer service conversation data is converted into a structured format to obtain the customer service standard text.

[0014] In a preferred embodiment, the deep learning-based sentiment vocabulary enhancement model identifies sentiment intensity words in the customer service standard text and assigns sentiment scores through a sentiment dictionary and a sentiment tag library to obtain the customer service sentiment text, specifically: fine-tuning the BERT pre-training model based on preset system requirements; based on the pre-training model, enhancing the customer service standard text through a customized sentiment vocabulary library; using the pre-training model to identify the sentiment words and intensity words of the enhanced customer service standard text through a deep learning algorithm; assigning sentiment scores to the identified sentiment words and intensity words according to the sentiment dictionary and the sentiment tag library to obtain the customer service sentiment text.

[0015] In a preferred embodiment, the customer service emotional text is stratified based on a preset emotional threshold combination to obtain the customer's emotional level, specifically: different emotional threshold combinations are set, each threshold combination defines the range of different emotional categories; based on the emotional score, the customer service emotional text is classified according to the emotional threshold to obtain the customer's emotional level, and a corresponding response is given.

[0016] In a preferred embodiment, the emotion matching data includes an emotion recall rate fluctuation abnormal coefficient and an emotion matching coefficient; the specific method for obtaining the emotion recall rate fluctuation abnormal coefficient is as follows: obtain the system recall rate and response delay time of the emotion analysis corresponding to the same time interval within a preset time period when performing emotion analysis; and mark the system recall rate and response delay time of the emotion analysis corresponding to the same time interval within a preset time period when performing emotion analysis; calculate the emotion recall rate fluctuation standard deviation and the emotion recall rate fluctuation average value; calculate the emotion recall rate fluctuation variation coefficient based on the emotion recall rate fluctuation standard deviation and the emotion recall rate fluctuation average value; calculate the emotion recall rate fluctuation abnormal coefficient based on the emotion recall rate fluctuation variation coefficient.

[0017] In a preferred embodiment, the specific method for obtaining the emotion matching coefficient is as follows: obtain the emotion intensity score and tone feature value of the customer service response evaluation, and match them to establish an emotion intensity model; obtain the response delay time of the customer service's emotion adjustment when the customer service is emotionally fluctuating, and build an emotion adaptation model based on different emotion adaptation durations; obtain the preset customer service emotion matching standard, and build a matching evaluation model based on the matching standard under different emotional situations, wherein the emotional situation includes a complaint situation, a suggestion situation, and a satisfaction situation; determine the matching evaluation model of the corresponding emotional situation according to the emotional response characteristics of the customer service system; and calculate the emotion matching coefficient in combination with the emotion intensity model and the emotion adaptation model.

[0018] In a preferred embodiment, the customer emotion change data includes an emotion fluctuation influence coefficient; the specific method for obtaining the emotion fluctuation influence coefficient is as follows: obtain the emotion change value of the customer under each emotion threshold after responding in multiple time periods, and construct an emotion change set; pre-process the emotion change value of the customer under each emotion threshold; count the occurrence ratio of each emotion fluctuation in the emotion change set; calculate the information entropy of the emotion change set to obtain the emotion fluctuation influence coefficient.

[0019] In a preferred embodiment, the effect display model is subjected to data analysis to obtain an optimized emotion threshold combination, and the combination is applied to customer service response evaluation, specifically: several Y-axis maximum values ​​of the effect display model are obtained, and the change curvature of the points in the effect display model corresponding to the several Y-axis maximum values ​​is analyzed respectively, and the emotion threshold combination corresponding to the Y-axis maximum value with the largest change curvature is used as the optimized emotion threshold combination, and it is applied to customer service response evaluation.

[0020] Technical effects and advantages of the customer service response evaluation method and system based on a large language model of the present invention:

[0021] 1. The present invention provides an accurate and pure data source for sentiment analysis by carefully preprocessing and denoising customer service dialogue data, removing irrelevant information and ensuring privacy security. The application of word segmentation and part-of-speech tagging enables the large language model to better understand the emotional meaning expressed by customers, thereby improving the accuracy of sentiment recognition. Secondly, through the combination of deep learning enhancement model and sentiment vocabulary library, the emotional intensity of customers can be accurately identified and quantified to form high-precision sentiment analysis results. In addition, by dynamically adjusting and monitoring the sentiment threshold combination, the method effectively avoids the misjudgment of sentiment recognition caused by improper threshold setting, and improves the stability of sentiment analysis and the adaptability of response. The introduction of the abnormal coefficient of sentiment recall rate fluctuation further enhances the recognition ability of the system. By analyzing the impact of the sentiment threshold combination on the recall rate, the sentiment threshold setting is optimized, the accuracy of the system in strong emotion recognition is improved, and the accurate recognition and response to important customer emotions are ensured, and ultimately the customer service quality and customer satisfaction are improved.

[0022] 2. The present invention constructs an emotion threshold evaluation model through emotion matching data and customer emotion change data, thereby realizing accurate analysis and evaluation of customer emotion changes. Specifically, the emotion threshold evaluation model in the scheme comprehensively considers the emotion recall rate fluctuation abnormality coefficient, emotion matching coefficient and emotion fluctuation influence coefficient, and calculates the emotion threshold evaluation coefficient by weighted calculation, so that the model can more comprehensively reflect the actual changes in customer emotions. Further, through the effect display model, the evaluation coefficients under different emotion threshold combinations are visualized and analyzed, and finally the optimized emotion threshold combination is screened out. This optimized combination is applied in customer service response evaluation, which can significantly improve the system's performance in emotion recognition and response accuracy, thereby enhancing customer satisfaction and service effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flow chart of the customer service response evaluation method based on a large language model of the present invention.

[0024] Figure 2 It is a schematic diagram of the structure of the customer service response evaluation system based on the large language model of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] Embodiment 1, Figure 1The present invention provides a customer service response evaluation method based on a large language model, comprising the following steps:

[0027] S1, obtain customer service conversation data, perform data preprocessing and word segmentation, and obtain customer service standard text.

[0028] The customer service dialogue data is obtained, and data preprocessing and word segmentation are performed to obtain the customer service standard text, specifically:

[0029] Collect customer service conversation data from the customer service system and remove invalid characters, repeated content, and punctuation interference;

[0030] Anonymize customer service conversation data and filter out any personal sensitive information;

[0031] Split customer service conversation data into words or phrases based on word segmentation tools, and annotate the part of speech for each word;

[0032] The customer service conversation data is denoised and converted into a structured format to obtain customer service standard text.

[0033] It should be noted that the part of speech is annotated for each word so that the large language model can understand the meaning of the text more accurately; denoising is to process some common noise information, such as the welcome message or general reply automatically generated by the system, to ensure that the conversation data purely reflects the interaction between the customer and the customer service; to protect customer privacy and data security, the conversation data is anonymized and any personal sensitive information (such as name, contact information, address, etc.) that may be contained is filtered out; this not only protects privacy, but also improves the versatility and adaptability of subsequent models;

[0034] The preprocessed text is segmented to split the sentence into more fine-grained words for sentiment analysis. In this step, some Chinese word segmentation tools (such as Jieba or Harbin Institute of Technology LTP) can be used to identify word boundaries. After the word segmentation is completed, part-of-speech tagging is performed, which is very important for the identification of sentiment words and the determination of sentiment intensity, because sentiment words usually have specific parts of speech, and verbs and adjectives have stronger sentiment expressions.

[0035] After completing word segmentation and part-of-speech tagging, convert the text into a structured format, such as recording it in the form of "sentence ID-word-part-of-speech" or "sentence ID-segmentation result", to ensure that the text can be used directly in subsequent analysis. This standard text format can greatly reduce the input complexity of the model and also provide a good foundation for subsequent sentiment vocabulary recognition and sentiment scoring.

[0036] S2, a sentiment vocabulary enhancement model based on deep learning, identifies sentiment intensity words in the customer service standard text and assigns sentiment scores through the sentiment dictionary and sentiment tag library to obtain the customer service sentiment text.

[0037] The deep learning-based sentiment vocabulary enhancement model identifies sentiment intensity words in the customer service standard text and assigns sentiment scores through the sentiment dictionary and sentiment tag library to obtain the customer service sentiment text, specifically:

[0038] Fine-tune the BERT pre-trained model based on preset system requirements;

[0039] Based on the pre-trained model, the customer service standard text is enhanced through a customized sentiment vocabulary library;

[0040] The pre-trained model is used to identify sentiment words and intensity words in the enhanced customer service standard text through deep learning algorithms;

[0041] According to the sentiment dictionary and sentiment tag library, sentiment scores are assigned to the identified sentiment words and intensity words to obtain the customer service sentiment text.

[0042] It should be noted that the custom sentiment vocabulary includes sentiment words, sentiment intensity words, and sentiment polarity (such as positive, negative, neutral, etc.). Sentiment intensity words (such as "very angry", "very happy", etc.) refer to words that can significantly enhance or weaken the expression of emotion. In the model, the sentiment expression in the text is "enhanced" by increasing the weight of these words.

[0043] Identify sentiment words and intensity words in the text through deep learning algorithms (such as LSTM, GRU and other recurrent neural networks, or Transformer structures). This process usually involves feature extraction, using word vectors (such as Word2Vec, GloVe, etc.) or context vectors (such as BERT embeddings) to understand the sentiment tendency and sentiment intensity of words in sentences. The above technologies are already mature and will not be repeated here;

[0044] The sentiment dictionary contains a large number of words that have been annotated with sentiment polarity (such as "positive", "negative", "neutral"). Each word in the dictionary has a corresponding sentiment score, for example, "happy" may correspond to a high positive sentiment score, and "angry" may correspond to a high negative sentiment score.

[0045] Emotional tag library: The emotional tag library refers to the intensity labels set for specific emotional words (such as "slightly happy" and "very angry"). This tag library can quantify emotions in a more detailed manner. In the training of deep learning models, the emotional vocabulary enhancement model will combine the emotional tag library to assign scores to the identified emotional words, taking into account the intensity and context of the words; the above content already has mature technology, so I will not repeat it here.

[0046] S3, obtain several groups of different emotion threshold combinations, and stratify the customer service emotion text based on the preset emotion threshold combinations, obtain the customer's emotion level, and respond accordingly.

[0047] The customer service emotional text is layered based on the preset emotional threshold combination to obtain the customer's emotional level, specifically:

[0048] Set different emotion threshold combinations, each threshold combination defines the range of different emotion categories;

[0049] Based on the sentiment score, the customer service sentiment text is classified according to the sentiment threshold, the customer's sentiment level is obtained, and a corresponding response is given.

[0050] It should be noted that each threshold combination defines the range of different emotion categories (such as positive, negative, neutral). For example, positive emotions may be defined as scores between +0.5 and +1, negative emotions as scores between -1 and -0.5, and neutral emotions as scores between -0.5 and +0.5.

[0051] It should be noted that these threshold combinations can be obtained by analyzing historical customer service data, combined with expert experience or machine learning optimization models. Each threshold combination sets an "emotion classification standard" to distinguish the size and direction of the emotion score, and then divide the emotion level.

[0052] The sentiment threshold combination is a multi-dimensional threshold setting that takes into account sentiment polarity and intensity, for example:

[0053] Negative sentiment: Sentiment score is less than -0.5 and sentiment intensity is greater than 0.3.

[0054] Positive sentiment: Sentiment score is above +0.5 and sentiment intensity is greater than 0.3.

[0055] Neutral sentiment: Sentiment scores range from -0.5 to +0.5, with low intensity (e.g., less than 0.2).

[0056] It is important to note that once the sentiment level and intensity are determined, the customer service team can respond accordingly based on this data. Specifically:

[0057] For positive emotions: If the customer's emotion level is positive and the intensity is high, customer service can continue to encourage the customer and provide more support or promotional information.

[0058] For negative emotions: If the customer's emotion level is negative and the intensity is high, customer service should immediately comfort them and take remedial measures such as apologizing, providing compensation, etc.

[0059] For neutral emotions: When customers show neutral emotions, customer service can continue to provide necessary help or information to maintain a smooth service process.

[0060] S4, obtaining the emotion matching data and customer emotion change data in the emotion analysis process corresponding to each emotion threshold combination.

[0061] The emotion matching data includes the emotion recall rate fluctuation abnormality coefficient and the emotion matching coefficient; the customer emotion change data includes the emotion fluctuation influence coefficient;

[0062] The emotion recall rate fluctuation anomaly coefficient is used to measure the performance stability of the emotion analysis system under different emotion threshold combinations, especially when identifying strong emotions (such as extreme dissatisfaction, anger, etc.). Recall refers to the proportion of a certain emotion category correctly identified by the system, and the fluctuation anomaly coefficient reflects the impact of emotion threshold adjustment on the recall rate. When the system's emotion threshold is set too high or too low, it may not accurately capture certain emotions, especially strong negative emotions. A higher recall rate fluctuation anomaly coefficient indicates that the system fluctuates when identifying strong emotions, and may miss key emotional feedback, thereby affecting customer service quality. Therefore, the emotion recall rate fluctuation anomaly coefficient helps to evaluate the effect of emotion analysis under different emotion threshold settings and ensure the accuracy and stability of the system when processing various emotions.

[0063] Analyzing the abnormal coefficient of emotion recall rate fluctuation is helpful for analyzing the emotion analysis effect corresponding to different emotion thresholds. It can solve the problem that the threshold of emotion analysis is set based on experience, which leads to inaccurate analysis results and unstable service effects, and ultimately affects customer satisfaction and service quality. The advantages are as follows:

[0064] Improve the accuracy of emotion recognition: Through the analysis of the emotion recall rate fluctuation anomaly coefficient, it is possible to quantify the system's ability to recognize strong emotions (such as anger and extreme dissatisfaction) under different threshold settings. A higher recall rate fluctuation anomaly coefficient usually means that the system may have missed some key emotional feedback. Therefore, this coefficient can be used to optimize the threshold to ensure that the system can more comprehensively identify all important emotions, thereby improving the accuracy of emotion analysis.

[0065] Improved stability: The emotion recall rate fluctuation anomaly coefficient can identify the fluctuations of different emotion threshold combinations during the emotion recognition process, thereby effectively reducing unstable analysis results caused by too high or too low thresholds. Stable emotion analysis can ensure that customer emotions are accurately identified and responded to, avoiding fluctuations in service results due to emotional misjudgment.

[0066] Dynamic optimization of sentiment threshold: Through continuous monitoring of the abnormal coefficient of sentiment recall rate fluctuation, the sentiment analysis threshold can be dynamically adjusted based on real-time data feedback, avoiding the need to set a fixed threshold based on empirical values ​​at the beginning, thus improving the adaptive ability of the sentiment analysis system. In this way, the system can adjust the analysis rules based on actual customer sentiment data and continuously optimize service quality.

[0067] Enhance the stability of service quality: The sentiment analysis system can respond appropriately and timely based on the accurately identified sentiment changes. The sentiment recall rate fluctuation anomaly coefficient can help find the most appropriate threshold setting, reduce the inconsistency caused by sentiment recognition errors during the service process, and ultimately ensure that customers get a more stable and personalized service experience.

[0068] Improve customer satisfaction: With the improvement of sentiment analysis accuracy, customers’ emotional feedback can be responded to more promptly and effectively. When the emotional threshold is optimized, customers will feel understood and cared for when they encounter problems, thereby enhancing their trust and satisfaction with the service and improving the overall customer experience.

[0069] Reduce human intervention and experience bias: Relying on the abnormal coefficient of sentiment recall rate fluctuation can reduce the interference of human experience judgment on the setting of sentiment thresholds, avoid the bias that may be caused by traditional sentiment analysis methods based on manual settings, and ensure that the analysis results are more objective and accurate.

[0070] The specific method for obtaining the emotion recall rate fluctuation abnormality coefficient is as follows:

[0071] Obtain the recall rate and response delay time of the system during sentiment analysis corresponding to the same time interval within a preset time period when performing sentiment analysis;

[0072] And mark the recall rate and response delay time of the system during sentiment analysis corresponding to the same time interval within the preset time period when performing sentiment analysis;

[0073] Calculate the standard deviation of emotion recall rate fluctuation and the average value of emotion recall rate fluctuation;

[0074] The coefficient of variation of emotion recall rate fluctuation is calculated based on the standard deviation of emotion recall rate fluctuation and the average value of emotion recall rate fluctuation;

[0075] The emotion recall rate fluctuation anomaly coefficient is calculated based on the emotion recall rate fluctuation variation coefficient.

[0076] The specific calculation formula of the standard deviation of the emotion recall rate fluctuation is as follows:

[0077]

[0078] The specific calculation formula of the emotional recall rate fluctuation average value is as follows:

[0079]

[0080] The specific calculation formula of the emotion recall rate fluctuation abnormal coefficient is as follows:

[0081]

[0082] In the formula, is the standard deviation of sentiment recall rate fluctuation, is the average fluctuation of sentiment recall rate, is the number of tokens for the system’s recall and response latency during sentiment analysis, is the recall rate and response delay time of the system during sentiment analysis, is the recall rate of the system during sentiment analysis, For the response delay time, is the abnormal coefficient of emotion recall fluctuation.

[0083] The sentiment matching coefficient is used to measure the emotional consistency between customers and customer service, that is, to evaluate whether the customer service response is consistent with the customer's emotions. This coefficient reflects the accuracy of sentiment analysis by calculating the degree of emotional matching between customer emotions and customer service responses. Specifically, the sentiment matching coefficient includes two aspects: matching ratio and non-matching analysis. The matching ratio represents the ratio of the number of times the customer's emotions and customer service responses match to the total number of responses. A higher matching ratio indicates that the sentiment analysis system performs better in terms of sentiment matching. Non-matching analysis records and analyzes conversations with mismatched emotions, identifies deficiencies in the threshold settings, and further adjusts the thresholds to improve matching. Through these data analyses, the matching effect can be evaluated under different combinations of emotional thresholds, the system threshold settings can be optimized, and customer service responses can be more accurately matched to the emotional needs of customers, thereby improving customer satisfaction and service quality.

[0084] Analyzing the emotion matching coefficient is helpful for analyzing the emotion analysis effect corresponding to different emotion thresholds. It can solve the problem that the threshold of emotion analysis is set based on experience, which leads to inaccurate analysis results and unstable service effects, and ultimately affects customer satisfaction and service quality. The advantages are as follows:

[0085] Optimize the setting of emotional thresholds: Through the analysis of the emotional matching coefficient, the matching degree between customer emotions and customer service response emotions under different threshold combinations can be quantified, thereby helping the system optimize the setting of emotional thresholds. Compared with setting fixed thresholds based on experience, the emotional matching coefficient provides a data-driven dynamic adjustment solution to ensure that the emotional analysis results are more accurate and avoid emotional misjudgments caused by improper threshold settings.

[0086] Improve the stability of service effects: The emotional matching coefficient can reveal the consistency between customer service response and customer emotions at different emotional thresholds. A higher matching coefficient indicates that the customer service emotional response is consistent with the customer's emotional state, ensuring stable emotional feedback during the service process. This helps reduce service fluctuations caused by emotional mismatch and improves the consistency and stability of service effects.

[0087] Enhanced personalized customer experience: By analyzing the emotional matching degree, it can ensure that customer service not only recognizes the customer's emotions when responding, but also accurately adjusts its response strategy to make it closer to the customer's emotional needs. Accurate emotional matching can provide personalized services, enhance the customer experience, and thus improve customer satisfaction and loyalty.

[0088] Reduce sentiment misjudgment and mismatch: The sentiment matching coefficient can identify mismatched conversations in sentiment analysis and provide feedback to the system, revealing in which cases there may be problems with sentiment threshold settings. Through non-matching situation analysis, the system can adjust the threshold in time to reduce sentiment misjudgment or mismatch, and avoid affecting service quality due to sentiment analysis errors.

[0089] Improve customer satisfaction: With the continuous optimization of the emotional matching coefficient, customer service can respond to customers' emotional needs more accurately, thereby improving customers' emotional experience. The higher the emotional matching degree, the more understanding and care customers will feel when they encounter problems, which enhances their trust in the service and thus improves overall customer satisfaction.

[0090] Enhance the adaptability of sentiment analysis: The sentiment matching coefficient provides a basis for the dynamic adjustment of sentiment thresholds. By real-time monitoring and analyzing the sentiment matching under different threshold combinations, the system can continuously adapt to changes in customer emotions, ensure that sentiment analysis can be adjusted as customer needs change, and improve the accuracy and flexibility of services.

[0091] The specific method for obtaining the emotion matching coefficient is as follows:

[0092] Obtain the emotional intensity score and intonation feature value of customer service response evaluation, and match them to establish an emotional intensity model;

[0093] Obtain the response delay time of customer service staff's emotional adjustment when their emotions fluctuate, and build an emotional adaptation model based on different emotional adaptation durations;

[0094] Obtaining preset customer service emotion matching standards, and establishing a matching evaluation model based on the matching standards under different emotional situations, wherein the emotional situations include complaint situations, suggestion situations, and satisfaction situations;

[0095] Determine the matching evaluation model of the corresponding emotional context according to the emotional response characteristics of the customer service system;

[0096] The emotion matching coefficient is calculated by combining the emotion intensity model and the emotion adaptation model.

[0097] The emotion intensity model is specifically:

[0098]

[0099] The establishment of the emotional adaptation model is specifically as follows:

[0100]

[0101] The matching evaluation model of the complaint situation is:

[0102]

[0103] The matching evaluation model of the proposed scenario is:

[0104]

[0105] The matching evaluation model of the satisfactory situation is:

[0106]

[0107] The specific calculation formula of the emotion matching coefficient is:

[0108]

[0109] Where P is the emotion intensity matching value, is the intonation feature value, is the emotion intensity score, f is the emotion change frequency, is the average sentiment intensity score change value, The duration of emotional adaptation; is the preset standard emotion intensity score, The emotional intensity score of the complaint situation, For the situation to exist for a time, For system response delay, is the sentiment intensity score of the suggested scenario, Add value to the sentiment intensity score, is the preset emotional intensity score in the satisfaction situation, The usage of the satisfactory situation, E is the matching evaluation model, is the emotion matching coefficient, is the emotion intensity model, which is used to obtain the emotion intensity scores in different scenarios, and z is the label of the emotion scenario.

[0110] It should be noted that z is the label of the emotional context, and z={0, 1, 2}.

[0111] It should be noted that the emotion intensity score is assigned through the emotion classification model of the conversation text. The intonation feature values ​​include factors such as pitch, speaking speed, and volume, which reflect the emotional consistency of the two parties in the conversation. The emotion adaptation time data includes the adjustment speed and adaptation time of customer service under different emotion intensities, which can reflect the customer service's empathy and timely response to customer emotions.

[0112] The emotion fluctuation impact coefficient is a key indicator to measure the fluctuation range and change trend of customer emotions during the conversation. It reflects the change range of customer emotions, such as from anger to neutral or satisfied, or from happy to disappointed. The emotion fluctuation impact coefficient helps evaluate the sensitivity and adaptability of the emotion analysis system in dealing with dynamic emotions by analyzing the change rate and fluctuation intensity of customer emotions. Different emotion threshold combinations may show different effects in capturing customer emotion fluctuations. Therefore, the emotion fluctuation impact coefficient can be used to compare and optimize emotion threshold combinations to ensure that the system can accurately capture subtle changes in customer emotions, thereby improving service quality and customer experience.

[0113] Analyzing the emotional fluctuation influence coefficient is helpful for analyzing the emotional analysis effect corresponding to different emotional thresholds. It can solve the problem that the threshold of emotional analysis is set based on experience, which leads to inaccurate analysis results and unstable service effects, and ultimately affects customer satisfaction and service quality. The specific advantages are as follows:

[0114] Improve the accuracy of capturing emotional changes: The emotional fluctuation impact coefficient can effectively track changes in customer emotions and ensure that the system can capture subtle fluctuations in emotions in a timely manner, whether it is from anger to neutrality, or from happiness to disappointment. This can avoid the problem of the emotional threshold being too high or too low, and failing to accurately reflect customer emotional changes.

[0115] Optimize threshold adjustment: By analyzing the amplitude and frequency of emotional fluctuations, the emotional fluctuation impact coefficient can help identify whether the current emotional threshold setting is too loose or strict, thereby adjusting the threshold combination to make it more accurate and avoid the limitations of traditional empirical settings.

[0116] Improve the timeliness of service response: When the emotional fluctuation impact coefficient is accurately analyzed, customer service can more quickly identify and respond to changes in customer emotions. For example, when a customer changes from anger to neutrality, customer service can adjust service strategies in a timely manner to avoid affecting customer experience due to slow response.

[0117] Enhance the accuracy of emotion matching: By capturing the fluctuations in customer emotions, the emotion fluctuation impact coefficient helps optimize the emotion matching coefficient, allowing customer service to respond more in line with the customer's current emotions and reduce emotion mismatches.

[0118] Improve customer satisfaction and service quality: By optimizing the emotion threshold setting and reducing the inaccuracy of emotion analysis, the emotion fluctuation influence coefficient can improve customer satisfaction and ensure the stability of service quality. It can ensure that the customer service system can continuously and accurately analyze customer emotions and provide more efficient and personalized services.

[0119] In summary, the use of sentiment fluctuation influence coefficient can effectively optimize the sentiment analysis threshold, improve the accuracy and stability of sentiment analysis, and thus improve customer experience and the service quality of the enterprise.

[0120] The specific method for obtaining the emotional fluctuation influence coefficient is as follows:

[0121] Obtain the customer's emotion change value under each emotion threshold after responding in multiple time periods, and construct an emotion change set;

[0122] Preprocess the emotional change value of customers under each emotional threshold;

[0123] Count the occurrence ratio of each emotional fluctuation in the emotional change set;

[0124] The information entropy of the emotion change set is calculated to obtain the emotion fluctuation influence coefficient.

[0125] Among them, the specific calculation formula for the occurrence ratio of emotional fluctuations in the emotional change set is as follows:

[0126]

[0127] The specific calculation formula of the emotional fluctuation influence coefficient is as follows:

[0128]

[0129] The final calculation formula of the emotional fluctuation influence coefficient is as follows:

[0130]

[0131] In the formula, is the emotional fluctuation influence coefficient, is the total number of emotional fluctuation types, and j are the types of emotional fluctuations, is the fluctuation of the i-th emotion, is the proportion of emotional fluctuations in the set of emotional changes.

[0132] It should be noted that when analyzing the distribution of customer emotional fluctuations, the higher the information entropy, the more dispersed and unstable the emotional fluctuations are. The emotional fluctuation influence coefficient is obtained based on the calculation results of information entropy.

[0133] This embodiment provides an accurate and pure data source for sentiment analysis by carefully preprocessing and denoising the customer service dialogue data, removing irrelevant information and ensuring privacy security. The application of word segmentation and part-of-speech tagging enables the large language model to better understand the emotional meaning expressed by customers and improves the accuracy of sentiment recognition. Secondly, through the combination of deep learning enhancement model and sentiment vocabulary library, the emotional intensity of customers can be accurately identified and quantified to form high-precision sentiment analysis results. In addition, by dynamically adjusting and monitoring the sentiment threshold combination, the method effectively avoids the misjudgment of sentiment recognition caused by improper threshold setting, and improves the stability of sentiment analysis and the adaptability of response. The introduction of the abnormal coefficient of sentiment recall rate fluctuation further enhances the recognition ability of the system. By analyzing the impact of the sentiment threshold combination on the recall rate, the sentiment threshold setting is optimized, the accuracy of the system in strong emotion recognition is improved, and the accurate recognition and response to important customer emotions are ensured, and ultimately the customer service quality and customer satisfaction are improved.

[0134] Embodiment 2, S5, constructs an emotion threshold evaluation model based on machine learning according to emotion matching data and customer emotion change data.

[0135] The emotion threshold evaluation model is constructed based on machine learning according to the emotion matching data and the customer emotion change data, specifically:

[0136] The obtained emotion recall rate fluctuation abnormal coefficient, emotion matching coefficient and emotion fluctuation influence coefficient are used to establish an emotion threshold evaluation model to generate an emotion threshold evaluation coefficient;

[0137] The calculation formula of the emotion threshold evaluation coefficient is as follows:

[0138]

[0139] In the formula, is the emotion threshold evaluation coefficient, is the preset proportional coefficient of the abnormal coefficient of sentiment recall rate fluctuation, is the preset proportional coefficient of the emotion matching coefficient, is the preset proportional coefficient of the emotional fluctuation influence coefficient, is the emotion recall rate fluctuation abnormality coefficient, is the emotion matching coefficient, is the emotional fluctuation influence coefficient.

[0140] S6, constructing an effect display model according to the output of the emotion threshold evaluation model and the corresponding emotion threshold combination, and performing data analysis on the effect display model to obtain an optimized emotion threshold combination, which is applied to customer service response evaluation.

[0141] The effect display model is constructed based on the output of the emotion threshold evaluation model and the corresponding emotion threshold combination, specifically:

[0142] Obtaining the emotion threshold evaluation coefficient of the output under different emotion threshold combinations from the emotion threshold evaluation model;

[0143] The X-axis is set as the emotional threshold combination, and the Y-axis is represented as the emotional threshold evaluation coefficient to establish a rectangular coordinate system;

[0144] The emotion threshold evaluation coefficient corresponding to each emotion threshold combination is marked on the rectangular coordinate system to form marking points, and adjacent marking points are connected to obtain the effect display model.

[0145] The data analysis of the effect display model is performed to obtain an optimized emotion threshold combination, which is applied to customer service response evaluation, specifically:

[0146] Obtain several Y-axis maximum values ​​of the effect display model, analyze the change curvature of the points in the effect display model corresponding to several Y-axis maximum values, and use the emotion threshold combination corresponding to the Y-axis maximum value with the largest change curvature as the optimized emotion threshold combination, and apply it to customer service response evaluation.

[0147] This embodiment constructs an emotion threshold evaluation model through emotion matching data and customer emotion change data, thereby realizing accurate analysis and evaluation of customer emotion changes. Specifically, the emotion threshold evaluation model in the scheme comprehensively considers the emotion recall rate fluctuation abnormality coefficient, emotion matching coefficient and emotion fluctuation impact coefficient, and calculates the emotion threshold evaluation coefficient by weighted calculation, so that the model can more comprehensively reflect the actual changes in customer emotions. Further, through the effect display model, the evaluation coefficients under different emotion threshold combinations are visualized and analyzed, and finally the optimized emotion threshold combination is screened out. This optimized combination is applied in customer service response evaluation, which can significantly improve the system's performance in emotion recognition and response accuracy, thereby enhancing customer satisfaction and service effects.

[0148] Embodiment 3, Figure 2It is a customer service response evaluation system based on a large language model, including a dialogue data processing module, a sentiment assignment module, a sentiment stratification module, a sentiment analysis process evaluation module, a sentiment threshold evaluation module, and a sentiment threshold optimization module, and there are connections between the modules;

[0149] The dialogue data processing module is used to obtain customer service dialogue data, perform data preprocessing and word segmentation, and obtain customer service standard text;

[0150] The sentiment assignment module is used to enhance the sentiment vocabulary model based on deep learning, identify the sentiment intensity words in the customer service standard text and assign sentiment scores through the sentiment dictionary and sentiment tag library to obtain the customer service sentiment text;

[0151] The emotion stratification module is used to obtain several groups of different emotion threshold combinations, and stratify the customer service emotion text based on the preset emotion threshold combinations, obtain the customer's emotion level, and respond accordingly;

[0152] The sentiment analysis process evaluation module is used to obtain the sentiment matching data and customer sentiment change data in the sentiment analysis process corresponding to each sentiment threshold combination;

[0153] The emotion threshold evaluation module is used to build an emotion threshold evaluation model based on machine learning according to emotion matching data and customer emotion change data;

[0154] The emotion threshold optimization module is used to build an effect display model based on the output of the emotion threshold evaluation model and the corresponding emotion threshold combination, and perform data analysis on the effect display model to obtain the optimized emotion threshold combination, which is then applied to customer service response evaluation.

[0155] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0156] The above embodiments may be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments may be implemented in whole or in part in the form of a computer program product.

[0157] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0158] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0159] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0160] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A customer service response evaluation method based on a large language model, characterized in that: The steps include: Obtain customer service conversation data, perform data preprocessing and word segmentation, and obtain customer service standard text; Based on the deep learning-based sentiment vocabulary enhancement model, the sentiment intensity words in the customer service standard text are identified and sentiment scores are assigned through the sentiment dictionary and sentiment tag library to obtain the customer service sentiment text; Obtain several groups of different emotion threshold combinations, and stratify customer service emotion texts based on the preset emotion threshold combinations, obtain the customer's emotion level, and respond accordingly; Obtaining emotion matching data and customer emotion change data in the emotion analysis process corresponding to each emotion threshold combination, wherein the emotion matching data includes an emotion recall rate fluctuation anomaly coefficient and an emotion matching coefficient, and the emotion matching coefficient is used to measure the emotion consistency between the customer and the customer service, that is, to evaluate whether the customer service response is consistent with the customer's emotion; Build an emotion threshold assessment model based on machine learning according to emotion matching data and customer emotion change data; An effect display model is constructed based on the output of the emotion threshold evaluation model and the corresponding emotion threshold combination. Data analysis is performed on the effect display model to obtain the optimized emotion threshold combination, which is then applied to customer service response evaluation.

2. The customer service response evaluation method based on a large language model according to claim 1, characterized in that: The customer service dialogue data is obtained, and data preprocessing and word segmentation are performed to obtain the customer service standard text, specifically: Collect customer service conversation data from the customer service system and remove invalid characters, repeated content, and punctuation interference; Anonymize customer service conversation data and filter out any personal sensitive information; Split customer service conversation data into words or phrases based on word segmentation tools, and annotate the part of speech for each word; The customer service conversation data is denoised and converted into a structured format to obtain customer service standard text.

3. The customer service response evaluation method based on a large language model according to claim 2, characterized in that: The deep learning-based sentiment vocabulary enhancement model identifies sentiment intensity words in the customer service standard text and assigns sentiment scores through the sentiment dictionary and sentiment tag library to obtain the customer service sentiment text, specifically: Fine-tune the BERT pre-trained model based on preset system requirements; Based on the pre-trained model, the customer service standard text is enhanced through a customized sentiment vocabulary library; The pre-trained model is used to identify sentiment words and intensity words in the enhanced customer service standard text through deep learning algorithms; According to the sentiment dictionary and sentiment tag library, sentiment scores are assigned to the identified sentiment words and intensity words to obtain the customer service sentiment text.

4. The customer service response evaluation method based on a large language model according to claim 3 is characterized in that: The customer service emotional text is layered based on the preset emotional threshold combination to obtain the customer's emotional level, specifically: Set different emotion threshold combinations, each threshold combination defines the range of different emotion categories; Based on the sentiment score, the customer service sentiment text is classified according to the sentiment threshold, the customer's sentiment level is obtained, and a corresponding response is given.

5. The customer service response evaluation method based on a large language model according to claim 4 is characterized in that: The specific method for obtaining the emotion recall rate fluctuation abnormality coefficient is as follows: Obtain the recall rate and response delay time of the system during sentiment analysis corresponding to the same time interval within a preset time period when performing sentiment analysis; And mark the recall rate and response delay time of the system during sentiment analysis corresponding to the same time interval within the preset time period when performing sentiment analysis; Calculate the standard deviation of emotion recall rate fluctuation and the average value of emotion recall rate fluctuation; The coefficient of variation of emotion recall rate fluctuation is calculated based on the standard deviation of emotion recall rate fluctuation and the average value of emotion recall rate fluctuation; The emotion recall rate fluctuation anomaly coefficient is calculated based on the emotion recall rate fluctuation variation coefficient.

6. The customer service response evaluation method based on a large language model according to claim 5, characterized in that: The specific method for obtaining the emotion matching coefficient is as follows: Obtain the emotional intensity score and intonation feature value of customer service response evaluation, and match them to establish an emotional intensity model; Obtain the response delay time of customer service staff's emotional adjustment when their emotions fluctuate, and build an emotional adaptation model based on different emotional adaptation durations; Obtaining preset customer service emotion matching standards, and establishing a matching evaluation model based on the matching standards under different emotional situations, wherein the emotional situations include complaint situations, suggestion situations, and satisfaction situations; Determine the matching evaluation model of the corresponding emotional context according to the emotional response characteristics of the customer service system; The emotion matching coefficient is calculated by combining the emotion intensity model and the emotion adaptation model.

7. The customer service response evaluation method based on a large language model according to claim 6, characterized in that: The customer emotion change data includes an emotion fluctuation influence coefficient; the specific method for obtaining the emotion fluctuation influence coefficient is as follows: Obtain the customer's emotion change value under each emotion threshold after responding in multiple time periods, and construct an emotion change set; Preprocess the emotional change value of customers under each emotional threshold; Count the occurrence ratio of each emotional fluctuation in the emotional change set; The information entropy of the emotion change set is calculated to obtain the emotion fluctuation influence coefficient.

8. The customer service response evaluation method based on a large language model according to claim 7, characterized in that: The data analysis of the effect display model is performed to obtain an optimized emotion threshold combination, which is applied to customer service response evaluation, specifically: The effect display model is constructed based on the output of the emotion threshold evaluation model and the corresponding emotion threshold combination, specifically: Obtaining the emotion threshold evaluation coefficient of the output under different emotion threshold combinations from the emotion threshold evaluation model; The X-axis is set as the emotional threshold combination, and the Y-axis is represented as the emotional threshold evaluation coefficient to establish a rectangular coordinate system; Mark the emotion threshold evaluation coefficient corresponding to each emotion threshold combination on the rectangular coordinate system to form a marking point, connect the adjacent marking points, and obtain the effect display model; Obtain several Y-axis maximum values ​​of the effect display model, analyze the change curvature of the points in the effect display model corresponding to several Y-axis maximum values, and use the emotion threshold combination corresponding to the Y-axis maximum value with the largest change curvature as the optimized emotion threshold combination, and apply it to customer service response evaluation.

9. The customer service response evaluation method based on a large language model according to claim 8, characterized in that: The emotion intensity model is specifically: The emotional adaptation model is specifically: The matching evaluation model of the complaint situation is: The matching evaluation model of the proposed scenario is: The matching evaluation model of the satisfactory situation is: The specific calculation formula of the emotion matching coefficient is: Where P is the emotion intensity matching value, is the intonation feature value, is the emotion intensity score, f is the emotion change frequency, is the average sentiment intensity score change value, The duration of emotional adaptation; is the preset standard emotion intensity score, Score the emotional intensity of the complaint situation, For the situation to exist for a time, For system response delay, is the sentiment intensity score of the suggested scenario, Add value to the sentiment intensity score, is the preset emotional intensity score in the satisfaction situation, The usage of the satisfactory situation, E is the matching evaluation model, is the emotion matching coefficient, is the emotion intensity model, which is used to obtain the emotion intensity scores in different scenarios, and z is the label of the emotion scenario.

10. A system using the customer service response evaluation method based on a large language model as described in any one of claims 1 to 9, comprising a dialogue data processing module, a sentiment assignment module, a sentiment stratification module, a sentiment analysis process evaluation module, a sentiment threshold evaluation module and a sentiment threshold optimization module, and there are connections between the modules; The dialogue data processing module is used to obtain customer service dialogue data, perform data preprocessing and word segmentation, and obtain customer service standard text; The sentiment assignment module is used to enhance the sentiment vocabulary model based on deep learning, identify the sentiment intensity words in the customer service standard text and assign sentiment scores through the sentiment dictionary and sentiment tag library to obtain the customer service sentiment text; The emotion stratification module is used to obtain several groups of different emotion threshold combinations, and stratify the customer service emotion text based on the preset emotion threshold combinations, obtain the customer's emotion level, and respond accordingly; The sentiment analysis process evaluation module is used to obtain the sentiment matching data and customer sentiment change data in the sentiment analysis process corresponding to each sentiment threshold combination; The emotion threshold evaluation module is used to build an emotion threshold evaluation model based on machine learning according to emotion matching data and customer emotion change data; The emotion threshold optimization module is used to build an effect display model based on the output of the emotion threshold evaluation model and the corresponding emotion threshold combination, and perform data analysis on the effect display model to obtain the optimized emotion threshold combination, which is then applied to customer service response evaluation.

Images