Automatic evaluation method and system for intelligent customer service Agent
By employing an end-to-end intelligent customer service automation evaluation method, and combining a large language model for intent recognition, contextual coherence, and sentiment matching assessment, this approach addresses the subjectivity and adaptability issues of complex scenarios in existing intelligent customer service evaluation technologies, enabling efficient and accurate multi-dimensional assessment and optimization suggestion generation.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-09
- Publication Date
- 2026-03-13
AI Technical Summary
Existing intelligent customer service performance evaluation methods mainly rely on manual assessment, which results in highly subjective results, is time-consuming and costly, and is difficult to cover large-scale scenarios. Furthermore, existing automated evaluation schemes cannot simulate complex multi-turn dialogues and contextual changes, and lack assessment of semantic understanding and sentiment matching.
An end-to-end automated evaluation method is adopted. User dialogue logs are collected and preprocessed, and large language models are used to evaluate intent recognition, contextual coherence, and sentiment matching. The indicator weights are dynamically adjusted according to business scenarios, and a visual report is generated to provide optimization suggestions.
It achieves fully automated, multi-dimensional intelligent customer service performance evaluation, improving evaluation efficiency and accuracy, enhancing the practicality and interpretability of evaluation results, and enabling comprehensive evaluation in complex scenarios.
Smart Images
Figure CN121659025A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of artificial intelligence technology, and relates to an evaluation method and system for intelligent customer service agents, particularly an automated evaluation method and system for intelligent customer service agents. Background Technology
[0002] Intelligent customer service systems are widely used in e-commerce, finance, government affairs, and other fields, and their performance directly affects user experience. As business scenarios become increasingly complex, intelligent customer service systems need to handle multiple tasks, including multi-turn dialogues, complex intent recognition, sentiment assessment, and business process execution.
[0003] However, for a long time, the performance evaluation methods for intelligent customer service have mainly included manual sampling evaluation schemes, rule-based automated testing schemes, and single-index automated evaluation schemes. However, these schemes all have certain shortcomings.
[0004] 1. Manual sampling evaluation scheme.
[0005] The scheme involves manually reviewing dialogue logs and scoring based on experience, including dimensions such as response accuracy, service attitude, and process standardization. Its shortcomings are as follows: (1) The evaluation process relies on human experience, and the results are highly subjective; (2) It is time-consuming and costly, and cannot achieve large-scale coverage; (3) It is difficult to reproduce consistent evaluation standards, and its coverage of different business scenarios is also limited.
[0006] 2. Automated testing solutions based on rule scripts.
[0007] The scheme uses a fixed question-and-answer script to detect whether the system response is correct. Its shortcomings include: (1) it cannot simulate complex multi-turn dialogues and context changes; (2) it is difficult to evaluate the system's ability to handle new intentions or boundary scenarios; and (3) it lacks measurement of advanced capabilities such as semantic understanding and sentiment matching.
[0008] 3. Single-indicator automated evaluation solution.
[0009] Single-metric automated evaluation solutions may only measure metrics such as response time, keyword matching rate, and intent recognition accuracy. However, they neglect comprehensive metrics such as dialogue state tracking and contextual coherence, making it difficult to reflect the overall performance of intelligent customer service.
[0010] Given the aforementioned shortcomings of existing technologies, there is an urgent need for a new evaluation technology that can cover real business scenarios, automate execution, and comprehensively evaluate intelligent customer service systems from multiple dimensions. Summary of the Invention
[0011] To overcome the shortcomings of existing technologies, this invention proposes an automated evaluation method and system for intelligent customer service agents, which can achieve end-to-end, real-time, and multi-dimensional comprehensive evaluation of intelligent customer service agents, meeting the needs of modern application scenarios.
[0012] To achieve the above objectives, the present invention provides the following technical solution: An automated evaluation method for intelligent customer service agents, characterized by the following steps: S1: Collect and preprocess user dialogue logs from the intelligent customer agent to obtain the user's intent in asking questions, structured dialogue information, and the sentiment of the user's dialogue. S2: Compare the user's question intent with the gold standard generated by the big language model, evaluate intent recognition by calculating accuracy and recall, evaluate the contextual coherence in multi-turn dialogues based on the state tracking capability of the big language model based on the structured dialogue information, and evaluate the sentiment matching degree based on the sentiment of the user's dialogue and the user's emotions. S3: Dynamically adjust the weights of the indicators for intent recognition assessment, contextual coherence assessment, and sentiment matching assessment based on the business scenario to obtain automated assessment results; S4: Display the results of automated assessments and provide optimization suggestions.
[0013] Preferably, in step S1, the user dialogue log includes a user text dialogue log and a user voice dialogue log, and the user voice dialogue log is converted from voice to text.
[0014] Preferably, in step S1, obtaining the user's intent in asking the question specifically includes: Remove stop words and special characters from the text, and then segment it into words; Use a language model to extract semantic features to generate contextual semantic vectors; Use a classification model to classify the intent; Regular rules or knowledge graph detection can be used to enhance classification results, thereby revealing the intent behind user questions.
[0015] Preferably, in step S1, obtaining the emotion of the user's dialogue specifically includes: Semantic features are extracted by combining acoustic emotion features of speech audio. Sentiment classification is performed using a sentiment classification model based on the aforementioned semantic features; By combining the results of sentiment classification with context, the sentiment of the user's conversation can be obtained.
[0016] Preferably, in step S1, obtaining the structured dialogue information specifically includes: Perform round numbering and timestamp processing; Perform multiple rounds of context merging; Build the global state of the dialogue; Mark errors; Encapsulate the dialogue information in a unified format to obtain a structured format.
[0017] Preferably, in step S2, evaluating the contextual coherence in multi-turn dialogues using the state tracking capability of a large language model based on structured dialogue information specifically includes: State extraction is performed for each round of dialogue using a large language model; By checking whether the state is updated reasonably and whether the state is lost, a multi-turn dialogue consistency comparison is performed to obtain the contextual coherence assessment results.
[0018] Preferably, in step S3, dynamically adjusting the weights of the indicators for intent recognition assessment, contextual coherence assessment, and sentiment matching assessment based on the business scenario specifically includes: Identify the scenario type of the current dialogue using a classifier or large language model; Initial indicator weights are selected from a predefined scenario-weight mapping table based on the scenario type of the current dialogue. The initial indicator weights are dynamically adjusted based on the performance data of the intelligent customer service agent, or an algorithm based on reinforcement learning is used to reward or punish the weights of each initial indicator according to the overall evaluation effect, thereby achieving automatic and continuous adjustment of the weights.
[0019] Furthermore, the present invention also provides an automated evaluation system for intelligent customer service agents, characterized in that it includes: The data acquisition and preprocessing module is used to collect user dialogue logs from the intelligent customer agent and preprocess them to obtain the user's intent in asking questions, structured dialogue information, and the sentiment of the user's dialogue. The multi-dimensional evaluation module compares the user's question intent with the gold standard generated by the large language model. It evaluates intent recognition by calculating accuracy and recall, evaluates the contextual coherence in multi-turn dialogues based on the state tracking capability of the large language model using structured dialogue information, and evaluates sentiment matching based on the emotion of the user's dialogue and the user's emotions. The dynamic weight adjustment module is used to dynamically adjust the weights of the indicators for intent recognition assessment, contextual coherence assessment, and sentiment matching assessment according to the business scenario in order to obtain automated assessment results. The visualization report generation module is used to display the results of automated assessments and provide optimization suggestions.
[0020] Furthermore, the present invention also provides an automated evaluation device for intelligent customer service agents, characterized in that it includes: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the automated evaluation method for the intelligent customer service agent as described above. Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, characterized in that, when the program is executed by a processor, it implements the steps of the automated evaluation method for the intelligent customer service agent as described above.
[0021] Compared with existing technologies, the automated evaluation method and system for intelligent customer service agents of the present invention have one or more of the following beneficial technical effects: 1. Fully Automated Evaluation: This invention uses multimodal data processing technologies such as text, voice, and user behavior logs to build an end-to-end evaluation process that requires no manual intervention, greatly improving evaluation efficiency and accuracy.
[0022] 2. Multi-dimensional comprehensive evaluation: This invention integrates multiple core indicators, including intent recognition, dialogue state tracking, and sentiment matching, to achieve a comprehensive performance analysis of the intelligent customer service system.
[0023] 3. Dynamic weight adjustment: This invention can automatically adjust the weight of each indicator according to the business scenario, thereby improving the practicality and business relevance of the evaluation results.
[0024] 4. Enhanced explainability: This invention can pinpoint specific defects in the intelligent customer service system through visual reports, thereby enhancing explainability. Attached Figure Description
[0025] Figure 1 This is a flowchart of the automated evaluation method for the intelligent customer service agent of the present invention.
[0026] Figure 2 This is a schematic diagram of the automated evaluation system for the intelligent customer service agent of the present invention. Detailed Implementation
[0027] Before detailing any embodiment of the invention, it should be understood that the invention, in its application, is not limited to the details of the construction and arrangement of the components set forth in the following description or illustrated in the following figures. The invention can have other embodiments and can be practiced or carried out in various ways. Furthermore, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered limiting. The use of “comprising” or “having” and variations thereof is intended to cover the items set forth below and their equivalents, as well as any additional items. Unless otherwise specified or limited, the terms “installation,” “connection,” “support,” and “linkage,” and variations thereof are used broadly and cover both direct and indirect installation, connection, support, and linking. Moreover, “connection” and “linkage” are not limited to physical or mechanical connections or links. Furthermore, firstly, in the disclosure of this invention, the terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the above terms should not be construed as limiting this invention. Secondly, the term "a" should be understood as "at least one" or "one or more," that is, in one embodiment, the number of an element can be one, while in another embodiment, the number of the element can be multiple. The term "a" should not be construed as a limitation on the quantity.
[0028] Figure 1 A flowchart of the automated evaluation method for the intelligent customer service agent of the present invention is shown. Figure 1 As shown, the automated evaluation method for the intelligent customer service agent of the present invention includes the following steps: S1: Data Acquisition and Preprocessing.
[0029] In this invention, user dialogue logs of the intelligent customer agent are collected and preprocessed to obtain the user's intent in asking questions, structured dialogue information, and the sentiment of the user's dialogue.
[0030] The user dialogue log includes user text dialogue logs and user voice dialogue logs, and the user voice dialogue logs are converted from voice to text.
[0031] The user text dialogue log includes user natural language questions and system responses, and the format may include timestamps, session IDs, round numbers, etc.
[0032] The user voice dialogue log is the original audio file input by the user through voice, which may contain different device sampling formats (such as 8kHz / 16kHz PCM).
[0033] Furthermore, the data collected by this invention may also include a business knowledge base and historical manually annotated data. The business knowledge base includes structured or semi-structured data such as intent definitions, slot fields, process rules, and FAQ question-and-answer pairs. The historical manually annotated data may include intent tags, sentiment tags, entity boundaries, standard answers, and dialogue states.
[0034] In this invention, the voice-to-text operation on the user's voice dialogue log specifically includes: 1. Audio standardization processing.
[0035] First, audio with different sampling rates and encoding methods is uniformly converted into an internal standard format (such as 16kHz, 16-bit PCM).
[0036] Then, noise reduction (spectral subtraction), echo cancellation, and voice endpoint detection (VAD) are performed on the standard format audio.
[0037] 2. Acoustic model recognition.
[0038] First, acoustic features (MFCC, FBank, MelSpectrogram) are extracted using a deep neural network-based acoustic model.
[0039] Then, acoustic decoding is performed on the acoustic features.
[0040] 3. Language model fusion.
[0041] First, the acoustic model output is fused with an n-gram or Transformer-based language model to improve recognition accuracy.
[0042] Then, the recognition results are corrected by combining common entity names and business terms from the business knowledge base to further improve the recognition accuracy.
[0043] 4. Text post-processing.
[0044] First, the identified text is segmented and punctuation restored.
[0045] Then, the recognized text is normalized in terms of number format (e.g., "ten point five" → "10.5").
[0046] Finally, correct any errors in business terminology (e.g., if "warranty period" is mistakenly identified as "repair period", it will be automatically corrected to "warranty period").
[0047] After speech-to-text conversion, the user's intent in asking questions, structured dialogue information, and the emotions in the user's dialogue can be obtained based on the user's text dialogue log and the converted speech text.
[0048] 1. Obtain the user's intent in asking the question, which specifically includes: (1) Text cleaning and word segmentation.
[0049] First, remove stop words, special characters, etc. from the text.
[0050] Then, the Chinese characters in the text are encoded using BPE, SentencePiece, or character-level encoding.
[0051] (2) Semantic feature extraction.
[0052] Use a language model (BERT or RoBERTa) to generate contextual semantic vectors. During the semantic feature extraction process, it supports extracting semantic features by using multi-round context concatenation (such as concatenating the first 3 rounds of user messages).
[0053] (3) Intent classification model discrimination.
[0054] Intent classification is performed using a softmax-based multi-classification model or a Siamese / Bi-encoder-based semantic matching model (suitable for open domains). In intent classification, multi-intent and multi-label scenarios (such as complaint + refund) are supported.
[0055] (4) Enhanced business rules.
[0056] Regular rules and knowledge graphs are used to detect whether business keywords are included, and these are combined with the classification results of the intent classification model to improve the accuracy of specific category identification.
[0057] 2. Obtain the emotional state of the user's conversation, which specifically includes: (1) Semantic feature extraction.
[0058] Use a language model (BERT or RoBERTa) to generate contextual semantic vectors. During the semantic feature extraction process, it supports extracting semantic features by using multi-round context concatenation (such as concatenating the first 3 rounds of user messages).
[0059] (2) Emotional classification.
[0060] Sentiment classification is performed using a sentiment classification model, such as RoBERTa-wwm-ext-Sentiment. During the sentiment classification process, acoustic emotion features of the speech audio (MFCC, pitch, and speech rate) can be incorporated. Furthermore, the sentiment classification model supports three levels of classification: positive, neutral, and negative; and can perform fine-grained classification: anger, anxiety, complaining, happiness, request for help, etc., ultimately providing the emotion intensity quantified from 0 to 1.
[0061] (3) Context fusion.
[0062] The system can consider the emotional shift between the user's current message and the previous system response. If the user's negative emotions increase for three consecutive rounds, the system records it as an "emotional escalation event".
[0063] 3. Obtain structured dialogue information, specifically including: (1) Perform round numbering and timestamp processing.
[0064] Mark the order of each round of dialogue and establish a round sequence structure of user-system-user.
[0065] (2) Perform multiple rounds of context merging.
[0066] Combine consecutive short sentences into one round or break down very long sentences into parsable small units.
[0067] (3) Construct the global state of the dialogue.
[0068] Build a global state of the dialogue, including: current intent, confirmed slots, missing slots, whether it is interrupted, current business process node, etc., to facilitate dialogue state tracking (DST).
[0069] (4) Mark errors.
[0070] For example: Are there repeated questions? Is there a loss of context? Is the assistant incorrectly redirecting intent?
[0071] (5) Unified format packaging.
[0072] It performs uniform format encapsulation and finally outputs JSON / structured format.
[0073] S2: Multi-dimensional assessment.
[0074] In this invention, the intent of the user's question is compared with the gold standard generated by a large language model. Intent recognition is evaluated by calculating accuracy and recall. The contextual coherence in multi-turn dialogues is evaluated using the state tracking capability of the large language model based on structured dialogue information. Emotional matching is evaluated based on the sentiment of the user's dialogue and the user's emotions. Specifically, this includes: 1. Intent Recognition Evaluation: Compare the intent recognition results of the intelligent customer service agent (that is, the intent of the user's question obtained in step S1) with the gold standard (true value), and calculate the accuracy, recall, etc., to achieve the measurement of intent recognition.
[0075] (1) Accuracy: TP (True Positive) represents the number of times the intelligent customer service agent correctly identifies the intent; FP (False Positive) represents the number of times the intelligent customer service agent incorrectly identifies the intent.
[0076] (2) Recall rate: FN (False Negative) represents the number of times the actual intent was not recognized by the intelligent customer service agent.
[0077] 2. Contextual Coherence Assessment (DST): Based on the state tracking capabilities of a large language model, it detects logical jumps, repeated questions, or loss of context in multi-turn dialogues, including: (1) Large language model state extraction: Use a large language model (such as the dialogue version of GPT / BERT) to extract the state for each round of dialogue. The extracted state includes: current intent, slot value (filled / unfilled), current task flow node, and dialogue history memory state.
[0078] (2) Consistency comparison of multiple rounds of dialogue: First, check whether the status is updated reasonably. For example, if the user does not provide an order number but the intelligent customer service agent directly replies "Your order is on its way", it is judged as a context inference error. When the intelligent customer service agent asks the user to provide the same slot again, it is judged as a duplicate question. Second, check whether the status is lost. For example, if the user has provided an order number in the previous round, and the order number is 123456, but the intelligent customer service agent asks again in the next round: Can you provide the order number? It is judged as a loss of context.
[0079] 3. Emotional Matching Assessment: Analyze whether the user's emotions are consistent with the emotional tone of the intelligent customer service agent's response (that is, the emotional tone of the user's dialogue obtained in step S1), such as whether the user was soothed in an angry situation.
[0080] The sentiment matching evaluation typically begins by using a sentiment classification model (such as RoBERTa-emotion or a domain-specific self-trained model) to output sentiment labels (e.g., anger, anxiety, joy, neutral) for both user input and the intelligent customer service agent's response, generating sentiment intensity scores. Then, a matching degree is calculated based on a business-defined sentiment matching matrix. For example, when a user is angry, the intelligent customer service agent's "soothing sentiment" scores higher than "neutral," while "joy" scores the lowest. In practice, the sentiment labels of both parties can be mapped to vectors and quantified using cosine similarity or a sentiment distance function to obtain a matching degree score between 0 and 1. Furthermore, combining interjection detection (e.g., "Don't worry," "I understand how you feel") and style consistency scoring from a generative large language model can form a comprehensive sentiment matching score.
[0081] S3: Dynamic weight adjustment.
[0082] The weights of the indicators for intent recognition assessment, contextual coherence assessment, and sentiment matching assessment are dynamically adjusted according to the business scenario to obtain automated assessment results.
[0083] Specifically, dynamic weight adjustment is based on real-time identification results of business scenarios (such as complaint scenarios, emergency scenarios, consultation scenarios, etc.). First, the scenario type of the current dialogue is identified through a classifier or large language model, and then initial weights are selected from a predefined scenario-weight mapping table. Afterwards, the initial weights are dynamically fine-tuned based on recent model performance data (fluctuations in the last n scores, error frequency of a certain dimension, etc.). For example, a simple proportional adjustment algorithm can be used: if the "emotion dimension score is consistently low," its weight is automatically increased; or a reinforcement learning-based Bandit algorithm can be used to reward or penalize each dimension based on the overall evaluation effect, aligning the overall optimization direction with business priorities, thereby achieving automatic and continuous weight adjustment.
[0084] S4: Visual report generation.
[0085] Display the results of automated assessments and provide optimization suggestions.
[0086] In this invention, a visual report containing radar charts and error heatmaps can be generated, highlighting scenarios of misidentified intent, high-frequency contextual errors, and cases of sentiment mismatch, and automatically providing optimization suggestions, such as supplementing training data for a certain type of intent if needed.
[0087] Furthermore, this invention also enables a feedback optimization closed loop. This loop is typically triggered when the score in any dimension falls below a threshold. Improvement suggestions are automatically generated based on the cause of the error: if the error is "intent recognition error," features are collected from the error samples to generate a small dataset for fine-tuning the intent model, and configurations directly applicable to model fine-tuning or LoRA training are provided; if the error is "inappropriate sentiment," rule optimization suggestions are given, such as adding specific sentiment-related phrase templates or adjusting the strategy engine logic. These optimization suggestions are automatically summarized by the large language model based on error examples (e.g., common misidentification of refund scenarios – suggestion to supplement refund-related training samples). Finally, the optimization suggestions are pushed to the training module or strategy library, achieving automated closed-loop optimization.
[0088] Figure 2 A schematic diagram of the automated evaluation system for the intelligent customer service agent of the present invention is shown. Figure 2 As shown, the automated evaluation system for the intelligent customer service agent of the present invention includes: 1. Data acquisition and preprocessing module.
[0089] The data acquisition and preprocessing module is used to collect user dialogue logs from the intelligent customer agent and preprocess them to obtain the user's intent in asking questions, structured dialogue information, and the sentiment of the user's dialogue.
[0090] 2. Multi-dimensional assessment module.
[0091] The multi-dimensional evaluation module is used to compare the user's question intent with the gold standard generated by the large language model, evaluate intent recognition by calculating accuracy and recall, evaluate the contextual coherence in multi-turn dialogues based on the state tracking capability of the large language model using structured dialogue information, and evaluate the sentiment matching degree based on the emotion of the user's dialogue and the user's emotions.
[0092] 3. Dynamic weight adjustment module.
[0093] The dynamic weight adjustment module is used to dynamically adjust the weights of the indicators for intent recognition assessment, contextual coherence assessment, and sentiment matching assessment according to the business scenario, so as to obtain automated assessment results.
[0094] 4. Visual report generation module.
[0095] The visualization report generation module is used to display the results of automated assessments and provide optimization suggestions.
[0096] Furthermore, the present invention also provides an automated evaluation device for intelligent customer service agents, comprising: one or more processors; a memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the automated evaluation method for intelligent customer service agents of the present invention. Finally, the present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the automated evaluation method for the intelligent customer service agent of the present invention.
[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit the scope of protection of the present invention. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention based on the concept of the present invention, without departing from the essence and scope of the technical solutions of the present invention.
Claims
1. An automated evaluation method for intelligent customer service agents, characterized in that, Includes the following steps: S1: Collect and preprocess user dialogue logs from the intelligent customer agent to obtain the user's intent in asking questions, structured dialogue information, and the sentiment of the user's dialogue. S2: Compare the user's question intent with the gold standard generated by the big language model, evaluate intent recognition by calculating accuracy and recall, evaluate the contextual coherence in multi-turn dialogues based on the state tracking capability of the big language model based on the structured dialogue information, and evaluate the sentiment matching degree based on the sentiment of the user's dialogue and the user's emotions. S3: Dynamically adjust the weights of the indicators for intent recognition assessment, contextual coherence assessment, and sentiment matching assessment based on the business scenario to obtain automated assessment results; S4: Display the results of automated assessments and provide optimization suggestions.
2. The automated evaluation method for the intelligent customer service agent according to claim 1, characterized in that, In step S1, the user dialogue log includes user text dialogue log and user voice dialogue log, and the user voice dialogue log is converted from voice to text.
3. The automated evaluation method for the intelligent customer service agent according to claim 2, characterized in that, In step S1, obtaining the user's intent in asking the question specifically includes: Remove stop words and special characters from the text, and then segment it into words; Use a language model to extract semantic features to generate contextual semantic vectors; Use a classification model to classify the intent; Regular rules or knowledge graphs can be used to enhance the classification results, thereby revealing the intent behind the user's question.
4. The automated evaluation method for the intelligent customer service agent according to claim 2, characterized in that, In step S1, obtaining the emotion of the user's dialogue specifically includes: Semantic features are extracted by combining acoustic emotion features of speech audio. Sentiment classification is performed using a sentiment classification model based on the aforementioned semantic features; By combining the results of sentiment classification with context, the sentiment of the user's conversation can be obtained.
5. The automated evaluation method for the intelligent customer service agent according to claim 2, characterized in that, In step S1, obtaining the structured dialogue information specifically includes: Perform round numbering and timestamp processing; Perform multiple rounds of context merging; Build the global state of the dialogue; Mark errors; Encapsulate the dialogue information in a unified format to obtain a structured format.
6. The automated evaluation method for the intelligent customer service agent according to any one of claims 1-5, characterized in that, In step S2, evaluating the contextual coherence in multi-turn dialogues based on the state tracking capability of a large language model using structured dialogue information specifically includes: State extraction is performed for each round of dialogue using a large language model; By checking whether the state is updated reasonably and whether the state is lost, a multi-turn dialogue consistency comparison is performed to obtain the contextual coherence assessment results.
7. The automated evaluation method for the intelligent customer service agent according to claim 6, characterized in that, In step S3, dynamically adjusting the weights of the indicators for intent recognition assessment, contextual coherence assessment, and sentiment matching assessment based on the business scenario specifically includes: Identify the scenario type of the current dialogue using a classifier or large language model; Initial indicator weights are selected from a predefined scenario-weight mapping table based on the scenario type of the current dialogue. The initial indicator weights are dynamically adjusted based on the performance data of the intelligent customer service agent, or an algorithm based on reinforcement learning is used to reward or punish the weights of each initial indicator according to the overall evaluation effect, thereby achieving automatic and continuous adjustment of the weights.
8. An automated evaluation system for intelligent customer service agents, characterized in that, include: The data acquisition and preprocessing module is used to collect user dialogue logs from the intelligent customer agent and preprocess them to obtain the user's intent in asking questions, structured dialogue information, and the sentiment of the user's dialogue. The multi-dimensional evaluation module compares the user's question intent with the gold standard generated by the large language model. It evaluates intent recognition by calculating accuracy and recall, evaluates the contextual coherence in multi-turn dialogues based on the state tracking capability of the large language model using structured dialogue information, and evaluates sentiment matching based on the emotion of the user's dialogue and the user's emotions. The dynamic weight adjustment module is used to dynamically adjust the weights of the indicators for intent recognition assessment, contextual coherence assessment, and sentiment matching assessment according to the business scenario in order to obtain automated assessment results. The visualization report generation module is used to display the results of automated assessments and provide optimization suggestions.
9. An automated evaluation device for intelligent customer service agents, characterized in that, include: One or more processors; Memory, used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the automated evaluation method for the intelligent customer service agent as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the automated evaluation method for the intelligent customer service agent as described in any one of claims 1-7.