Automatic session quality detection method applied to customer service scene

Abstract

The invention relates to the technical field of digital data processing, in particular to an automatic session quality detection method applied to a customer service scene. The detection method comprises the following steps: model training: selecting an LLaMA pre-training large language model as a core engine, and guiding training by means of cue words with session key element extraction and semantic consistency comparison functions; judging the format of the target session data to be subjected to quality detection, and converting the voice format into a text format; performing key element extraction and comparative analysis on the session data in the text format by means of the trained model; consistency comparison is carried out on the robot reply and a corresponding standard reply in a knowledge base; verifying whether business data contained in the robot reply is consistent with real data of a back-end business system; and generating a detailed report of the quality inspection result. Defects are accurately recognized, understood and expressed by means of the large model reasoning ability, and robot reply and a semantic verification closed loop of a knowledge source are achieved.

Description

Technical Field

[0001] This invention relates to the field of digital data processing technology, and more specifically to an automated session quality detection method applied to customer service scenarios. Background Technology

[0002] With the gradual implementation of Large Language Modeling (LLM) in vertical industries of customer service, intelligent chatbots and human-machine collaborative agents have become the main channels for enterprises to serve users. The quality of conversational services, including the accuracy, timeliness, compliance, and emotional experience of responses, directly affects user satisfaction, customer loyalty, and even business risks for enterprises. Currently, the mainstream conversation quality inspection methods in the industry mainly rely on two models:

[0003] Manual sampling quality inspection: Quality inspectors randomly sample and manually listen to / read through a massive amount of recorded conversations or texts, and score them according to a quality inspection checklist. This method has obvious drawbacks, such as high cost, extremely low coverage, strong subjectivity of quality inspectors, inconsistent standards, and long feedback cycles, resulting in serious delays in problem detection.

[0004] Traditional rule / keyword quality inspection: This method automatically screens conversations using preset keywords, regular expressions, and silence duration rules. While this improves efficiency, it has a very weak ability to understand complex semantics and implicit issues, making it difficult to adapt to changing business scenarios and colloquial expressions, resulting in persistently high false negative and false positive rates.

[0005] Traditional quality inspection methods are insufficient to effectively detect and analyze the following three types of deep-seated, high-risk critical defects:

[0006] (i) Key elements of the conversation are missing or incorrect: For example, the robot fails to accurately understand the user's intent, resulting in irrelevant answers; system garbled characters appear in the reply, affecting the user experience; or the business information provided does not match the user's current scenario.

[0007] (ii) The content of the response is inconsistent with the semantics of the knowledge base: For example, the standard answer to a certain question clearly exists in the enterprise's knowledge base, but the robot fails to provide the answer in the dialogue, i.e. "knowledge omission", or provides an answer that contradicts the semantics of the standard answer or is incorrectly expressed.

[0008] (iii) Inconsistency between the response content and the data in the business system: For example, the numerical or status information such as account balance, order status, and product price reported by the robot to the user may deviate from the actual data in the back-end real-time business system, which may lead to customer complaints or even business losses.

[0009] Therefore, there is an urgent need for a new technology that can overcome the limitations of traditional methods and achieve full-volume, automated, in-depth, and fine-grained quality inspection. Summary of the Invention

[0010] This invention aims to construct a full-link intelligent quality inspection system based on a large model, achieving automatic, fine-grained, and interpretable quality inspection. The specific technical solution is as follows:

[0011] An automated session quality detection method for customer service scenarios includes the following process:

[0012] Step 1: Model training; specifically including the following steps:

[0013] Step 1.1: Export a certain amount of conversation data in text and voice formats from the customer service system. The conversation data covers a variety of business scenarios and exception types to ensure the model's generalization ability.

[0014] Step 1.2: Convert the speech-formatted conversation data into text-formatted conversation data and perform noise reduction processing to ensure transcription accuracy; use Automatic Speech Recognition (ASR) technology, such as the Whisper model, to convert it into text format and perform noise reduction processing to ensure transcription accuracy.

[0015] Step 1.3: Select a pre-trained Large Language Model (LLM) as the core engine, and use prompt words to guide the model in quality control; the prompt words include: prompt words for extracting key conversational elements and prompt words for semantic consistency comparison; then conduct model training;

[0016] Step 1.4: Deploy the trained model to a cloud platform or local server, providing an API interface for integration into existing quality inspection systems. Ensure low latency and high concurrency processing capabilities.

[0017] Step 2: Determine the format of the target session data to be quality checked. If it is text data, use it directly. If it is voice data, convert the voice data to text data.

[0018] Step 3: Use the trained model to extract and compare key elements of the text-formatted conversation data;

[0019] Step 4: Compare the robot's response with the corresponding standard response in the knowledge base;

[0020] Step 5: Verify whether the business data contained in the robot's response is consistent with the actual data in the backend business system;

[0021] Step 6: Generate a detailed report of the quality inspection results.

[0022] Preferably, step 3, which uses a trained model to extract key elements and perform comparative analysis on the text-formatted conversation data, includes the following steps:

[0023] Enter single-turn or multi-turn dialogue text ,in, This is the user's message in round i. This is the corresponding reply from the robot or an agent;

[0024] The prompt includes a task description, role definition, few-shot examples, and output format constraints;

[0025] Output: A structured JSON object containing the user intent (Intent), key slots, and error signal field for each round of dialogue;

[0026] Use rules or lightweight models, such as CRF, to check the validity and consistency of key fields.

[0027] Preferably, the process also includes the following:

[0028] In the offline phase, a pre-trained Chinese language model, such as BERT-Base-Chinese or RoBERTa-large, is used to encode all standard questions qⱼ and standard answers aⱼ in the knowledge base into high-dimensional semantic vectors, thus constructing a vector knowledge base.

[0029] Preferably, step 4, which compares the robot's response with the corresponding standard response in the knowledge base, includes the following steps:

[0030] Input: User question Robot Reply and enterprise knowledge base ,in It is a standard question-and-answer pair;

[0031] Objective: To determine Is it? The correct answer in KB; this includes the following two exceptions:

[0032] Knowledge gap: KB contains... Highly relevant issues ,but Unable to provide Content; Knowledge errors: semantics and Corresponding standard answer Inconsistent or contradictory;

[0033] Current user issues Encoded as query vector ;

[0034] Utilize efficient vector retrieval engines, such as FAISS, to quickly retrieve relevant data from vector knowledge bases. Find the top-K standard question vectors with the highest cosine similarity and their corresponding standard answers, a_best;

[0035] The robot's actual response Encode as a vector And calculate its cosine similarity with the standard answer vector V(a_best). ;

[0036] Adaptive threshold determination:

[0037] if The similarity to the most similar standard problem q_best is higher than a dynamic threshold. This indicates that the answer should be present in the knowledge base; but However, the similarity with a_best is below another threshold. If so, it is judged as "knowledge omission" or "knowledge error";

[0038] Dynamic threshold and dynamic threshold By evaluating and calibrating on historical datasets, adaptive adjustments are achieved to accommodate semantic differences across different business domains.

[0039] Preferably, step 5, verifying whether the business data contained in the robot's response is consistent with the actual data in the backend business system, includes the following specific steps:

[0040] Deploy an API call wrapper layer between the chatbot and the backend business system; this layer intercepts all data query requests, records a unique session ID, timestamp, request parameters, and raw, unprocessed JSON data returned from the business system, forming a transaction log;

[0041] The trained model is used to extract all entities and their values ​​related to business data from the robot's responses;

[0042] Based on the session ID and timestamp, locate the API call record corresponding to the current conversation round in the transaction log;

[0043] Compare the data extracted from the dialogue with the original response data recorded in the API logs; for status data, such as "electricity bill issued" or "no arrears", perform strict string matching; for numerical data, such as amount or quantity, set a reasonable difference threshold and fault tolerance strategy.

[0044] The advantages of this invention over the prior art are as follows:

[0045] Traditional quality inspection methods based on keywords or small models are insufficient for detecting deep semantic issues such as the robot "not understanding" the user's true intent. When the robot's responses are semantically incoherent, logically confused, contain garbled characters, or are grammatically correct but completely irrelevant to the business context, existing methods often miss these issues due to a lack of robust contextual understanding and quality assessment capabilities. This invention leverages the reasoning capabilities of large models to accurately identify these deficiencies at the understanding and expression levels.

[0046] Existing technologies lack effective means to verify the semantic consistency between robot responses and their knowledge base. Current quality inspection methods typically cannot determine whether a robot has correctly and accurately used knowledge from the knowledge base. When a standard answer to a question clearly exists in the knowledge base, but the robot fails to provide it or provides incorrect information, traditional quality inspection systems cannot perform in-depth semantic comparison and logical verification between the robot's response and the original knowledge base text, thus failing to detect issues of "knowledge omission" or "knowledge misuse." This invention achieves a closed-loop semantic verification between the robot's response and the knowledge source.

[0047] Existing technologies cannot achieve cross-validation and consistency checks between the robot's responses and real-time data from the backend business system. When providing dynamic data such as user account information, fault status, and electricity bills, the accuracy of the robot's responses is crucial. However, because quality inspection systems and business systems are typically isolated, existing quality inspection methods cannot verify whether the data provided by the robot is consistent with the actual data in the business system. This leads to problems such as inaccurate data acquisition, delayed data updates, or data responses that do not meet customer requirements going undetected in a timely manner, potentially causing user complaints and business risks. This invention integrates the quality inspection process with the business system, enabling automated and accurate verification of key data in the responses. Attached Figure Description

[0048] Figure 1 This is a schematic diagram of the hardware system structure involved in an automated session quality detection method applied to a customer service scenario according to an embodiment of the present invention. Detailed Implementation

[0049] An automated session quality detection method for customer service scenarios includes the following process:

[0050] Step 1: Model training; specifically including the following steps:

[0051] Step 1.1: Export a certain amount of conversation data in text and voice formats from the customer service system. The conversation data covers a variety of business scenarios and exception types to ensure the model's generalization ability.

[0052] Step 1.2: Convert the voice-formatted conversation data into text-formatted conversation data.

[0053] For speech data, Automatic Speech Recognition (ASR) technology, such as the Whisper model, is used to convert it into text format and perform noise reduction processing. This involves first using algorithms or tools such as spectral subtraction and adaptive filtering to remove background noise, echoes, and other interference signals. Then, combined with the text result after ASR conversion, rule matching or lightweight models are used to correct problems such as garbled characters and repeated segments in order to optimize speech quality and ensure transcription accuracy.

[0054] The text data undergoes basic NLP preprocessing, specifically including the following three types of operations, and is ultimately stored in a structured format as a multi-turn dialogue record:

[0055] Sentence segmentation: Based on Chinese grammar rules and the colloquial characteristics of customer service conversations, the period ".", question mark "?", and exclamation mark "!" are used as the main sentence segmentation markers; for colloquial pauses such as "um," "okay," etc., a "comma + semantic pause judgment" logic is added to determine whether a comma constitutes a semantic sentence boundary through word collocation and dialogue context; sentence segmentation modules of NLP toolkits such as jieba and HanLP are used to reduce sentence segmentation errors caused by omissions in manual rules.

[0056] Remove irrelevant characters: Filter by "character type + business relevance", specifically including:

[0057] Basic invalid characters: Remove #, @, Special symbols without actual semantic meaning, redundant spaces, newlines, tabs, and other whitespace characters, as well as " Garbled characters such as “□” are generated due to encoding errors;

[0058] Business-irrelevant characters: Filter redundant information that is not related to the conversation content, such as system identifiers such as "[Customer Service System Prompt: Conversation is connected]" at the beginning / end of the conversation, non-business-related URLs sent by customer service by mistake, and repetitive and meaningless verbal redundancy such as "uh uh uh" and "that um".

[0059] Standardized format: Focusing on the inconsistent representation of key information such as dates and numbers, a unified format is adopted according to business standards, specifically including:

[0060] a. Date standardization: Colloquial and non-standard dates such as "yesterday", "the 15th of last month", and "2024.5.28" are uniformly converted to the "YYYY-MM-DD" format, such as "2024-05-28", by matching rules, such as "yesterday" corresponding to "current date - 1 day" and "the 15th of last month" corresponding to "current year - current month - 1 - 15", or by using Python's datetime module.

[0061] b. Number standardization: Standardize the format of numerical expressions such as amount, quantity, and number. For example, unify "two hundred yuan" and "200 yuan" into the "number + unit" format, such as "200 yuan". Standardize mixed case numbers such as "ID123a" into a fixed case format, such as "id123a".

[0062] Data annotation: Some data were manually annotated. The annotation content and specific measures are as follows. The annotation results are used for model fine-tuning or evaluation:

[0063] Anomaly Type Labeling: Abnormal responses from the robot / agent in the conversation data are precisely labeled according to preset anomaly categories. For example, when the robot replies with "Sorry, I didn't understand your question," "Please repeat your request," or provides irrelevant answers or replies containing garbled characters, these are uniformly labeled as "Ununderstood" anomalies. Irrelevant answers include situations such as a user inquiring about "electricity bill inquiry" but receiving a reply about "fault reporting process," or replies containing garbled characters. Replies containing garbled characters include situations such as: "Your account balance is..." ? If the reply content is completely unrelated to the business scenario, such as a user inquiring about a power outage but being promoted financial products, it will be marked as an "unrelated to business" exception, ensuring that each type of exception corresponds to a clear conversation segment.

[0064] Semantic consistency labeling: This involves labeling the semantic matching degree between the robot's response and the corresponding standard question-and-answer pair in the knowledge base. First, locate the standard question and answer corresponding to the user's question in the knowledge base. For example, if a user asks "What are the ways to pay residential electricity bills?", the knowledge base contains a clear list of standard payment methods. Then, determine whether the robot's response fully covers the core information of the standard answer and whether the semantics are consistent. If the response accurately includes the content of the standard answer, it is labeled "knowledge match"; if the response omits key information or contradicts the meaning, it is labeled "knowledge mismatch". Examples of omitted key information include: not mentioning the "online APP payment" method; examples of semantic contradiction include: incorrectly stating "residential electricity bills cannot be paid online".

[0065] Data Consistency Labeling: Verify and label the consistency between business data in the robot's response and the actual data in the backend business system. Extract key business data from the robot's response, such as "last four digits 1234" and "balance 56.8 yuan" in "Your account with the last four digits 1234 has an electricity balance of 56.8 yuan". Query the real-time data of the corresponding session ID in the business system, such as retrieving the actual balance of the user's account with the last four digits 1234 from the electricity management system. If the two values ​​and statuses are completely consistent, such as the system displaying a balance of 56.8 yuan, label it as "Business data is correct". If there are discrepancies or data representation errors, such as a balance of 56.8 yuan being displayed as 65.8 yuan, or a mislabeled last four digit as 1243, the label will be incorrect.

[0066] Step 1.3: Select a pre-trained large language model (LLM) as the core engine, such as GPT-3, LLaMA, or Llama2-Chat. These models have powerful few-shot learning and contextual understanding capabilities, making them suitable for handling complex quality inspection tasks.

[0067] Configuration method:

[0068] Cue word engineering: Design targeted cue words to guide the model through quality checks, avoiding costly fine-tuning. Cue word configuration should follow the principles of "clear objectives, scenario adaptation, and structured output." Specific examples are as follows:

[0069] The prompt for extracting key conversation elements is: "Please analyze the following dialogue and identify whether the bot has misunderstood the user, displayed garbled text, or provided content that is irrelevant to the business. Output the anomaly type and reason." This process directly and clearly defines the goal of "detecting anomalies in key conversation elements," specifies the output dimension of "anomaly type + reason," and conforms to the configuration rules.

[0070] The prompt for semantic consistency comparison is: "There is an answer to [knowledge point] in the knowledge base, but does the robot's response effectively provide this information? Answer yes or no, and explain." This process adapts to the business needs of customer service scenarios through "[knowledge point]", clearly defines the output format of "yes / no judgment + explanation", and conforms to the configuration rules.

[0071] Fine-tuning: If the business scenario is highly specific, such as fault reporting, electricity consumption and electricity costs, fine-tuning can be performed using manual feedback data. This invention uses the InstructGPT method for fine-tuning to improve model alignment and accuracy.

[0072] Model training: First, prioritize pre-trained large language models (LLMs) with strong few-shot learning and contextual understanding capabilities as the core engine, such as GPT-3, LLaMA, or Llama2-Chat, to meet the complex needs of customer service conversation quality inspection; then design targeted prompt words to be integrated into the training. For the task of "extracting key elements of the conversation", design prompt words containing "task objective + output requirements". For example, the task objective can be input as: identify ununderstood users, garbled characters, or content that is not relevant to the business. The output requirements can be input as: exception type + reason.

[0073] For the "semantic consistency comparison" task, prompt words containing "scenario variable + judgment requirement + output format" are designed. The scenario variable can be [knowledge point], the judgment requirement can be "whether knowledge is effectively provided", and the output format can be "yes / no + explanation" to guide the model to learn the quality inspection logic. Finally, combined with human feedback data of highly specific business scenarios, the model is fine-tuned using the InstructGPT method. During the training process, preprocessed session data covering various business scenarios and anomaly types are continuously input, and prompt words are adjusted or supplemented samples are added in multiple rounds of iteration to ensure that the model can stably identify key quality inspection points such as anomalies in key session elements and differences in semantic consistency.

[0074] Step 1.4: Deploy the trained model to a cloud platform or local server and provide an API interface to integrate it into the existing quality inspection system, ensuring low latency and high concurrency processing capabilities.

[0075] Step 2: Determine the format of the target session data to be quality checked. If it is text data, use it directly. If it is voice data, convert the voice data to text data.

[0076] Step 3: Use the trained model to extract and compare key elements of the text-formatted conversation data:

[0077] Key elements include:

[0078] The user intent in each round of dialogue, that is, the core need of the user to initiate the dialogue, such as "electricity consumption and electricity bill inquiry" or "fault repair application";

[0079] The key slots in each round of dialogue are the core information related to user needs, such as "electricity usage type" and "time range" when checking electricity bills, and "address" and "fault type" when reporting a problem.

[0080] Abnormal signals in each round of dialogue, i.e. problems with the robot's response, such as "not understanding the user's intent", "response containing garbled characters", "content irrelevant to the business scenario", and "logical contradiction";

[0081] The model will first extract the key elements mentioned above, and then combine them with lightweight models such as rules or CRF to check the validity and consistency of the key fields and complete the comparative analysis.

[0082] Prompt Design: Design a complex prompt that includes a task description, role definition, a few-shot example, and output format constraints. This prompt guides the large model to identify not only general business intent, but also, as illustrated by the example prompt: "In a conversation, detect if the customer service chatbot's responses have the following problems: 1. Failure to understand user intent, such as replying 'Please repeat' or answering an irrelevant question; 2. Content irrelevant to the business scenario, such as promoting unrelated products. List all problems and provide evidence."

[0083] The following abnormal signals should also be identified:

[0084] unintelligible: The bot's response indicates that it did not understand the user's question, such as "Sorry, I didn't understand your question."

[0085] garbled_response: The response contains garbled characters, invalid characters, or incorrect formatting.

[0086] irrelevant_content: The reply content is logically unrelated to the user's current question or context, i.e., it is not the answer to the question.

[0087] logical_conflict: The reply contains a logical contradiction with the preceding dialogue or generally accepted facts.

[0088] Structured Output: The Prompt requires LLM to output results in a predefined JSON format, including exception type, location, and confidence level. This ensures that downstream programs can reliably parse and use the quality inspection results, avoiding secondary processing of unstructured text. Example output is as follows:

[0089] {

[0090] "round": 1,

[0091] "user_intent": "Electricity and Bill Inquiry",

[0092] "slots": {"card_type": "residential electricity consumption", "time_range": "last month"},

[0093] "error_signal": null},{

[0094] "round": 2,

[0095] "slots": {},

[0096] "error_signal": "garbled_response"}

[0097] Post-processing and validation: Perform final validation on the JSON output by the LLM. Use rules or lightweight model CRF to check the validity and consistency of key fields, for example, check whether the extracted slot values ​​are within the enumeration range allowed by the business; for example, if the model identifies "strong negative sentiment".

[0098] Step 3 combines the generalized semantic understanding capabilities of the large model with the rigor of structured data. By using the large model LLaMA to extract key conversation elements and identify implicit abnormal signals such as "not understanding user intent" and "garbled replies," the validity of key information is verified by combining rules or CRF models. This overcomes the limitations of traditional "keyword matching" which relies solely on surface character matching and can accurately capture deep conversation defects that affect user experience but cannot be identified by traditional methods. This achieves a leap from "keyword matching" to "semantic defect identification." The key conversation elements include: user intent and key slots.

[0099] Step 4: Compare the robot's response with the corresponding standard response in the knowledge base using the following three methods;

[0100] Semantic vectorization: The BERT sentence vector model is used to encode the robot's response and knowledge points in the knowledge base into semantic vectors, and cosine similarity is calculated. If the similarity is below a threshold, such as 0.9, it is judged as inconsistent.

[0101] Large-scale model-assisted judgment: Combining prompt word engineering, for example: "User asks: [User question]. Knowledge base answer: [Knowledge content]. Robot reply: [Reply content]. Is the reply semantically consistent? Output yes / no and the reason." This method can effectively identify and process subtle semantic differences in the conversation, thereby improving the accuracy of quality inspection results.

[0102] Knowledge Coverage Detection: For "unanswered response" scenarios, a large model is used to identify user intent and map it to the knowledge base. If the knowledge base contains relevant knowledge points but the chatbot's response is invalid, such as "Sorry, I couldn't find your information," it is marked as a defect.

[0103] The method used in step 4 goes beyond literal matching. By comparing the semantic vector space, it can accurately determine the deep semantic consistency between the response content and the standard answer in the knowledge base, effectively solving the core pain point of "knowledge exists but is not provided correctly".

[0104] Step 5: Verify whether the business data contained in the robot's response is consistent with the actual data in the backend business system;

[0105] Data entity extraction: Extract key business entities from the robot's responses, such as customer ID, amount, and date. For example, a prompt might be: "Extract all business entity types and values ​​from the following text."

[0106] For example, extract the following from the query "Your electricity balance ending in 8888 is 234.56 yuan": { "entity": "balance", "value": 234.56, "card_tail": "8888"}.

[0107] Automated validation: Generate validation instructions using a large model, such as SQL queries or API calls; execute instructions to retrieve real data from the business system and compare it with the extracted data at the field level, such as status and value.

[0108] For status data, such as "electricity bill issued" or "no outstanding fees", strict string matching is performed.

[0109] For numerical data, such as amount or quantity, set a reasonable difference threshold and fault tolerance strategy.

[0110] Consistency check: If the comparison results are inconsistent, such as the electricity bill result reported in the reply being inconsistent with the query result in the business system, it will be automatically marked as a data error and a correction process will be triggered.

[0111] Step 5 establishes a reliable "chain of evidence" by connecting the front-end dialogue content with the back-end data source. This enables automated and precise verification of the accuracy of dynamic business data during the conversation for the first time, effectively preventing business risks caused by data errors.

[0112] Step 6: Generate a detailed report of the quality inspection results, including anomaly statistics, consistency scores, etc., to help managers identify problem trends.

[0113] Model iteration: Regularly retrain with new data or optimize prompt words to adapt to business changes. Utilize human feedback, such as annotation evaluations, for reinforcement learning fine-tuning to improve model performance.

[0114] Closed-loop processing: For detected issues, relevant personnel, such as the customer service team, are automatically notified, and the issues are integrated into the workflow for repair, forming a closed loop for quality improvement.

Claims

1. An automated session quality detection method applied to customer service scenarios, characterized in that, The process includes the following: Step 1: Model training; specifically including the following steps: Step 1.1: Export a certain amount of conversation data in text and voice formats from the customer service system. The conversation data covers a variety of business scenarios and exception types. Step 1.2: Convert the speech data into text data and perform noise reduction; use Automatic Speech Recognition (ASR) technology to convert it into text format and perform noise reduction to ensure transcription accuracy; Step 1.3: Select a pre-trained Large Language Model (LLM) as the core engine, and use prompt words to guide the model in quality control; the prompt words include: prompt words for extracting key conversational elements and prompt words for semantic consistency comparison; conduct model training; Step 1.4: Deploy the trained model to a cloud platform or local server and provide an API interface; Step 2: Determine the format of the target session data to be quality checked. If it is text data, use it directly. If it is voice data, convert the voice data to text data. Step 3: Use the trained model to extract and compare key elements of the text-formatted conversation data; Step 4: Compare the robot's response with the corresponding standard response in the knowledge base; Step 5: Verify whether the business data contained in the robot's response is consistent with the actual data in the backend business system; Step 6: Generate a detailed report of the quality inspection results.

2. The automated session quality detection method for customer service scenarios according to claim 1, characterized in that, Step 3, which uses the trained model to extract key elements and perform comparative analysis on the text-formatted conversation data, includes the following steps: Input single-turn or multi-turn dialogue text. Conversation = {(u1,a1), (u2, a2), ..., (u n ,a n )}, where uᵢ is the user's message in the i-th round, and aᵢ is the corresponding reply from the robot (or agent); The prompt includes a task description, role definition, few-shot examples, and output format constraints; Output: A structured JSON object containing the user intent (Intent), key slots, and error signal field for each round of dialogue; Use rules or lightweight models to check the validity and consistency of key fields.

3. The automated session quality detection method for customer service scenarios according to claim 1, characterized in that, It also includes the following processes: In the offline phase, a Chinese pre-trained language model is used to encode all standard questions qⱼ and standard answers aⱼ in the knowledge base into high-dimensional semantic vectors, thus constructing a vector knowledge base.

4. The automated session quality detection method for customer service scenarios according to claim 3, characterized in that, Step 4, which compares the robot's response with the corresponding standard response in the knowledge base, includes the following steps: Input: user question uᵢ, robot response aᵢ, and enterprise knowledge base KB ={q1, a1; q2, a2; ...}, where (qⱼ, aⱼ) is a standard question-answer pair; Objective: Determine if aᵢ is the correct answer to uᵢ in KB; this includes the following two exceptions: Knowledge gap: The KB contains an issue qⱼ that is highly related to uᵢ, but aᵢ fails to provide content for aⱼ; knowledge gap Error: The semantics of aᵢ are inconsistent with or contradict the standard answer aⱼ corresponding to uᵢ; Encode the current user question uᵢ into a query vector V(uᵢ); Using an efficient vector retrieval engine, the top-K standard question vectors with the highest cosine similarity to V(uᵢ) are quickly retrieved from the vector knowledge base, and the corresponding standard answer a_best is found; the robot's actual response aᵢ is encoded into a vector V(aᵢ), and its cosine similarity Sim(V(aᵢ), V(a_best)) with the standard answer vector V(a_best) is calculated. If the similarity between uᵢ and the most similar standard question q_best is higher than a dynamic threshold T1, but the similarity between aᵢ and a_best is lower than another threshold T2, then it is judged as "knowledge omission" or "knowledge error". Dynamic thresholds T1 and T2 are evaluated and calibrated on historical datasets.

5. The automated session quality detection method for customer service scenarios according to claim 2, characterized in that, Step 5, verifying whether the business data contained in the robot's response is consistent with the actual data in the backend business system, includes the following specific steps: Deploy an API call wrapper layer between the chatbot and the backend business system; this layer intercepts all data query requests, records a unique session ID, timestamp, request parameters, and raw, unprocessed JSON data returned from the business system, forming a transaction log; The trained model is used to extract all entities and their values ​​related to business data from the robot's responses; Based on the session ID and timestamp, locate the API call record corresponding to the current conversation round in the transaction log; Compare the data extracted from the dialogue with the original response data recorded in the API logs; for status data, perform strict string matching; for numerical data, set a reasonable difference threshold and fault tolerance strategy.

Images