Multi-agent collaborative large model multi-round dialogue capability automatic evaluation method and system

Abstract

The application discloses a multi-agent cooperative large model multi-turn dialogue capability automatic evaluation method and system, and belongs to the technical field of artificial intelligence. The application solves the problem of urgently needing an efficient, automatic, quantifiable, low-cost and comprehensive large model multi-turn dialogue capability evaluation method. The application comprises the following steps: S1: according to the target evaluation requirement, configuring the user role attribute for the user simulation agent; S2: the user simulation agent carries out multi-turn dialogue with the measured large model according to the set dialogue scene, and obtains multi-turn dialogue samples; S3: the evaluation agent evaluates the multi-turn dialogue samples based on the target evaluation requirement, and outputs the evaluation result. Through the cooperative work of the three types of agents, the application realizes efficient, comprehensive and quantifiable automatic evaluation of the large language model for the voice-to-customer scene, significantly improves the evaluation efficiency and accuracy, and provides a clear direction for model iteration optimization.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to an automatic evaluation method and system for the multi-turn dialogue capability of large models with multi-agent collaboration. Background Technology

[0002] Large Language Models (LLMs), with their powerful natural language understanding and generation capabilities, are rapidly being applied in various voice-based customer service scenarios, acting as "digital employees" to achieve human-like interactions in tasks such as business processing, customer inquiries, problem-solving, and marketing promotions. However, in practical applications, how to efficiently, accurately, and quantifiably evaluate the performance of such large models in specific scenarios has become one of the key challenges for technology implementation. Currently, the industry generally relies on traditional subjective experience and manual telephone testing to evaluate the multi-turn dialogue effectiveness of such models. While these methods can reflect the interaction quality of the model to some extent, they have the following significant drawbacks:

[0003] 1. Low testing efficiency: Manual telephone testing requires a large amount of human resources and has a long testing cycle, which cannot meet the needs of large-scale, high-frequency model iteration.

[0004] 2. High labor costs: Reliance on manual assessment leads to a sharp increase in operating costs, and assessment results are easily affected by subjective factors, resulting in poor consistency.

[0005] 3. Limited scenario coverage: Manual testing is difficult to cover all business scenarios and edge cases, resulting in insufficient comprehensiveness and representativeness of the evaluation results.

[0006] 4. Lack of quantitative standards: Subjective evaluation makes it difficult to form a unified and quantifiable indicator system, which is not conducive to the horizontal comparison and optimization iteration of models.

[0007] 5. Severe response delay: The cycle from testing to feedback is long, slowing down the overall progress of model optimization and product launch.

[0008] Therefore, there is an urgent need for an efficient, automated, quantifiable, low-cost, and comprehensive evaluation method to support the continuous optimization and efficient implementation of voice-based customer-facing LLMs. Summary of the Invention

[0009] To address the urgent need in existing technologies for an efficient, automated, quantifiable, low-cost, and comprehensive method for evaluating the multi-turn dialogue capabilities of large-scale models, this invention provides an automatic evaluation method and system for the multi-agent collaborative capabilities of large-scale models in multi-turn dialogue. This method is used to efficiently, objectively, comprehensively, and quantifiably evaluate the overall performance of large language models (AUTs) in multi-turn dialogues for voice-to-customer scenarios, significantly improving evaluation efficiency and accuracy, reducing reliance on manual intervention and costs, and accelerating model iteration and optimization.

[0010] The technical solution adopted in this invention is as follows:

[0011] An automatic evaluation method for the multi-turn dialogue capability of large-scale multi-agent collaborative models includes the following steps:

[0012] S1: Configure user role attributes for the user simulated intelligent agent based on the target assessment requirements;

[0013] S2: The user simulates an intelligent agent to conduct multiple rounds of dialogue with the large model under test according to the set dialogue scenario, and obtains multi-round dialogue samples.

[0014] S3: The evaluation agent evaluates multi-turn dialogue samples based on the target evaluation requirements and outputs the evaluation results.

[0015] As a preferred option, user role attributes in S1 include user personality.

[0016] Preferably, the user's personality includes emotional state, language style, knowledge background, and behavioral tendencies.

[0017] Preferably, in S1, user role attributes are configured for the user-simulated intelligent agent based on the constructed USA role library. The steps for constructing the USA role library are as follows:

[0018] S001: Based on historical dialogue logs, user survey data, and business expert experience, abstractly define typical user profiles;

[0019] S002: Design behavioral characteristic parameters for typical user profiles;

[0020] S003: Generate corresponding USA prompt words based on behavioral feature parameters using templated or parameterized methods.

[0021] Preferably, the evaluation results output in S3 include: a unique identifier for the dialogue, a list of round scores for each evaluation dimension, the overall dialogue score for each evaluation dimension, the overall dialogue score, and detailed scoring reasons; the detailed scoring reasons include specific text snippets and explanations for each scoring point.

[0022] As preferred, the evaluation dimensions include: naturalness, fluency, logical consistency, factual accuracy, appropriateness of emotional response, task completion, compliance, and initiative.

[0023] Preferably, the multi-turn dialogue sample includes the input and output of each round of interaction, and the input and output include timestamps, role identifiers, and dialogue content.

[0024] An automatic evaluation system for the multi-agent collaborative large-scale model multi-turn dialogue capability, comprising:

[0025] The User Simulation Agent Module (USA) is used to simulate real users and the test target large-scale model to conduct multi-turn dialogues and obtain multi-turn dialogue sample data.

[0026] The Evaluation Agent (EA) module is used to perform multi-dimensional, fine-grained automated evaluation of multi-turn dialogue sample data based on a predefined, structured quantitative indicator system and scoring rules, and output the evaluation results.

[0027] The dialogue management and storage module is used to maintain the dialogue state and store multi-turn dialogue sample data;

[0028] The automated pipeline scheduling module is used to coordinate the dialogue generation of the user simulated agent module, the evaluation triggering of the evaluation agent module, and the result processing flow.

[0029] Preferably, a results visualization and analysis module is also included to display the evaluation results and perform results analysis.

[0030] Preferably, the user simulation agent module, the evaluation agent module, the dialogue management and storage module, and the automated pipeline scheduling module are connected and communicate through application programming interfaces or local service calls.

[0031] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:

[0032] 1. Efficiency Leap: Automated generation of massive dialogue samples and real-time evaluation, shortening the testing cycle from days / weeks to minutes / hours, improving testing capabilities by several orders of magnitude, perfectly adapting to agile development and rapid iteration.

[0033] 2. Sharply reduced costs: Significantly reduces or even completely replaces expensive and inefficient manual surveying, thereby significantly reducing assessment and operating costs.

[0034] 3. Objective and Quantitative Assessment: Establish a unified, fine-grained, multi-dimensional quantitative indicator system, output structured scores and detailed reasons, eliminate subjective bias, and ensure that the assessment results are objective, fair, and highly comparable.

[0035] 3. Comprehensive and in-depth coverage: Test cases can be easily configured and generated to cover all business scenarios and various user profiles (including long-tail and edge cases), with evaluation depth and breadth far exceeding that of manual testing.

[0036] 4. Real-time and efficient feedback: The automated pipeline achieves a closed-loop "test-evaluation-feedback" process that is completed in minutes, greatly accelerating the model defect location and optimization process.

[0037] 5. Reusability and scalability: By configuring the USA role library and EA prompts, it can quickly adapt to new business scenarios, assessment dimensions and compliance requirements, making the system highly versatile.

[0038] 6. Support data-driven decision-making: Provides rich quantitative data and problem insights, offering solid data support for model selection, version release, and performance optimization. Attached Figure Description

[0039] Figure 1 This is a flowchart of the process of the present invention;

[0040] Figure 2 This is a system architecture diagram of the present invention;

[0041] Figure 3 This is a schematic diagram of the USA role configuration in this invention;

[0042] Figure 4 This is a schematic diagram of a visualization dashboard for the evaluation results of this invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0044] An automatic evaluation system for the multi-agent collaborative large-scale model multi-turn dialogue capability includes:

[0045] The user-simulated intelligent agent module is used to simulate multi-turn dialogues between real users and the tested large-scale client model, obtaining multi-turn dialogue sample data. This module possesses configurable and diverse personality traits, including but not limited to emotional states (such as calm, anger, impatience, and hesitation), language styles (such as polite, rude, concise, and verbose), knowledge background (such as novice or experienced user), and behavioral tendencies (such as proactive questioning, easily led, easily interrupted, and prone to vague expression). It can generate context-appropriate intelligent agents based on preset business scenarios (such as "credit card repayment consultation," "broadband fault reporting," and "goods return / exchange") and target intent. User dialogue can simulate typical user behavior patterns in different situations, such as asking questions, requesting clarification, changing the subject, and expressing dissatisfaction. The dialogue will end when the user says "goodbye" to the customer model or when a threshold is reached. It should be noted that the threshold is generally set based on experience, typically 60 rounds in complex scenarios and 30 rounds in simple scenarios. The user simulation agent module can be implemented based on existing LLM user simulators, such as SDialog (an open-source toolkit that supports persona-driven multi-agent dialogue generation) and RecUserSim (an LLM-based user simulator that supports diverse user profiles).

[0046] The Evaluation Agent (EA) module is used to perform multi-dimensional, fine-grained automated evaluation of multi-turn dialogue sample data based on a predefined, structured quantitative indicator system and scoring rules, and output the evaluation results. EA prompt word design: clearly defines the specific meaning and scoring criteria for each evaluation dimension (e.g., factual accuracy: completely accurate = 10 points, core factual error = 0 points; emotional response: perfectly fitting = 10 points, exacerbating conflict = 0 points). Strict output format requirements (JSON Schema) are specified. A Prompt structure is designed to guide the EA in detailed reasoning and provide sufficient justification. In other embodiments, a consistency verification mechanism (such as averaging multiple scores or majority voting) can be added. The evaluation agent module can adopt existing LLM-as-Judge evaluation paradigms, such as NVIDIA NeMo Evaluator (providing LLM-as-Judge scoring and structured evaluation capabilities) and DeepEval (40+ out-of-the-box indicators).

[0047] The dialogue management and storage module is used to maintain dialogue state and store multi-turn dialogue sample data (database or file system). Specifically, it realizes the start, progress and end of dialogue by maintaining dialogue state data such as historical dialogue, current user question, dialogue turn number, and end marker. The dialogue management and storage module can adopt existing dialogue state tracking technologies, such as Prompt-Guided DST (based on GPT-2) or Noetic State Representation Graph.

[0048] The automated pipeline scheduling module coordinates the dialogue generation of the user-simulated agent module, the evaluation triggering of the evaluation agent module, and the result processing. Specifically, it can use a workflow engine (such as Dify) or custom scripts to orchestrate the entire process: USA role / scene sampling -> Initiate USA-AUT dialogue -> Dialogue record storage -> Trigger EA evaluation -> Evaluation result storage -> Report generation. It supports on-demand triggering, scheduled tasks, or continuous integration / continuous deployment (CI / CD) integration. The automated pipeline scheduling module can adopt existing workflow engines (such as Airflow) or MLOps pipeline frameworks (such as NVIDIA NeMo Evaluator's automated workflow).

[0049] The results visualization and analysis module is used to display evaluation results and perform result analysis. It can develop or integrate a visualization dashboard to intuitively display multi-dimensional evaluation results and problem clustering. It provides result export (CSV, Excel) and API interfaces for use by downstream systems (such as model training platforms and project management tools). It can establish a correlation mechanism between evaluation results and model optimization action items (Bug Tickets). The results visualization and analysis module can utilize existing visualization tools, such as Vibes-eval, the Data-to-Dashboard multi-agent framework, or Kibana.

[0050] The user-simulated agent module, evaluation agent module, dialogue management and storage module, automated pipeline scheduling module, and results visualization and analysis module are connected and communicate through application programming interfaces or local service calls. For example, they connect to AUT, USA, and EA through standardized APIs (such as OpenAI API, Hugging Face Inference Endpoint) or local model services.

[0051] like Figure 1-3 As shown, the automatic evaluation method for the multi-turn dialogue capability of large-scale multi-agent collaborative models includes the following steps:

[0052] S1: As Figure 3 As shown, based on the target evaluation requirements, user role attributes are configured for the user simulated intelligent agent. These attributes include user personality. Specifically, this embodiment configures user role attributes for the user simulated intelligent agent based on the constructed USA role library. The steps for constructing the USA role library are as follows:

[0053] S001: Based on historical dialogue logs, user survey data, and business expert experience, abstract and define typical user profiles (Persona).

[0054] S002: Design behavioral characteristic parameters (emotional state, language style, knowledge background, behavioral tendencies) for typical user profiles.

[0055] S003: Based on behavioral feature parameters, use templated or parameterized methods to generate corresponding USA prompt words to ensure that the character's behavior is controllable and reproducible.

[0056] S2: The user simulates an intelligent agent to conduct multiple rounds of dialogue with the tested large model according to the set dialogue scenario (such as "credit card repayment consultation", "broadband fault repair", "goods return and exchange"), and obtains multi-round dialogue samples; the multi-round dialogue samples include the input and output of each round of interaction, and the input and output include timestamps, role identifiers and dialogue content.

[0057] S3: The evaluation agent assesses multi-turn dialogue samples based on the target evaluation requirements and outputs the evaluation results. Output format: Based on strict rules defined by the prompts (e.g., a 0-10 scoring system, a graded system), the output evaluation results include:

[0058] (1) Unique identifier for a conversation (a conversation will generate a unique hash code as a unique identifier for the conversation).

[0059] (2) Round score list for each evaluation dimension;

[0060] (3) Overall dialogue scores for each assessment dimension;

[0061] (4) Overall dialogue score (which can be a dimensional weighted average);

[0062] (5) Detailed scoring reasons: For each scoring point, provide specific text snippets and explanations (e.g., "In the third round, the user asked 'What is the interest rate?', and AUT answered 'Annualized interest rate of 3.8%', which is inconsistent with the knowledge base 'Product A's interest rate is 3.85%', so 2 points are deducted for factual accuracy").

[0063] The detailed scoring criteria include specific text snippets and explanations for each scoring point. Evaluation dimensions include: Naturalness: Whether the language is human-like and conforms to everyday spoken language habits, avoiding a mechanical feel; Fluency: Whether sentences are fluent, without grammatical errors or semantic breaks; Logical Coherence: Whether the dialogue context flows smoothly and logically, and whether the answers are self-consistent, avoiding contradictions or logical jumps; Factual Accuracy: Whether the business knowledge, product information, and user data in the answers are completely consistent with the preset, authoritative business knowledge base; Emotional Appropriateness: Whether the AUT's response effectively identifies and adapts to the user's current emotional state in the USA simulation (e.g., soothing anger, encouraging hesitation); Task Completion: Whether the dialogue effectively guides the user to achieve preset business goals (e.g., successfully answering questions, completing procedures); and Compliance. Does the response comply with relevant laws and regulations (such as the Consumer Rights Protection Law), industry regulatory requirements, and company policies (such as privacy statements and risk warnings)? Proactiveness: Does AUT proactively provide helpful information or guide the conversation at appropriate times? Supports dialogue-level, turn-level, and aggregated statistics analysis.

[0064] Results Analysis and Iteration Phase:

[0065] 1. Visualization and Summary: Evaluation results are displayed through an integrated visual dashboard (e.g., ...). Figure 4 As shown), it includes: score distribution of each dimension (histogram, box plot), comparative analysis of different scenarios / user types, clustering of high-frequency error issues (NLP analysis based on the reasons provided by EA), quantitative comparison between model version iterations (such as A / B testing), and statistics on the achievement rate of key indicators (KPIs).

[0066] 2. Problem identification:

[0067] By combining the detailed evaluation reasons output by the EA, the root cause of the AUT's defects can be accurately located, for example:

[0068] (1) Incorrect answers to specific business knowledge points (factual defects);

[0069] (2) Inappropriate response to certain emotions (such as intense complaints) (emotional adaptation deficit);

[0070] (3) Logical breaks occur after complex multi-round interactions (logic consistency defects).

[0071] (4) Key compliance statements were not triggered in specific scenarios (compliance defects).

[0072] 3. Guidance on optimization:

[0073] Provide model training engineers with clear data support and optimization directions (such as supplementing specific domain knowledge data, adjusting fine-tuning strategies, optimizing prompt word templates, and fixing logical vulnerabilities), driving rapid iteration and performance improvement of AUT.

[0074] 4. Scalable applications:

[0075] Large-scale model "Hallucination" testing: In the dialogue scripts or role behaviors of the USA, questions or challenges to key factual information (such as specific product interest rates, fees, user privacy information) are strategically inserted. The EA rigorously examines the factual accuracy of the AUT's answers and quantifies the occurrence rate of hallucination.

[0076] Automated Consumer Rights Protection (CUP) Compliance Approval: The "Compliance Reviewer" role is pre-set in the USA role library, or typical CUP-related questions (such as "What is your privacy policy?", "How do I file a complaint if I'm dissatisfied with the service?", "Are there any hidden fees in this clause?") are randomly inserted into regular conversations. EA automatically assesses the compliance and accuracy of AUT responses based on CUP regulations and company policy libraries, generating a compliance audit report.

[0077] Multilingual and cross-cultural adaptability assessment: By configuring the language model of the USA (such as multilingual LLM) and cultural background parameters (such as expression habits and taboo topics), the AUT's conversational fluency, cultural sensitivity and task completion ability in different language environments and cultural backgrounds are tested.

[0078] Stress testing and robustness assessment: Configure USA to simulate user behavior under high stress, high noise (such as voice recognition error input), and high concurrency scenarios to evaluate the stability and fault tolerance of the AUT.

[0079] Benchmarking before new model / version goes live: As part of an automated pipeline, quickly quantify the performance improvement or regression of the new version of the AUT relative to the baseline version.

[0080] This example uses "credit card repayment consultation" as a business scenario to evaluate the overall performance of a financial customer service big data model (AUT), as follows:

[0081] S1. Based on the target evaluation requirements, configure user role attributes for the user-simulated intelligent agent:

[0082] 1. System initialization configuration

[0083] # USA Role Configuration Example (JSON Format)

[0084] {

[0085] "scene": "credit_card_repayment",

[0086] "persona": {

[0087] "emotion_state": "anxious", # Initial emotion: Anxiety

[0088] "language_style": "polite",

[0089] "knowledge_level": "novice", # Business newbie

[0090] "behavior_tendency": {

[0091] "query_frequency": 0.8, # High tendency to ask follow-up questions

[0092] "interruption_prob": 0.3

[0093] }

[0094] },

[0095] "target_intent": "clarify_minimum_repayment", # Target intent: clarify the minimum repayment amount

[0096] "key_entities": ["card_number", "due_date"]

[0097] }

[0098] (2) EA evaluation rule configuration

[0099] Taking factual accuracy as an example, set matching rules based on the business knowledge base:

[0100] # Factual accuracy scoring rules (pseudocode)

[0101] def factual_accuracy_score(turn_text, knowledge_base):

[0102] extracted_info = extract_entities(turn_text) # Extract business entities

[0103] match_scores = []

[0104] for entity in extracted_info:

[0105] kb_match = query_knowledge_base(knowledge_base, entity)

[0106] if kb_match:

[0107] # Calculate semantic matching degree (using cosine similarity)

[0108] similarity = cosine_similarity(entity.embedding, kb_match.embedding)

[0109] match_scores.append(similarity)

[0110] return np.mean(match_scores) * 10 # Convert to a 0-10 score scale;

[0111] S2. The user-simulated intelligent agent engages in multi-turn dialogues with the large model under test according to the set dialogue scenario, obtaining multi-turn dialogue samples:

[0112] Example of interaction between USA and AUT (key rounds highlighted):

[0113] Round 1 (USA→AUT):

[0114] My credit card expires tomorrow. What is the minimum payment due?

[0115] (USA generates emergency queries based on anxiety)

[0116] Round 1 (AUT→USA):

[0117] "Hello, your minimum repayment amount is 10% of the current bill amount. You can check the specific amount by logging into your mobile banking app."

[0118] (AUT responds to standard business knowledge)

[0119] Round 2 (USA→AUT):

[0120] How is interest calculated if I only pay the minimum payment?

[0121] (USA raised derivative questions based on the white persona)

[0122] Round 2 (AUT→USA):

[0123] "Interest on the remaining outstanding amount will be charged at a daily rate of 0.05% from the date the transaction is recorded."

[0124] (AUT accurately responds to interest calculation rules)

[0125] The dialogue management module records the complete interaction sequence (including timestamps and role identifiers) and terminates the dialogue when it detects that the USA's objective intention has been achieved or the number of rounds has exceeded the limit (e.g., 10 rounds).

[0126] S3. The evaluation agent evaluates multi-turn dialogue samples based on the target evaluation requirements and outputs the evaluation results:

[0127] (1) Example of a multi-dimensional scoring Prompt (taking logical consistency assessment as an example):

[0128] You are a professional conversation evaluator. Please evaluate the logical consistency of responses according to the following rules:

[0129] - Check if the reply contradicts information already confirmed in the conversation history (such as the card number and date previously stated);

[0130] - Determine whether this response is consistent with your previous statements;

[0131] - Analyze whether the reasoning chain is reasonable (e.g., explain the concept first and then give suggestions).

[0132] Current conversation history:

[0133] [History]

[0134] This response: [AUT's response in this round]

[0135] Please output the following JSON format: {"score": 0-10, "reason": "detailed explanation"}

[0136] (2) Scoring aggregation logic

[0137] EA performs evaluations across all dimensions in parallel for each round of dialogue, with the dialogue-level score using a time-decay-weighted average:

[0138] Dialogue score = Σ(round score × γ^(total rounds - current round)) / Σ(decay weight)

[0139] (γ is the decay factor, usually taken as 0.9, emphasizing the impact of subsequent rounds on the overall performance)

[0140] Results analysis and optimization guidance:

[0141] This example generates a fragment of a structured evaluation report:

[0142] {

[0143] "dialogue_id": "CC_Repayment_001",

[0144] "turn_scores": {

[0145] "factual_accuracy": [8, 9],

[0146] "logical_coherence": [7, 8]

[0147] },

[0148] "overall_scores": {

[0149] "factual_accuracy": 8.5,

[0150] "task_completion": 9.0,

[0151] "compliance": 8.8

[0152] },

[0153] "key_issues": [

[0154] {

[0155] "turn": 1,

[0156] "dimension": "factual_accuracy",

[0157] "reason": "The reply did not specify the exact calculation method for the minimum repayment amount (which must include handling fees)."

[0158] "suggestion": "Add supplementary data: Include calculation examples for different repayment scenarios"

[0159] } ]

[0161] }

[0162] System extended application example:

[0163] Automated approval process for consumer protection compliance:

[0164] Configure the "Compliance Reviewer" role in the USA role library, whose dialogue script includes standard consumer protection questions:

[0165] How do you protect my personal information?

[0166] What if I disagree with the interest calculation?

[0167] EA automatically detects the compliance of AUT responses and generates an audit report in accordance with Article 29 of the Consumer Rights Protection Law and the company's privacy policy library.

[0168] This invention achieves efficient, comprehensive, and quantifiable automated evaluation of large language models for voice-based customer interaction scenarios through the collaborative work of three types of intelligent agents. This significantly improves evaluation efficiency and accuracy, providing a clear direction for model iterative optimization. Specific effects are as follows:

[0169] 1. Efficiency Improvement

[0170] (1) Comparison object: the present invention solution vs. traditional manual telephone dialing test.

[0171] (2) Key Indicators:

[0172] A: The time required for a single full regression test: covering 1000 test cases, manual testing would require 5 person-days, while this solution only requires 3-5 hours.

[0173] B: Concurrent dialogue generation capability: This solution supports the concurrent execution of multiple groups of dialogues (with adjustable concurrency), whereas manual execution can only be done sequentially.

[0174] C: Iteration response speed: the time from code submission to receiving the evaluation report (hours vs. days).

[0175] 2. Coverage and Scenarios Diversity

[0176] (1) Comparison object: the present invention solution (USA more personalized modeling) vs. static test set / simple rule simulator.

[0177] (2) Key Indicators:

[0178] A: Edge Cases Discovery Rate: This solution discovers 20% more anomalous scenarios (such as user interruptions, sudden mood changes, and ambiguous expressions) than traditional methods.

[0179] B: Personality trait coverage: By controlling the combination of user emotions, language style and business status in the large model through prompt words, the actual generated dialogue path diversity is more comprehensive than that of static test sets / simple rule simulators.

[0180] C: Number of long-tail issues detected: The number of logic vulnerabilities that were not covered by traditional testing, discovered in 10,000 simulated dialogues.

[0181] 3. Assessment accuracy and consistency

[0182] (1) Comparison object: The present invention scheme (EA agent) vs. human expert scoring

[0183] (2) Key Indicators:

[0184] A: Rating consistency (Kappa coefficient): The rating volatility of this solution when running the same dialogue multiple times is lower than the subjective volatility of human rating.

[0185] B: Golden Set fit: Selecting 100 dialogues with known good and bad results, this solution achieves an accuracy of over 95%, close to human standards, and is not affected by fatigue, while traditional rule-based automatic methods achieve around 70%.

[0186] 4. Cost savings

[0187] In the finance / telecom industry, testers with business knowledge typically earn between ¥600 and ¥1000 per person per day. Completing in-depth testing of 1000 complex multi-turn dialogues (each requiring at least 3-5 minutes of listening to recordings / reading logs) would require 5 person-days of work.

[0188] The main cost of this invention system is computing power consumption. With the advancement of model quantization technology and inference optimization, the inference cost per dialogue has been reduced to a few cents or even lower. Therefore, a computing power cost of ¥350 is a reasonable and conservative estimate for 10,000 dialogue rounds.

[0189] The embodiments described above merely illustrate specific implementation methods of this application, and while the descriptions are detailed and specific, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the technical solution of this application, and these modifications and improvements all fall within the scope of protection of this application.

Claims

1. An automatic evaluation method for the multi-turn dialogue capability of large-scale multi-agent collaborative models, characterized by: Includes the following steps: S1: Configure user role attributes for the user simulated intelligent agent based on the target assessment requirements; S2: The user simulates an intelligent agent to conduct multiple rounds of dialogue with the large model under test according to the set dialogue scenario, and obtains multi-round dialogue samples. S3: The evaluation agent evaluates multi-turn dialogue samples based on the target evaluation requirements and outputs the evaluation results.

2. The automatic evaluation method for multi-agent collaborative large-scale model multi-turn dialogue capability according to claim 1, characterized in that: User role attributes in S1 include user personality.

3. The automatic evaluation method for multi-agent collaborative large-scale model multi-turn dialogue capability according to claim 2, characterized in that: The user's personality includes emotional state, language style, knowledge background, and behavioral tendencies.

4. The automatic evaluation method and system for multi-agent collaborative large-scale model multi-turn dialogue capability according to claim 1, characterized in that: In S1, user role attributes are configured for the user-simulated intelligent agent based on the constructed USA role library. The steps for constructing the USA role library are as follows: S001: Based on historical dialogue logs, user survey data, and business expert experience, abstractly define typical user profiles; S002: Design behavioral characteristic parameters for typical user profiles; S003: Generate corresponding USA prompt words based on behavioral feature parameters using templated or parameterized methods.

5. The automatic evaluation method for multi-agent collaborative large-scale model multi-turn dialogue capability according to any one of claims 1-4, characterized in that: The evaluation results output in S3 include: a unique identifier for the dialogue, round scores for each evaluation dimension, overall dialogue scores for each evaluation dimension, total dialogue score, and detailed scoring reasons for round scores for each evaluation dimension. The detailed scoring reasons for round scores for each evaluation dimension include specific text snippets and explanations for each score point.

6. The automatic evaluation method for multi-agent collaborative large-scale model multi-turn dialogue capability according to claim 5, characterized in that: The evaluation dimensions include: naturalness, fluency, logical consistency, factual accuracy, appropriateness of emotional response, task completion, compliance, and initiative.

7. The automatic evaluation method for multi-agent collaborative large-scale model multi-turn dialogue capability according to any one of claims 1-4, characterized in that: The multi-turn dialogue sample includes the input and output of each round of interaction, and the input and output include timestamps, role identifiers, and dialogue content.

8. An automatic evaluation system for the multi-agent collaborative large-scale model multi-turn dialogue capability, characterized by: The method for automatically evaluating the multi-turn dialogue capability of large models with multi-agent collaboration as described in any one of claims 1-7 includes: The user simulation intelligent agent module is used to simulate real users and the tested large-scale model to conduct multi-turn dialogues and obtain multi-turn dialogue sample data. The evaluation agent module is used to perform multi-dimensional and fine-grained automated evaluation of multi-turn dialogue sample data based on a predefined, structured quantitative indicator system and scoring rules, and output the evaluation results. The dialogue management and storage module is used to maintain the dialogue state and store multi-turn dialogue sample data; The automated pipeline scheduling module is used to coordinate the dialogue generation of the user simulated agent module, the evaluation triggering of the evaluation agent module, and the result processing flow.

9. The automatic evaluation system for multi-agent collaborative large-scale model multi-turn dialogue capability according to claim 8, characterized in that: It also includes a results visualization and analysis module, used to display evaluation results and perform results analysis.

10. The automatic evaluation system for multi-agent collaborative large-scale model multi-turn dialogue capability according to claim 8, characterized in that: The user simulation agent module, evaluation agent module, dialogue management and storage module, and automated pipeline scheduling module connect and communicate through application programming interfaces or local service calls.

Images