A large model autonomous evaluation method, device and equipment based on multi-agent technology and a storage medium

Abstract

This invention provides a method, apparatus, device, and storage medium for autonomous evaluation of large models based on multi-agent technology. It generates an evaluation task set covering multiple scenarios and varying levels of difficulty by setting dimensional weights for basic and extended indicators. Based on this task set, a multi-agent evaluation system is constructed, comprising a questioner, an evaluator, and a reference agent. A parallel interaction mechanism is used to simultaneously initiate tasks to both the model to be evaluated and the reference agent. The evaluator agent compares the outputs of both agents according to multi-dimensional scoring criteria, and constructs a structured dataset from the task instances, output results, and scores. By analyzing this dataset, subsequent task sets are dynamically optimized, achieving a comprehensive and iterative evaluation of the large model's capabilities. This solves the problem that existing evaluation methods cannot fully reflect the true capability level of large models.

Description

Technical Field

[0001] This invention relates to the field of model performance evaluation, and in particular to a method, apparatus, device, and storage medium for autonomous evaluation of large models based on multi-agent technology. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models have demonstrated powerful capabilities in fields such as natural language processing, code generation, and reasoning and decision-making. However, how to effectively evaluate the overall performance of large models has become a key bottleneck restricting their further development and application.

[0003] Traditional evaluation methods rely on large, mixed evaluation datasets that often focus only on a single aspect of model performance, lacking systematic integration and failing to reflect the model's overall performance in real-world applications. The evaluation process requires comparing each item against pre-set standard answers and statistically analyzing metrics such as accuracy, resulting in low automation and efficiency. Furthermore, while autonomous evaluation techniques can achieve automated evaluation, they primarily rely on pre-set linguistic metrics such as BLEU and ROUGE, which struggle to capture subjective evaluation needs such as semantic rationality and creativity, thus failing to comprehensively reflect the true capabilities of large models.

[0004] In view of the above, this application is hereby submitted. Summary of the Invention

[0005] This invention discloses a method, apparatus, device, and storage medium for autonomous evaluation of large models based on multi-agent technology, aiming to solve the problem that existing evaluation methods cannot fully reflect the true capability level of large models.

[0006] The first embodiment of the present invention provides a method for autonomous evaluation of large models based on multi-agent technology, including:

[0007] The application scenarios and capability boundaries of the large model to be evaluated are obtained, evaluation tasks are extracted from different fields and difficulty gradients are configured, and basic and extended indicators are allocated by setting dimensional weights to generate a set of evaluation tasks containing task descriptions, difficulty levels and evaluation dimensions.

[0008] A multi-agent evaluation system is constructed based on the evaluation task set, including setting up a questioner agent, an evaluator agent, and a reference agent;

[0009] The evaluation is conducted through the multi-agent evaluation system. The questioner agent generates task instances from the task set and sends them to the large model to be evaluated and the reference agent simultaneously through the line interaction mechanism to obtain the output results of both. The evaluator agent performs multi-dimensional comparison and scoring of the output results of both based on the scoring criteria, which are formulated by the evaluator agent according to the basic indicators and extended indicators.

[0010] A structured dataset is constructed based on task instances, output results, and scoring data. The structured dataset includes task descriptions, outputs of the large model to be evaluated, outputs of the reference agent, and scores for each dimension. The structured dataset is used for analysis and to optimize the construction of the next round of task sets based on the analysis results.

[0011] Preferably, the evaluation task set adopts a hierarchical task generation strategy, wherein:

[0012] The first level is the domain division, including the domain of natural language processing, code generation, and reasoning and decision-making; the second level is the capability dimension, with four dimensions under each domain: comprehension, generation, reasoning, and innovation; the third level is the difficulty gradient, with each capability dimension including three difficulty levels: beginner, intermediate, and advanced; and task moments are generated by combining the three levels.

[0013] Preferably, the method further includes constructing an inter-task dependency graph of the evaluation task set, specifically:

[0014] Define the logical relationship between the preceding and subsequent tasks. When the score of the large model to be evaluated is lower than the threshold on the preceding task, the difficulty level of the subsequent task is automatically reduced.

[0015] Establish task clusters, in which semantically related tasks are organized into task clusters, and users verify the task rating results within the same task cluster.

[0016] Set task trigger conditions, where specific tasks are dynamically activated or skipped based on the real-time performance of the large model to be evaluated.

[0017] Preferably, the questioner agent receives the task metadata of the evaluation task set, generates task instances in batches according to the difficulty level, and refines the instances in the same batch according to the task description template and indicator weights. The task metadata includes task ID, description template and expected output format.

[0018] Preferably, the evaluator agent performs a multi-dimensional comparative scoring of the outputs of the two based on the scoring criteria, specifically as follows:

[0019] Output the large model to be evaluated and reference agent output Convert to vector representation and calculate cosine similarity. Converted into accuracy scores according to preset mapping rules. ;

[0020] Construct a logical dependency graph based on the two outputs and calculate the logical coherence score. ,in, The number of valid logical relations, The number of all possible logical relationships;

[0021] Compare the n-grams in the output of the large model to be evaluated with the corpus to calculate the innovation score. ,in, For the number of innovative n-grams, The total number of n-grams;

[0022] Calculate the overall score based on the task category and its dimensional weight. ,in, Weights for accuracy scores, The weights for logical coherence scores The weighting for innovation scores.

[0023] Preferably, after the questioner agent generates a task instance from the task set, the method further includes:

[0024] Establish a task priority queue and calculate the task priority according to the formula Priority=α×Importance+β×(1 / WaitTime)+γ×ResourceMatch, where Importance is the task importance, WaitTime is the waiting time, ResourceMatch is the resource matching degree, and α, β, and γ are the corresponding weights.

[0025] When the response time of an agent exceeds a preset threshold, some tasks in its task queue are automatically migrated to idle agents, and the execution time of different types of tasks is predicted based on historical data, reserving resources in advance for computationally intensive tasks.

[0026] The second embodiment of the present invention provides a large model autonomous evaluation device based on multi-agent technology, comprising:

[0027] The evaluation task set generation unit is used to obtain the application scenarios and capability boundaries of the large model to be evaluated, extract evaluation tasks from different fields and configure difficulty gradients, and generate an evaluation task set containing task descriptions, difficulty levels and evaluation dimensions by setting dimensional weights to allocate basic indicators and extended indicators.

[0028] An evaluation system construction unit is used to construct a multi-agent evaluation system based on the evaluation task set, including setting up a questioner agent, an evaluator agent, and a reference agent;

[0029] A multi-dimensional comparison and scoring unit is used for interactive evaluation through the multi-agent evaluation system. The questioner agent generates task instances from the task set and sends them to the large model to be evaluated and the reference agent simultaneously through the line interaction mechanism to obtain the output results of both. The evaluator agent performs multi-dimensional comparison and scoring of the output results of both based on the scoring criteria, which are formulated by the evaluator agent according to the basic indicators and extended indicators.

[0030] The structured dataset construction unit is used to construct a structured dataset based on task instances, output results, and scoring data. The structured dataset includes task descriptions, outputs of the large model to be evaluated, outputs of the reference agent, and scores for each dimension. The structured dataset is used for analysis and to optimize the construction of the next round of task sets based on the analysis results.

[0031] The third embodiment of the present invention provides a large model autonomous evaluation device based on multi-agent technology, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to realize the large model autonomous evaluation method based on multi-agent technology as described in any of the above embodiments.

[0032] The fourth embodiment of the present invention provides a computer-readable storage medium, characterized in that it stores a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to realize the large model autonomous evaluation method based on multi-agent technology as described in any of the above claims.

[0033] This invention provides a method, apparatus, device, and storage medium for autonomous evaluation of large models based on multi-agent technology. It generates an evaluation task set covering multiple scenarios and varying levels of difficulty by setting dimensional weights for basic and extended indicators. Based on this task set, a multi-agent evaluation system is constructed, comprising a questioner, an evaluator, and a reference agent. A parallel interaction mechanism is used to simultaneously initiate tasks to both the model to be evaluated and the reference agent. The evaluator agent compares the outputs of both agents according to multi-dimensional scoring criteria, and constructs a structured dataset from the task instances, output results, and scores. By analyzing this dataset, subsequent task sets are dynamically optimized, achieving a comprehensive and iterative evaluation of the large model's capabilities. This solves the problem that existing evaluation methods cannot fully reflect the true capability level of large models. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating a large-scale model autonomous evaluation method based on multi-agent technology provided in the first embodiment of the present invention.

[0035] Figure 2 This is a schematic diagram of a module of a large model autonomous evaluation device based on multi-agent technology provided in the second embodiment of the present invention. Detailed Implementation

[0036] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0037] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0038] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0039] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0040] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0041] Depending on the context, the word "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0042] The terms "first" and "second" used in the embodiments are merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" can be interchanged in a specific order or sequence where permissible. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein.

[0043] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0044] This invention discloses a method, apparatus, device, and storage medium for autonomous evaluation of large models based on multi-agent technology, aiming to solve the problem that existing evaluation methods cannot fully reflect the true capability level of large models.

[0045] Please see Figure 1 The first embodiment of the present invention provides a method for autonomous evaluation of large models based on multi-agent technology, which can be executed by an evaluation device (or system), specifically by one or more processors within the evaluation device, to at least implement the following steps:

[0046] S101: Obtain the application scenarios and capability boundaries of the large model to be evaluated, extract evaluation tasks from different fields and configure difficulty gradients, and generate a set of evaluation tasks containing task descriptions, difficulty levels and evaluation dimensions by setting dimensional weights to allocate basic and extended indicators.

[0047] In this embodiment, the evaluation device can be a desktop computer, laptop computer, workstation, or other terminal with data processing capabilities. The evaluation device can be equipped with a corresponding operating system and application software, and the functions required in this embodiment can be realized through the combination of the operating system and application software.

[0048] Specifically, in this embodiment, a hierarchical task generation strategy is established to construct the evaluation task set. First, by analyzing the technical specifications, API documentation, and practical application cases of the large model to be evaluated, its application scope in areas such as intelligent dialogue, document processing, code writing, and logical analysis is identified. Simultaneously, by testing the model's response capabilities to different types of input and the quality of its output, the upper and lower limits of its capabilities in areas such as language understanding depth, knowledge coverage breadth, and reasoning complexity are determined. Based on these analysis results, the system establishes a three-tiered task organization architecture. The first tier is divided according to the core domain of the technology application, categorizing all evaluation tasks into three main categories: natural language processing, code generation, and reasoning / decision-making.

[0049] In the field of Natural Language Processing, typical task types such as text summarization, multilingual translation, sentiment analysis, intelligent question answering, and content classification are extracted from real-world application scenarios. The code generation domain covers technical tasks such as function writing, algorithm implementation, code debugging, performance optimization, and architecture design in mainstream programming languages ​​like Python, Java, and JavaScript. The reasoning and decision-making domain includes complex tasks requiring deep thinking, such as mathematical computation, logical reasoning, causal analysis, risk assessment, and strategy formulation. The task extraction for each domain is based on real business needs and current technological trends, ensuring a high degree of alignment between the evaluation content and actual use cases.

[0050] The second tier uniformly sets four core dimensions for each domain: comprehension, generation, reasoning, and innovation. The comprehension dimension tests the model's ability to deeply analyze and accurately grasp input information through tasks such as complex context analysis, multi-layered meaning interpretation, and implicit information extraction. The generation dimension focuses on the quality of the model's content creation, comprehensively measuring the model's output level by evaluating indicators such as the fluency of language expression, the completeness of logical structure, and the rationality of information organization. The reasoning dimension designs comprehensive tasks requiring multi-step logical deduction, conditional relationship judgment, and causal chain analysis to test the model's reasoning process. The innovation dimension evaluates the model's creative thinking performance when facing novel problems through open-ended problem solving, generating diverse solutions, and producing creative content.

[0051] The third level configures three progressive difficulty levels—beginner, intermediate, and advanced—for each capability dimension. Beginner-level tasks typically involve the direct application of a single knowledge point or the simple execution of basic operations, such as basic fact-finding, simple format conversion, and direct information retrieval. These tasks require the model to possess basic understanding and response capabilities. Intermediate-level tasks require the model to integrate multiple related knowledge points or execute complex operations involving multiple steps, such as deep question answering requiring background knowledge support, program design involving the coordination of multiple modules, and logical analysis requiring moderate inference depth. These tasks test the model's knowledge integration and operational coordination abilities. Advanced-level tasks require the model to demonstrate a high level of comprehensive analysis and innovative problem-solving capabilities, such as solving complex problems requiring cross-domain knowledge integration, designing the complete architecture of large systems, and open-ended challenges requiring deep reasoning and innovative thinking. These tasks comprehensively test the model's higher-order cognitive abilities.

[0052] The allocation of dimensional weights employs a hybrid approach combining expert review and quantitative analysis. The system divides the evaluation index system into two levels: basic indicators and extended indicators. Basic indicators include key metrics that directly reflect the model's core functions, such as the accuracy of task completion, the completeness of the responses, and the relevance of the output results. These indicators dominate the weight allocation, typically accounting for approximately 65% ​​of the total weight. Extended indicators cover supplementary metrics that reflect the model's overall quality, such as the robustness of model performance, the ethical compliance of the output content, the security and stability of system operation, and the understandability of result interpretation. While these indicators do not directly affect the quality of task completion, they are significant for the model's practical application value and account for approximately 35% of the weight system.

[0053] Through a three-tiered combination mechanism, the system generates a complete identifier system for each specific assessment task instance, encompassing "domain category - capability dimension - difficulty level." Each task instance is accompanied by a detailed task description document, including structured information such as the task's application background, specific execution requirements, standard input / output format definitions, key assessment points, and estimated completion time. The difficulty level is precisely defined using quantifiable complexity parameters, such as objective indicators like text length, vocabulary complexity, grammatical structure hierarchy, number of reasoning steps, and number of knowledge domains involved, ensuring consistency and comparability in difficulty assessments.

[0054] The evaluation dimensions are designed to balance both automated measurement and intelligent judgment. Automated measurement dimensions include objective indicators that can be directly quantified by algorithms, such as text similarity calculation, syntax error detection, execution efficiency statistics, and formatting compliance checks. These dimensions provide a fast and accurate basic score. Intelligent judgment dimensions cover subjective indicators that require semantic understanding and reasoning analysis to evaluate, such as content creativity, logical rationality, practicality, and naturalness of expression. These dimensions better reflect the model's true performance level in complex application environments.

[0055] The entire evaluation task set possesses a dynamically evolving adaptive mechanism. The system continuously monitors the task execution performance and result feedback during the evaluation process. When it detects that a certain type of task lacks sufficient differentiation, the evaluation results are unconvincing, or they fail to effectively reflect differences in model capabilities, the system automatically initiates a task optimization and adjustment process. The optimization process includes various strategies such as introducing new task variants to increase the comprehensiveness of the evaluation, adjusting task parameter configurations to improve the rationality of the difficulty distribution, and revising the evaluation criterion definitions to enhance the accuracy of the judgment results. This ensures that the evaluation task set can always adapt to the rapid development of large-scale model technology and the continuous changes in application needs.

[0056] S102, Construct a multi-agent evaluation system based on the evaluation task set, including setting up a questioner agent, an evaluator agent, and a reference agent;

[0057] In this embodiment, based on the characteristics and requirements of the evaluation task set, three core intelligent agent roles are designed. Each agent has a unique functional positioning and working mechanism, forming a complete evaluation pipeline through mutual cooperation. The questioner agent, as the core component for task generation and distribution, is responsible for generating specific task instances based on templates and parameters in the evaluation task set, and simultaneously distributing these tasks to the large model to be evaluated and the reference agent. The questioner agent incorporates multiple task generation strategies, including a template-filling generation method. By replacing key variables in a preset template, such as specific text content, entity categories, and task parameters, it quickly generates a large number of task instances with the same structure but different content.

[0058] The questioner agent is also equipped with a randomized generation mechanism, enabling it to freely combine different text types, entity categories, and questioning methods to generate diverse and innovative task variations. For example, in natural language processing tasks, the system can combine social media text with a person entity recognition task to generate specific task instances such as "What person entities are included in the user post '@Zhang San, shall we go to the zoo tomorrow? @Li Si'?" Simultaneously, the questioner agent has parameterized difficulty control capabilities, allowing precise control of the difficulty level of the generated task by adjusting parameters such as text length, output format requirements, and knowledge complexity. More advanced multi-task fusion generation strategies can organically combine multiple sub-tasks such as text classification, sentiment analysis, and entity recognition to create complex evaluation tasks that require the integration of multiple capabilities.

[0059] The evaluator agent plays a crucial role in the entire evaluation process, performing judgments and scoring. It has established a comprehensive evaluation index system, covering quantitative evaluation standards across multiple dimensions, including accuracy, logicality, and innovativeness. For accuracy evaluation, the evaluator agent employs a vector space-based similarity calculation method. It converts the output of the model to be evaluated and the output of the reference agent into high-dimensional vector representations using word embedding technology, then calculates the cosine similarity between the two vectors to obtain an objective quantification of the matching result. When the similarity value reaches 0.9 or higher, the system considers the answer to be highly accurate and awards full marks; when the similarity is between 0.7 and 0.9, the system considers the answer basically accurate but with minor differences and awards a higher score; when the similarity is below 0.7, the system further analyzes the causes and nature of the differences and adjusts the score accordingly.

[0060] During the logical assessment process, the evaluator agent uses graph theory and knowledge reasoning techniques to construct a logical dependency graph of the answer content. The system treats each key information point in the answer as a node in the graph, and the logical relationships between these points, such as causal, progressive, and parallel relationships, as edges, forming a complete logical structure representation. By analyzing the connectivity, cyclicity, and completeness of the logical dependency graph, the system can accurately identify logical loopholes, reasoning errors, and structural defects in the answer. The ratio of the number of valid logical relationships to the total number of logical relationships directly reflects the logical quality of the answer, and the system calculates a quantitative logical score based on this ratio.

[0061] The novelty assessment employs a language model-based novelty detection algorithm. The evaluator agent maintains a reference corpus containing numerous common answer patterns and expressions, and analyzes the word combinations and expressive structures within the answers to be evaluated using n-gram technology. When an answer contains n-gram combinations that are rare or have never appeared in the reference corpus, the system considers that part to possess a certain degree of novelty. The proportion of novel n-grams to the total number of n-grams is used as a quantitative indicator of novelty; a higher proportion indicates a stronger degree of novelty in the answer.

[0062] The reference agent, acting as the provider of evaluation benchmarks, generates standardized reference answers for each evaluation task by integrating various high-quality information sources, including authoritative knowledge bases, professional literature, and domain expert experience. The reference agent's knowledge base covers authoritative content across multiple professional fields, including academic papers, technical specifications, industry standards, and best practice cases, ensuring the generated reference answers possess high accuracy and authority. The system employs multi-source information fusion technology to perform consistency checks, reliability assessments, and timeliness verifications on content from different information sources, selecting the highest quality information as the foundation for constructing the reference answers.

[0063] To adapt to the needs of different evaluation scenarios and objectives, the system sets up a differentiated capability parameter configuration mechanism for each agent. For evaluations simulating ordinary user scenarios, the system configures the corresponding agents with a limited knowledge base, mainly including common sense knowledge and everyday application information, and sets the reasoning depth to a relatively shallow level. The evaluation focuses on user experience-related indicators such as the ease of use, accessibility, and intuitiveness of the model output. In evaluation scenarios simulating domain experts, the system equips the agents with a deep professional knowledge system and enhanced logical reasoning capabilities, enabling them to perform multi-level complex analysis and judgment. The evaluation focus shifts to higher-order capability indicators such as technical accuracy, professional standardization, and innovative breakthroughs.

[0064] The collaboration mechanism between intelligent agents is implemented through standardized communication protocols and data exchange formats. After generating a task, the questioning agent simultaneously sends the task instance to both the large model to be evaluated and the reference agent, recording key information such as the sending timestamp, task identifier, and expected completion time. After completing the task, the large model to be evaluated and the reference agent return their respective outputs, along with metadata such as execution time, resource consumption, and processing status, to the evaluator agent. Upon receiving the outputs from both parties, the evaluator agent initiates a multi-dimensional evaluation process, independently calculating each evaluation indicator, then calculating a comprehensive score based on preset weighting ratios, and generating an evaluation report containing detailed analysis and explanations.

[0065] S103, interactive evaluation is conducted through the multi-agent evaluation system, wherein the questioner agent generates task instances from the task set and sends them to the large model to be evaluated and the reference agent simultaneously through the line interaction mechanism to obtain the output results of both; the evaluator agent performs multi-dimensional comparison and scoring of the output results of both based on the scoring criteria, wherein the scoring criteria are formulated by the evaluator agent based on the basic indicators and extended indicators.

[0066] The questioner agent first extracts evaluation tasks to be executed from the evaluation task set. In actual operation, the questioner agent maintains a task instance generation engine, which dynamically creates task instances according to different generation strategies. For example, when evaluating the entity recognition capability of a large model, the questioner agent may use a template filling strategy, replacing the placeholders in the template "Please identify the [entity category] entity in the text '[specific text]'" with actual content to generate a specific task instance such as "Please identify the location entity and time entity in the text 'City X Metro Line 14 will open in 2025'". For more complex evaluation needs, the questioner agent will adopt a multi-task fusion strategy, combining multiple sub-tasks such as text classification, sentiment analysis, and entity recognition into a composite task instance to comprehensively test the comprehensive processing capability of the large model.

[0067] After generating task instances, the questioner agent efficiently distributes tasks through a parallel interaction mechanism. At the core of this mechanism is an intelligent scheduler that monitors the status of each processing node in real time. Specifically, the scheduler maintains state information for each agent, including the number of tasks currently being processed (N_processing) and resource utilization (R_usage). When a new task instance needs to be distributed, the scheduler calculates the priority of each available agent according to the priority formula P = 1 / (N_processing + 1) × (1 - R_usage). This calculation method ensures that tasks are preferentially allocated to agents with lighter loads and sufficient resources, thereby achieving load balancing.

[0068] During task distribution, the questioning agent employs an asynchronous parallel approach, simultaneously sending the same task instance to both the large model to be evaluated and the reference agent. The system ensures reliable task transmission through a message queue mechanism, and each task is accompanied by a unique identifier and timestamp for subsequent result matching and performance analysis. Once the large model to be evaluated and the reference agent receive the task, they process it independently and generate output results. The system records detailed execution information, including task reception time, processing start time, processing end time, and complete output content.

[0069] When generating a reference answer, the reference agent utilizes its internally integrated authoritative knowledge base and professional literature resources. For example, for technical questions, the reference agent searches relevant academic papers and technical documents; for factual questions, it queries validated knowledge graphs. This mechanism ensures the high accuracy of the reference answer.

[0070] After both agents complete their tasks and return results, the evaluator agent begins a multi-dimensional comparative scoring process. Internally, the evaluator agent implements a complete scoring system, dynamically constructed based on fundamental and extended metrics defined in the evaluation task set. Fundamental metrics primarily focus on the model's core capabilities, such as accuracy and logical consistency; extended metrics focus on higher-order features such as robustness and ethical considerations.

[0071] In terms of accuracy assessment, the evaluator agent first takes the output of the large model to be evaluated. and the output of the reference agent Preprocessing is performed, including word segmentation, stop word removal, and stemming. Then, a pre-trained word embedding model is used to map the processed text into a high-dimensional vector space, obtaining the corresponding vector representation. Cosine similarity is then calculated. This is used to quantify the semantic similarity between two outputs. Based on practical experience, the system sets a mapping rule from similarity to accuracy score: when S_cos≥0.9, the accuracy score S_acc=10; when 0.7≤S_cos<0.9, S_acc=7; and so on, forming a continuous scoring system.

[0072] Logical reasoning evaluation employs more complex analytical methods. The evaluator agent first performs syntactic and semantic analysis on the output, identifying arguments, evidence, and reasoning relationships. Based on this information, a logical dependency graph is constructed, where nodes represent key information points, and edges represent logical relationships between these points, such as causal, conditional, and parallel relationships. Through graph analysis algorithms, the system can identify the completeness and rationality of logical chains. For example, in mathematical reasoning tasks, the evaluator checks whether each step of the deduction has sufficient evidence and whether there are logical jumps or circular arguments. Finally, the number of valid logical relationships is calculated. Number of all possible logical relationships The proportion is used to derive a logical score. .

[0073] The innovation assessment focuses on identifying novel content in the output. The evaluator agent maintains a large-scale corpus of common expressions, covering standard answers and conventional solutions across various domains. During evaluation, the system decomposes the output of the large model to be evaluated into n-grams of different lengths (e.g., 3-grams, 4-grams) and then checks the frequency of these n-grams in the corpus. N-grams that do not appear in the corpus or have extremely low frequencies are considered innovative by the system. The number of innovative n-grams is counted. With the total number of n-grams The proportion used to calculate the innovation score This method can effectively identify whether a large model can generate novel expressions or provide unique solutions to problems.

[0074] After completing independent scoring for each dimension, the evaluator agent needs to calculate a comprehensive score. The key here is the dynamic configuration of the weights. The system does not use a fixed weight allocation, but rather adjusts it flexibly according to the task type and evaluation objectives. For example, for factual question-answering tasks, accuracy is weighted... The weighting will be set relatively high (e.g., 0.6), while the weighting for innovation will be higher. The weighting is relatively low (e.g., 0.1); for creative writing tasks, the weighting for creativity is increased while the weighting for accuracy is decreased. This is achieved through a weighted summation formula. Calculate the final overall score.

[0075] S104. Construct a structured dataset based on task instances, output results, and scoring data. The structured dataset includes task descriptions, outputs of the large model to be evaluated, outputs of the reference agent, and scores for each dimension. The structured dataset is used for analysis and to optimize the construction of the next round of task sets based on the analysis results.

[0076] It should be noted that once the evaluator agent completes its scoring of a task, the system immediately initiates the data integration process. Each evaluation record is organized into a unified data structure, containing metadata such as task identifier, task description, task category, difficulty level, and generation timestamp. The task description not only records the specific question text generated by the questioner agent but also retains information about the task's generation strategy, such as whether it was generated through template filling, random combination, or multi-task fusion.

[0077] The output data of the large model to be evaluated is stored in complete text format, while the system automatically extracts key feature information. For example, for code generation tasks, the system parses the code structure and counts the number of lines of code, the characteristics of the programming language used, and the number of function calls; for natural language processing tasks, it extracts features such as text length, sentence complexity, and the number of technical terms used. These structured features are stored together with the original output text to form a multi-layered data representation.

[0078] The output of the reference agent undergoes a similar processing flow. Because the output of the reference agent typically possesses high standardization and accuracy, the system also additionally annotates the knowledge sources it references, such as specific documents, knowledge base entries, or expert rules. This annotation mechanism allows subsequent analysis to trace the basis of the reference answer.

[0079] The constructed structured dataset then proceeds to the analysis phase. The system first performs descriptive statistical analysis, calculating basic statistics such as average score, standard deviation, and score distribution for each task. By comparing the score distributions across different task categories and difficulty levels, the system can identify the capability characteristics of the large model being evaluated. For example, if it finds that the large model's average score on natural language processing tasks is significantly higher than on code generation tasks, while its score drops sharply at higher difficulty levels in reasoning and decision-making tasks, this provides direction for task set optimization.

[0080] In-depth analysis employs machine learning methods to uncover hidden patterns. The system uses clustering algorithms to group all evaluation records, identifying task groups with similar characteristics. For example, it might find that a certain type of mathematical problem involving multi-step reasoning generally scores low, even though these problems share similar language patterns in their task descriptions. Through association rule mining, the system can also discover correlations between different evaluation dimensions. For instance, outputs with high innovation scores often also demonstrate good logical reasoning, suggesting that such intrinsic connections between dimensions can be considered when designing tasks.

[0081] Optimization suggestions are fed back to the evaluation task building module in the form of structured improvement prompts. These prompts contain specific instructions, such as "add medium-complexity logical reasoning tasks involving 3-5 reasoning steps," "increase the weight of syntax error detection subtasks in code debugging tasks," and "add more text examples containing irony and metaphor to sentiment analysis tasks." Upon receiving these improvement prompts, the evaluation task building module adjusts its task generation strategy, updates its task template library, and modifies its difficulty grading criteria accordingly.

[0082] In one possible implementation of this invention, the system first establishes a dynamic task priority queue to manage all pending task instances. When each task enters the queue, the system assigns it a priority score according to the priority calculation formula: Priority = α × Importance + β × (1 / WaitTime) + γ × ResourceMatch. In actual operation, the task importance is determined by both the task type and the evaluation objective. For example, the importance of a core capability evaluation task is set to 1.0, an extended capability evaluation task to 0.7, and a boundary testing task to 0.5. The waiting time (WaitTime) is updated in real-time in seconds. As the time a task spends in the queue increases, the contribution of the 1 / WaitTime term gradually increases, ensuring that tasks with long waiting times are not indefinitely delayed. The resource matching degree (ResourceMatch) reflects the degree of matching between the task requirements and currently available resources. When the computing resources and memory capacity required by the task are highly consistent with the current system state, this value is close to 1.0; otherwise, it decreases.

[0083] The settings of the weight parameters α, β, and γ reflect the system's scheduling strategy. In the default configuration, α is set to 0.5 to ensure that important tasks are prioritized, β is set to 0.3 to prevent task starvation, and γ is set to 0.2 to improve resource utilization efficiency. These weights are dynamically adjusted according to the system's operating status. For example, when a large backlog of tasks is detected, the system will automatically increase the β value to speed up the processing of older tasks; when system resources are sufficient, the γ value will be increased to optimize resource allocation.

[0084] The system continuously monitors the operational status of each agent, especially response time metrics. When the response time of an agent in processing a task exceeds a preset threshold (e.g., 1.5 times the average response time), the system determines that the agent may be under high load or experiencing performance degradation. At this point, the task migration mechanism is automatically triggered. The system first assesses the migration cost of each task in the agent's task queue, prioritizing tasks that have not yet started processing or have a small processing progress. The selection of migration targets is based on the current load and historical performance data of other agents, ensuring that tasks are migrated to the most suitable execution environment.

[0085] To further improve scheduling efficiency, the system implements a task execution time prediction function based on historical data. The system maintains a task execution history database, recording the actual execution time of different types of tasks under different configurations. Through time series analysis and regression models, the system can predict the execution time of new tasks. For example, for tasks containing complex inference chains, the system predicts a longer processing time based on characteristics such as the number of inference steps and the knowledge domain involved. Based on these predictions, the system pre-allocates sufficient resources for computationally intensive tasks, including pre-allocating GPU memory, pre-loading necessary model parameters, and pre-building computational graphs, thereby reducing task startup latency.

[0086] In this implementation, to enhance the structural correlation and responsiveness between evaluation tasks, a task dependency graph is constructed for the evaluation task set. Specifically, the system first defines the logical sequence of tasks based on task design requirements, designating some tasks as prerequisites and others as successors. When the score of the large model to be evaluated on a prerequisite task is lower than a preset threshold, the system automatically adjusts the difficulty level of the corresponding successor task to reduce the evaluation intensity and avoid invalid evaluations. To further improve the semantic coherence and verification capability within the task set, the system also organizes tasks with similar semantics and capability requirements into task clusters and introduces a cross-validation mechanism for task scores within the cluster during execution, so that the evaluation result of a certain task can serve as a reference for the evaluation of other related tasks. In addition, to achieve a more flexible task control mechanism, the system also sets task triggering conditions based on the real-time performance of the large model to be evaluated in the current task. For example, advanced tasks are activated only when the score of a certain type of task is higher than a specific threshold, or specific task paths are skipped when low-quality output occurs, thereby dynamically optimizing the evaluation process and saving computing resources.

[0087] Please see Figure 2 The second embodiment of the present invention provides a large model autonomous evaluation device based on multi-agent technology, comprising:

[0088] The evaluation task set generation unit 201 is used to obtain the application scenarios and capability boundaries of the large model to be evaluated, extract evaluation tasks from different fields and configure difficulty gradients, and generate an evaluation task set containing task descriptions, difficulty levels and evaluation dimensions by setting dimensional weights to allocate basic indicators and extended indicators.

[0089] The evaluation system construction unit 202 is used to construct a multi-agent evaluation system based on the evaluation task set, including setting up a questioner agent, an evaluator agent, and a reference agent;

[0090] The multi-dimensional comparison and scoring unit 203 is used for interactive evaluation through the multi-agent evaluation system. The questioner agent generates task instances from the task set and sends them to the large model to be evaluated and the reference agent simultaneously through the line interaction mechanism to obtain the output results of both. The evaluator agent performs multi-dimensional comparison and scoring of the output results of both based on the scoring criteria, which are formulated by the evaluator agent according to the basic indicators and extended indicators.

[0091] The structured dataset construction unit 204 is used to construct a structured dataset based on task instances, output results, and scoring data. The structured dataset includes task descriptions, outputs of the large model to be evaluated, outputs of the reference agent, and scores for each dimension. The structured dataset is used for analysis and optimization of the construction of the next round of task sets based on the analysis results.

[0092] The third embodiment of the present invention provides a large model autonomous evaluation device based on multi-agent technology, including a memory and a processor. The memory stores a computer program, which can be executed by the processor to realize the large model autonomous evaluation method based on multi-agent technology as described in any of the above embodiments.

[0093] The fourth embodiment of the present invention provides a computer-readable storage medium, characterized in that it stores a computer program, which can be executed by the processor of the device where the computer-readable storage medium is located, to realize the large model autonomous evaluation method based on multi-agent technology as described in any of the above claims.

[0094] This invention provides a method, apparatus, device, and storage medium for autonomous evaluation of large models based on multi-agent technology. It generates an evaluation task set covering multiple scenarios and varying levels of difficulty by setting dimensional weights for basic and extended indicators. Based on this task set, a multi-agent evaluation system is constructed, comprising a questioner, an evaluator, and a reference agent. A parallel interaction mechanism is used to simultaneously initiate tasks to both the model to be evaluated and the reference agent. The evaluator agent compares the outputs of both agents according to multi-dimensional scoring criteria, and constructs a structured dataset from the task instances, output results, and scores. By analyzing this dataset, subsequent task sets are dynamically optimized, achieving a comprehensive and iterative evaluation of the large model's capabilities. This solves the problem that existing evaluation methods cannot fully reflect the true capability level of large models.

[0095] Exemplary examples show that the computer program described in the third and fourth embodiments of the present invention can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in implementing a large-scale autonomous evaluation device based on multi-agent technology. For example, the apparatus described in the second embodiment of the present invention.

[0096] The processor referred to can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. This processor is the control center of the large-scale model autonomous evaluation method based on multi-agent technology, connecting various parts of the method through various interfaces and lines.

[0097] The memory can be used to store the computer program and / or modules. The processor, by running or executing the computer program and / or modules stored in the memory, and by calling the data stored in the memory, realizes various functions of a large-scale autonomous evaluation method based on multi-agent technology. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, text conversion function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, text message data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital card (SD card), flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0098] If the implemented module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0099] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0100] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for autonomous evaluation of large models based on multi-agent technology, characterized in that, include: The application scenarios and capability boundaries of the large model to be evaluated are obtained, evaluation tasks are extracted from different fields and difficulty gradients are configured, and basic and extended indicators are allocated by setting dimensional weights to generate a set of evaluation tasks containing task descriptions, difficulty levels and evaluation dimensions. A multi-agent evaluation system is constructed based on the evaluation task set, including setting up a questioner agent, an evaluator agent, and a reference agent. The questioner agent receives the task metadata of the evaluation task set, generates task instances in batches according to the difficulty level, and refines the instances in the same batch according to the task description template and indicator weight. The task metadata includes task ID, description template, and expected output format. The evaluation is conducted interactively through the multi-agent evaluation system. The questioner agent generates task instances from the task set and simultaneously sends them to the large model to be evaluated and the reference agent via a line-interaction mechanism, obtaining the output results from both. The evaluator agent then performs a multi-dimensional comparison and scoring of the output results based on a scoring standard. Specifically, the large model to be evaluated outputs... and reference agent output Convert to vector representation and calculate cosine similarity. Converted into accuracy scores according to preset mapping rules. Construct a logical dependency graph based on the two outputs and calculate the logical coherence score. ,in, The number of valid logical relations, The number of all logical relations is given; the innovativeness score is calculated by comparing the n-grams in the output of the large model to be evaluated with those in the corpus. ,in, For the number of innovative n-grams, The total number of n-grams; calculate the comprehensive score based on the task category's weight in the dimension. ,in, Weights for accuracy scores, The weights for logical coherence scores The scoring criteria, which are weighted for the innovation score, are formulated by the evaluator agent based on the basic and extended indicators. A structured dataset is constructed based on task instances, output results, and scoring data. The structured dataset includes task descriptions, outputs of the large model to be evaluated, outputs of the reference agent, and scores for each dimension. The structured dataset is used for analysis and to optimize the construction of the next round of task sets based on the analysis results.

2. The method for autonomous evaluation of large models based on multi-agent technology according to claim 1, characterized in that, The evaluation task set adopts a hierarchical task generation strategy, wherein: The first level is the domain division, including the domain of natural language processing, code generation, and reasoning and decision-making; the second level is the capability dimension, with four dimensions under each domain: comprehension, generation, reasoning, and innovation; the third level is the difficulty gradient, with each capability dimension including three difficulty levels: beginner, intermediate, and advanced; and task moments are generated by combining the three levels.

3. The method for autonomous evaluation of large models based on multi-agent technology according to claim 1, characterized in that, It also includes constructing an inter-task dependency graph of the evaluation task set, specifically: Define the logical relationship between the preceding and subsequent tasks. When the score of the large model to be evaluated is lower than the threshold on the preceding task, the difficulty level of the subsequent task is automatically reduced. Establish task clusters, in which semantically related tasks are organized into task clusters, and users verify the task rating results within the same task cluster. Set task trigger conditions, where tasks are dynamically activated or skipped based on the real-time performance of the large model to be evaluated.

4. The method for autonomous evaluation of large models based on multi-agent technology according to claim 1, characterized in that, After the questioner agent generates a task instance from the task set, the process further includes: Establish a task priority queue and calculate the task priority according to the formula Priority=α×Importance+β×(1 / WaitTime)+γ×ResourceMatch, where Importance is the task importance, WaitTime is the waiting time, ResourceMatch is the resource matching degree, and α, β, and γ are the corresponding weights. When the response time of an agent exceeds a preset threshold, some tasks in its task queue are automatically migrated to idle agents, and the execution time of different types of tasks is predicted based on historical data, reserving resources in advance for computationally intensive tasks.

5. A large-scale autonomous evaluation device based on multi-agent technology, characterized in that, include: The evaluation task set generation unit is used to obtain the application scenarios and capability boundaries of the large model to be evaluated, extract evaluation tasks from different fields and configure difficulty gradients, and generate an evaluation task set containing task descriptions, difficulty levels and evaluation dimensions by setting dimensional weights to allocate basic indicators and extended indicators. An evaluation system construction unit is used to construct a multi-agent evaluation system based on the evaluation task set. It includes setting up a questioner agent, an evaluator agent, and a reference agent. The questioner agent receives task metadata of the evaluation task set and generates task instances in batches according to the difficulty level. It also refines the instances in the same batch according to the task description template and indicator weight. The task metadata includes task ID, description template, and expected output format. A multi-dimensional comparison and scoring unit is used for interactive evaluation through the multi-agent evaluation system. The questioner agent generates task instances from the task set and simultaneously sends them to the large model to be evaluated and the reference agent via a line-interaction mechanism, obtaining the output results of both. The evaluator agent performs a multi-dimensional comparison and scoring of the output results of both based on the scoring criteria. Specifically, it compares the output of the large model to be evaluated with the reference agent's output... and reference agent output Convert to vector representation and calculate cosine similarity. Converted into accuracy scores according to preset mapping rules. Construct a logical dependency graph based on the two outputs and calculate the logical coherence score. ,in, The number of valid logical relations, The number of all logical relations is given; the innovativeness score is calculated by comparing the n-grams in the output of the large model to be evaluated with those in the corpus. ,in, For the number of innovative n-grams, The total number of n-grams; calculate the comprehensive score based on the task category's weight in the dimension. ,in, Weights for accuracy scores, The weights for logical coherence scores The scoring criteria, which are weighted for the innovation score, are formulated by the evaluator agent based on the basic and extended indicators. The structured dataset construction unit is used to construct a structured dataset based on task instances, output results, and scoring data. The structured dataset includes task descriptions, outputs of the large model to be evaluated, outputs of the reference agent, and scores for each dimension. The structured dataset is used for analysis and to optimize the construction of the next round of task sets based on the analysis results.

6. A large-scale autonomous evaluation device based on multi-agent technology, characterized in that, It includes a memory and a processor, wherein the memory stores a computer program that can be executed by the processor to implement a large model autonomous evaluation method based on multi-agent technology as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The device contains a computer program that can be executed by a processor of the device where the computer-readable storage medium is located, to implement the large model autonomous evaluation method based on multi-agent technology as described in any one of claims 1 to 4.

Images