Large model autonomous evaluation method, device and equipment based on multi-agent technology and storage medium

Abstract

The invention provides a large model autonomous evaluation method, device and equipment based on a multi-agent technology, and a storage medium. The method comprises the following steps: generating an evaluation task set covering multiple scenes and grading multiple difficulties by setting dimension weights of basic indexes and extension indexes; a multi-agent evaluation system of a questioner, an evaluator and a reference agent is constructed based on the task set, a parallel interaction mechanism is adopted to simultaneously initiate tasks to a to-be-evaluated model and the reference agent, the evaluator agent compares the to-be-evaluated model and the reference agent according to a multi-dimensional scoring standard and outputs the two agents, and task instances, output results and scores are constructed into a structured data set; the subsequent task set is dynamically optimized through analysis of the data set, comprehensive and iterative evaluation of the capacity of the large model is achieved, and the problem that an existing evaluation mode cannot comprehensively reflect the real capacity level of the large model is solved.

Description

Technical Field

[0001] The present invention relates to the field of model performance evaluation, and in particular to a large model autonomous evaluation method, device, equipment and storage medium based on multi-agent technology. Background Art

[0002] With the rapid development of artificial intelligence (AI), large-scale language models have demonstrated powerful capabilities in natural language processing, code generation, reasoning, and decision-making. However, effectively evaluating the comprehensive performance of large models has become a key bottleneck restricting their further development and application.

[0003] Traditional evaluation methods rely on large, mixed evaluation datasets. These datasets often focus on a single aspect of model performance, lacking systematic integration and failing to reflect the model's overall performance in real-world application scenarios. The evaluation process requires item-by-item comparisons with pre-set standard answers and statistical accuracy, resulting in low automation and inefficiency. Furthermore, while autonomous evaluation technologies can achieve automated evaluation, they primarily rely on pre-set linguistic metrics like BLEU and ROUGE. These metrics struggle to capture subjective dimensions like semantic plausibility and creativity, and therefore fail to fully reflect the true capabilities of large models.

[0004] In view of this, this application is filed. Summary of the Invention

[0005] The present invention discloses a large-scale model autonomous evaluation method, device, equipment and storage medium based on multi-agent technology, aiming to solve the problem that existing evaluation methods cannot fully reflect the real ability level of large models.

[0006] The first embodiment of the present invention provides a large model autonomous evaluation method based on multi-agent technology, comprising: Obtain the application scenarios and capability boundaries of the large model to be evaluated, extract evaluation tasks from different fields and configure difficulty gradients, and assign basic indicators and extended indicators by setting dimension weights to generate an evaluation task set that includes task descriptions, difficulty levels, and evaluation dimensions; Constructing a multi-agent evaluation system based on the evaluation task set, including setting a questioner agent, an evaluator agent and a reference agent; Interactive evaluation is performed through the multi-agent evaluation system, wherein the questioner agent generates a task instance from the task set and simultaneously sends it to the large model to be evaluated and the reference agent through a row interaction mechanism to obtain output results of both. The evaluator agent performs a multi-dimensional comparative scoring of the output results of the two based on a scoring standard, wherein the scoring standard is formulated by the evaluator agent based on the basic indicators and extended indicators; A structured dataset is constructed based on task instances, output results, and scoring data. The structured dataset includes task descriptions, outputs of the large model to be evaluated, outputs of reference agents, and scores for each dimension. The structured dataset is used to analyze and optimize the construction of the next round of task sets based on the analysis results.

[0007] Preferably, the evaluation task set adopts a hierarchical task generation strategy, wherein: The first level is the domain division, including the natural language processing domain, code generation domain and reasoning and decision-making domain; the second level is the ability dimension, with four dimensions of understanding ability, generation ability, reasoning ability and innovation ability under each domain; the third level is the difficulty gradient, with each ability dimension containing three difficulty levels: elementary, intermediate and advanced; and the task matrix is ​​generated through the combination of the three levels.

[0008] Preferably, it also includes constructing a task dependency graph of the evaluation task set, specifically: Define the logical relationship between the preceding and subsequent tasks. When the score of the large model to be evaluated on the preceding task is lower than the threshold, the difficulty level of the subsequent task is automatically reduced. Establish task clusters, where semantically related tasks are organized into task clusters, and task scoring results within the same task cluster are mutually verified by users; Set task trigger conditions, where specific tasks are dynamically activated or skipped based on the real-time performance of the large model to be evaluated.

[0009] Preferably, the questioner agent receives task metadata of a set of evaluation tasks, generates task instances in batches in sequence according to difficulty levels, and refines the instances in the same batch according to task description templates and indicator weights, wherein the task metadata includes task ID, description template and expected output format.

[0010] Preferably, the evaluator agent performs a multi-dimensional comparative scoring of the two output results based on the scoring criteria, specifically: Output the large model to be evaluated and the reference agent output Convert to vector representation and calculate cosine similarity , converted into accuracy scores according to preset mapping rules ; Build a logical dependency graph based on the two outputs and calculate the logical coherence score ,in, is the number of valid logical relations, is the number of all possible logical relations; Compare the n-grams in the output of the large model to be evaluated with the corpus and calculate the innovation score ,in, is the number of innovative n-grams, is the total number of n-grams; Calculate the comprehensive score based on the dimension weight of the task category ,in, is the weight of the accuracy score, is the weight of the logical coherence score, is the weight of the innovation score.

[0011] Preferably, after the questioner agent generates a task instance from the task set, the method further comprises: Establish a task priority queue and calculate the task priority according to the formula Priority = α × Importance + β × (1 / WaitTime) + γ × ResourceMatch, where Importance is the task importance, WaitTime is the waiting time, ResourceMatch is the resource matching degree, and α, β, and γ are the corresponding weights; When it is detected that the response time of an agent exceeds the preset threshold, some tasks in its task queue are automatically migrated to idle agents. Based on historical data, the execution time of different types of tasks is predicted, and resources are reserved in advance for computationally intensive tasks.

[0012] A second embodiment of the present invention provides a large-scale model autonomous evaluation device based on multi-agent technology, comprising: The evaluation task set generation unit is used to obtain the application scenarios and capability boundaries of the large model to be evaluated, extract evaluation tasks from different fields and configure difficulty gradients, and allocate basic indicators and extended indicators by setting dimension weights to generate an evaluation task set that includes task descriptions, difficulty levels, and evaluation dimensions; An evaluation system construction unit, configured to construct a multi-agent evaluation system based on the evaluation task set, including setting a questioner agent, an evaluator agent, and a reference agent; a multi-dimensional comparative scoring unit, configured to interactively evaluate through the multi-agent evaluation system, wherein the questioner agent generates task instances from the task set and simultaneously sends them to the large model to be evaluated and the reference agent through a row interaction mechanism, obtaining output results of both, and the evaluator agent performs a multi-dimensional comparative scoring of the output results of the two based on a scoring standard formulated by the evaluator agent based on the basic indicators and extended indicators; A structured dataset construction unit is used to construct a structured dataset based on task instances, output results and scoring data, wherein the structured dataset includes task description, output of the large model to be evaluated, output of the reference intelligent agent, and scores of each dimension. The structured dataset is used to analyze and optimize the construction of the next round of task sets based on the analysis results.

[0013] The third embodiment of the present invention provides a large-model autonomous evaluation device based on multi-agent technology, including a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a large-model autonomous evaluation method based on multi-agent technology as described in any one of the above.

[0014] The fourth embodiment of the present invention provides a computer-readable storage medium, characterized in that it stores a computer program, and the computer program can be executed by a processor of the device where the computer-readable storage medium is located to implement a large-model autonomous evaluation method based on multi-agent technology as described in any one of the above items.

[0015] Based on the large-scale autonomous evaluation method, device, equipment and storage medium based on multi-agent technology provided by the present invention, an evaluation task set covering multiple scenarios and with multiple levels of difficulty is generated by setting the dimensional weights of basic indicators and extended indicators; a multi-agent evaluation system of questioners, evaluators and reference agents is constructed based on the task set, and a parallel interaction mechanism is adopted to simultaneously initiate tasks to the model to be evaluated and the reference agent. The evaluator agent compares the outputs of the two according to multi-dimensional scoring criteria, and constructs the task instances, output results and scores into a structured data set; through the analysis of the data set, the subsequent task sets are dynamically optimized to achieve a comprehensive and iterative evaluation of the large-scale model capabilities, thereby solving the problem that the existing evaluation method cannot fully reflect the true capability level of the large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 1 is a flowchart of a large model autonomous evaluation method based on multi-agent technology provided by the first embodiment of the present invention; Figure 2 This is a module diagram of a large-model autonomous evaluation device based on multi-agent technology provided by the second embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0018] In order to better understand the technical solution of the present invention, the embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0019] It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0020] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The singular forms "a", "an", "the" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise.

[0021] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0022] The word "if," as used herein, may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0023] The "first" and "second" mentioned in the embodiments are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that the specific order or precedence of "first" and "second" can be interchanged where appropriate. It should be understood that the objects distinguished by "first" and "second" can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein.

[0024] The specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0025] The present invention discloses a large-scale model autonomous evaluation method, device, equipment and storage medium based on multi-agent technology, aiming to solve the problem that existing evaluation methods cannot fully reflect the real ability level of large models.

[0026] See also Figure 1 The first embodiment of the present invention provides a large-model autonomous evaluation method based on multi-agent technology, which can be executed by an evaluation device (or system), and in particular, by one or more processors within the evaluation device, to implement at least the following steps: S101: Obtain the application scenarios and capability boundaries of the large model to be evaluated, extract evaluation tasks from different fields and configure difficulty gradients, and assign basic indicators and extended indicators by setting dimension weights to generate an evaluation task set that includes task descriptions, difficulty levels, and evaluation dimensions. In this embodiment, the evaluation device may be a terminal with data processing capabilities such as a desktop computer, laptop computer, workstation, etc. The evaluation device may be installed with a corresponding operating system and application software, and the functions required by this embodiment may be achieved through the combination of the operating system and application software.

[0027] Specifically, in this embodiment, the construction of the evaluation task set is achieved by establishing a hierarchical task generation strategy. First, by analyzing the technical specifications, API documents and actual application cases of the large model to be evaluated, its application scope in intelligent dialogue, document processing, code writing, logical analysis, etc. is identified. At the same time, by testing the model's responsiveness and output quality to different types of inputs, its upper and lower limits in terms of language understanding depth, knowledge coverage breadth, reasoning complexity, etc. are determined. Based on these analysis results, the system establishes a three-level task organization structure. The first level is divided according to the core areas of technical application, and all evaluation tasks are classified into three main categories: natural language processing, code generation, and reasoning and decision-making.

[0028] In the field of natural language processing, typical tasks such as text summarization, multilingual translation, sentiment analysis, intelligent question-answering, and content classification are extracted from real-world application scenarios. The code generation domain covers technical tasks such as function writing, algorithm implementation, code debugging, performance optimization, and architecture design in mainstream programming languages ​​like Python, Java, and JavaScript. The reasoning and decision-making domain encompasses complex tasks requiring deep thinking, such as mathematical calculations, logical reasoning, causal analysis, risk assessment, and strategy formulation. Task extraction in each domain is based on real business needs and hot technology applications, ensuring a close match between the evaluation content and actual use cases.

[0029] The second level uniformly sets four core dimensions in each field: comprehension ability, generation ability, reasoning ability, and innovation ability. The comprehension ability dimension tests the model's ability to deeply analyze and accurately grasp input information by designing tasks such as complex context analysis, multi-layer meaning interpretation, and implicit information extraction. The generation ability dimension focuses on the content creation quality of the model, and comprehensively measures the output level of the model by evaluating indicators such as the fluency of language expression, the integrity of logical structure, and the rationality of information organization. The reasoning ability dimension designs comprehensive tasks that require multi-step logical deduction, conditional relationship judgment, and causal chain analysis to test the model's thinking and reasoning process. The innovation ability dimension evaluates the model's creative thinking performance when facing novel problems through methods such as open-ended problem solving, diversified solution generation, and creative content output.

[0030] The third level configures three progressive difficulty levels for each capability dimension: elementary, intermediate, and advanced. Elementary difficulty tasks usually involve the direct application of a single knowledge point or the simple execution of basic operations, such as basic fact query, simple format conversion, direct information retrieval, etc. These tasks require the model to have basic understanding and response capabilities. Intermediate difficulty tasks require the model to integrate multiple related knowledge points or perform complex operations involving multiple steps, such as in-depth question answering that requires background knowledge support, program design involving coordination of multiple modules, and logical analysis that requires medium reasoning depth, etc., which test the model's knowledge integration and operational coordination capabilities. Advanced difficulty tasks require the model to demonstrate a high level of comprehensive analysis and innovative problem-solving capabilities, such as complex problem solving that requires cross-domain knowledge integration, complete architectural design of large systems, and open challenges that require deep reasoning and innovative thinking, etc., which comprehensively test the model's high-order cognitive capabilities.

[0031] The configuration of dimension weights uses a hybrid approach combining expert review with quantitative analysis. The system divides the evaluation index system into two levels: basic indicators and extended indicators. Basic indicators include key indicators that directly reflect the core functions of the model, such as the accuracy of task completion, the completeness of the answer content, and the relevance of the output results. These indicators play a dominant role in the weight distribution and are generally assigned approximately 65% ​​of the total weight. Extended indicators cover supplementary indicators that reflect the overall quality of the model, such as the robustness of model performance, the ethical compliance of output content, the security and stability of system operation, and the understandability of result interpretation. Although these indicators do not directly affect the quality of task completion, they are of great significance to the practical application value of the model and account for approximately 35% of the weight system.

[0032] Through a three-tiered mechanism, the system generates a comprehensive identification system for each specific assessment task instance, encompassing "domain category - capability dimension - difficulty level." Each task instance is accompanied by a detailed task description document, including structured information such as the task's application context, specific execution requirements, standard input and output format definitions, key assessment points, and an estimated completion time. The difficulty level is precisely defined using quantifiable complexity parameters, such as text length, vocabulary complexity, grammatical structure level, number of reasoning steps, and the number of knowledge areas involved, ensuring consistency and comparability in difficulty assessments.

[0033] The evaluation dimensions are designed to take into account both automated measurement and intelligent judgment. Automated measurement dimensions include objective metrics that can be directly quantified through algorithms, such as text similarity calculation, grammatical error detection, execution efficiency statistics, and formatting compliance checks. These dimensions provide a fast and accurate basic score. Intelligent judgment dimensions cover subjective metrics such as content creativity, logical rationality, practical adaptability, and natural expression, which require semantic understanding and reasoning analysis to evaluate. These dimensions better reflect the model's true performance in complex application environments.

[0034] The entire evaluation task set features a dynamically evolving, adaptive mechanism. The system continuously monitors the performance of task execution and feedback during the evaluation process. If it finds that a particular task lacks sufficient discrimination, the evaluation results lack persuasiveness, or the model's capabilities are not effectively reflected, it automatically initiates a task optimization and adjustment process. This optimization process includes introducing new task variants to increase the comprehensiveness of the evaluation, adjusting task parameter configurations to improve the rationality of the difficulty distribution, and revising the definition of evaluation criteria to enhance the accuracy of judgment results. These strategies ensure that the evaluation task set can consistently adapt to the rapid development of large-scale model technology and the ever-changing application needs.

[0035] S102, constructing a multi-agent evaluation system based on the evaluation task set, including setting a questioner agent, an evaluator agent, and a reference agent; In this embodiment, three core agent roles are first designed based on the characteristics and requirements of the evaluation task set. Each agent has a unique functional positioning and working mechanism, and through mutual cooperation, a complete evaluation pipeline is formed. The questioner agent, as the core component of task generation and distribution, is responsible for generating specific task instances based on the templates and parameters in the evaluation task set, and distributing these tasks to the large model to be evaluated and the reference agent at the same time. The questioner agent has built-in multiple task generation strategies, including a template-filling generation method. By replacing key variables in the preset template, such as specific text content, entity categories, task parameters, etc., a large number of task instances with the same structure but different content are quickly generated.

[0036] The questioner agent is also equipped with a random combination generation mechanism, which can freely combine different text types, entity categories, and questioning methods to generate task variants with diversity and innovation. For example, in natural language processing tasks, the system can combine social media texts with the task of person name entity recognition to generate a specific task instance like "Which person names are included in the user dynamic '@Zhang San, shall we go to the zoo together tomorrow? @Li Si'?". At the same time, the questioner agent has a parameterized difficulty control function. By adjusting parameters such as text length, output format requirements, and knowledge complexity, it can precisely control the difficulty level of the generated tasks. A more advanced multi-task fusion generation strategy can organically combine multiple subtasks such as text classification, sentiment analysis, and entity recognition to create a composite evaluation task that requires comprehensive multiple abilities to complete.

[0037] The evaluator agent undertakes the most crucial judgment and scoring functions in the entire evaluation process. It has established a complete evaluation index system, covering quantitative evaluation criteria in multiple dimensions such as accuracy, logic, and innovation. In terms of accuracy evaluation, the evaluator agent uses a similarity calculation method based on vector space. The outputs of the large model to be evaluated and the reference agent are converted into high-dimensional vector representations through word embedding technology, and then the cosine similarity between the two vectors is calculated to obtain an objective matching degree quantification result. When the similarity value reaches above 0.9, the system considers the answer to have extremely high accuracy and gives a full score evaluation; when the similarity is between 0.7 and 0.9, the system considers the answer to be basically accurate but with minor differences and gives a relatively high score; when the similarity is below 0.7, the system will further analyze the reasons and nature of the differences and give corresponding score adjustments.

[0038] In the process of logic evaluation, the evaluator agent uses graph theory and knowledge reasoning techniques to construct a logical dependency graph of the answer content. The system takes each key information point in the answer as a node in the graph, and the logical relationships between information points such as causal relationships, progressive relationships, and parallel relationships as edges in the graph to form a complete logical structure representation. By analyzing the characteristics of the logical dependency graph such as connectivity, cyclicity, and integrity, the system can accurately identify problems such as logical loopholes, reasoning errors, and structural defects in the answer. The ratio of the number of effective logical relationships to the total number of logical relationships directly reflects the logical quality of the answer, and the system calculates a quantitative logic score based on this.

[0039] Innovation assessment utilizes a novelty detection algorithm based on a language model. The evaluator agent maintains a reference corpus containing a large number of common answer patterns and expressions. Using n-gram technology, it analyzes the vocabulary and expression structures contained in the responses to be evaluated. When an answer contains n-gram combinations that are rare or have never appeared in the reference corpus, the system deems it to have a certain degree of innovation. The ratio of novel n-grams to the total number of n-grams is used as a quantitative indicator of innovation; a higher ratio indicates a more innovative response.

[0040] As the provider of the assessment benchmark, the Reference Agent generates standardized reference answers for each assessment task by integrating a variety of high-quality information sources, including authoritative knowledge bases, professional literature, and domain expert experience. The Reference Agent's knowledge base encompasses authoritative content from multiple professional fields, including academic papers, technical specifications, industry standards, and best practices, ensuring the generated reference answers are highly accurate and authoritative. The system utilizes multi-source information fusion technology to perform consistency checks, reliability assessments, and timeliness verification on content from different sources, selecting the highest-quality information as the foundation for the reference answers.

[0041] To adapt to the needs of different evaluation scenarios and objectives, the system sets up a differentiated capability parameter configuration mechanism for each agent. In response to the evaluation needs of simulating ordinary user scenarios, the system configures the corresponding agent to have a limited knowledge reserve range, mainly including common sense knowledge and daily application information, and sets the reasoning depth to a relatively shallow level. The evaluation focuses on user experience-related indicators such as the ease of use, popularity, and intuitiveness of the model output. In the evaluation scenario simulating experts in professional fields, the system equips the agent with a deep professional knowledge system and enhanced logical reasoning ability, enabling multi-level complex analysis and judgment. The evaluation focus shifts to high-level capability indicators such as technical accuracy, professional standardization, and innovative breakthroughs.

[0042] The collaborative mechanism between agents is implemented through standardized communication protocols and data exchange formats. After generating a task, the questioner agent simultaneously sends the task instance to the large model to be evaluated and the reference agent, recording key information such as the sending timestamp, task identifier, and expected completion time. After completing the task, the large model to be evaluated and the reference agent return their respective outputs, along with metadata such as execution time, resource consumption, and processing status, to the evaluator agent. After receiving the outputs from both parties, the evaluator agent initiates a multi-dimensional evaluation process, independently calculating each evaluation metric. It then calculates a comprehensive score based on pre-set weightings and generates an evaluation report with detailed analysis.

[0043] S103, interactive evaluation through the multi-agent evaluation system, wherein the questioner agent generates a task instance from the task set and simultaneously sends it to the large model to be evaluated and the reference agent through a row interaction mechanism, obtaining output results of both, and the evaluator agent performs a multi-dimensional comparative scoring of the output results of the two based on a scoring standard formulated by the evaluator agent based on the basic indicators and extended indicators; The Questioner agent first extracts the evaluation tasks to be performed from the evaluation task set. In actual operation, the Questioner agent maintains a task instance generation engine that dynamically creates task instances based on different generation strategies. For example, when evaluating the entity recognition capabilities of a large model, the Questioner agent might employ a template filling strategy, replacing the placeholders in the template "Please identify the [entity category] entity in the text '[specific text]'" with actual content, generating a specific task instance such as "Please identify the location entity and time entity in the text 'X City Subway Line 14 will open in 2025'." For more complex evaluation requirements, the Questioner agent employs a multi-task fusion strategy, combining multiple subtasks such as text classification, sentiment analysis, and entity recognition into a composite task instance to comprehensively test the comprehensive processing capabilities of the large model.

[0044] After generating task instances, the questioner agent efficiently distributes tasks through a parallel interaction mechanism. The core of this mechanism is an intelligent scheduler that monitors the status of each processing node in real time. Specifically, the scheduler maintains status information for each agent, including the number of tasks currently being processed, N_processing, and the resource utilization, R_usage. When a new task instance is available for dispatch, the scheduler calculates the priority of each available agent using the priority formula P = 1 / (N_processing + 1) × (1-R_usage). This calculation method ensures that tasks are preferentially assigned to agents with less load and sufficient resources, thereby achieving load balancing.

[0045] During task distribution, the questioner agent uses asynchronous parallelism to simultaneously send the same task instance to the large model to be evaluated and the reference agent. The system uses a message queue mechanism to ensure reliable task transmission. Each task is accompanied by a unique identifier and timestamp to facilitate subsequent result matching and performance analysis. Once the large model to be evaluated and the reference agent receive the task, they independently process it and generate output results. The system records detailed execution information, including the task receipt time, processing start and end times, and the complete output content.

[0046] When generating reference answers, the reference agent draws on its internally integrated authoritative knowledge base and professional literature resources. For example, for technical questions, the reference agent retrieves relevant academic papers and technical documentation; for factual questions, it consults a verified knowledge graph. This mechanism ensures the high accuracy of the reference answers.

[0047] After both agents complete their tasks and return results, the evaluator agent begins performing a multi-dimensional comparative scoring. The evaluator agent implements a comprehensive scoring system, dynamically constructed based on the basic and extended metrics defined in the evaluation task. Basic metrics focus on the core capabilities of the model, such as accuracy and logic; extended metrics focus on higher-level features such as robustness and ethicality.

[0048] In terms of accuracy evaluation, the evaluator agent first calculates the output of the large model to be evaluated. and the output of the reference agent Perform preprocessing, including word segmentation, stop word removal, stemming, etc. Then use the pre-trained word embedding model to map the processed text into a high-dimensional vector space to obtain the corresponding vector representation. By calculating the cosine similarity To quantify the semantic similarity between two outputs. Based on practical experience, the system sets a mapping rule from similarity to accuracy score: when S_cos ≥ 0.9, the accuracy score S_acc = 10 points; when 0.7 ≤ S_cos < 0.9, S_acc = 7 points; and so on, forming a continuous scoring system.

[0049] Logical evaluation uses a more complex analysis method. The evaluator agent first performs syntactic analysis and semantic parsing on the output content to identify the arguments, evidence, and reasoning relationships. Based on this information, a logical dependency graph is constructed. The nodes in the graph represent key information points, and the edges represent the logical relationships between information points, such as causal relationships, conditional relationships, parallel relationships, etc. Through the graph analysis algorithm, the system can identify the integrity and rationality of the logical chain. For example, in mathematical reasoning tasks, the evaluator will check whether each step of the deduction has sufficient basis and whether there are logical jumps or circular arguments. Finally, by calculating the number of valid logical relationships The number of all possible logical relationships The ratio of the logical score .

[0050] Innovation evaluation focuses on identifying novel content in the output. The evaluator agent maintains a large-scale corpus of common expressions, covering standard answers and conventional solutions in various fields. During the evaluation, the system decomposes the output of the large model to be evaluated into n-grams of different lengths (such as 3-grams, 4-grams), and then checks the frequency of occurrence of these n-grams in the corpus. For n-grams that do not appear in the corpus or appear very rarely, the system considers them innovative. By counting the number of innovative n-grams and the total number of n-grams Calculate the innovation score This approach is effective in identifying whether large models can generate novel expressions or provide unique problem solutions.

[0051] After completing the independent scoring of each dimension, the evaluator agent needs to calculate the comprehensive score. The key here is the dynamic configuration of weights. The system does not use a fixed weight distribution, but flexibly adjusts it according to the task type and evaluation objectives. For example, for factual question answering tasks, the accuracy weight will be set higher (such as 0.6), while the innovation weight Relatively low (such as 0.1); for creative writing tasks, it will increase the weight of innovation and reduce the weight of accuracy. Calculate the final comprehensive score.

[0052] S104, construct a structured dataset based on task instances, output results and scoring data, wherein the structured dataset includes task description, output of the large model to be evaluated, output of the reference intelligent agent, and scores of each dimension. The structured dataset is used to analyze and optimize the construction of the next round of task sets based on the analysis results.

[0053] It's important to note that once the evaluator agent completes scoring a task, the system immediately initiates the data integration process. Each evaluation record is organized into a unified data structure, containing metadata such as the task identifier, task description, task category, difficulty level, and generation timestamp. The task description not only records the specific question text generated by the questioner agent but also contains information about the generation strategy for the task, such as whether it was generated through template filling, random combination, or multi-task fusion.

[0054] The output data of the large model to be evaluated is stored as complete text, and the system automatically extracts key feature information. For example, for code generation tasks, the system analyzes the code structure and counts the number of lines of code, programming language features used, and the number of function calls. For natural language processing tasks, features such as text length, sentence complexity, and the number of professional terms used are extracted. These structured features are stored together with the original output text to form a multi-level data representation.

[0055] The output of the reference agent undergoes a similar process. Because the output of the reference agent is typically highly standardized and accurate, the system also annotates the knowledge sources it cites, such as specific literature, knowledge base entries, or expert rules. This annotation mechanism enables subsequent analysis to trace the basis of the reference answer.

[0056] The constructed structured dataset then enters the analysis phase. The system first performs descriptive statistical analysis, calculating basic statistics such as the average score, standard deviation, and score distribution for each task. By comparing the score distributions across different task categories and difficulty levels, the system can identify the capabilities of the large model being evaluated. For example, if a large model's average score on natural language processing tasks is significantly higher than on code generation tasks, while its score drops sharply on high-difficulty reasoning and decision-making tasks, this provides guidance for optimizing the task set.

[0057] In-depth analysis employs machine learning to uncover hidden patterns. The system uses clustering algorithms to group all assessment records and identify clusters of tasks with similar characteristics. For example, it might be discovered that a certain type of math problem involving multi-step reasoning generally receives low scores, even though these problems share similar linguistic patterns in their task descriptions. By mining association rules, the system can also identify correlations between different assessment dimensions. For example, outputs with high innovation scores often also perform well in terms of logic, suggesting that the inherent connections between these dimensions should be considered when designing tasks.

[0058] Optimization suggestions are fed back to the evaluation task construction module in the form of structured improvement prompts. These prompts contain specific instructions, such as "adding a medium-complexity logical reasoning task involving 3-5 reasoning steps," "increasing the weight of the syntax error detection subtask in code debugging tasks," and "adding more text examples containing sarcasm and metaphors to the sentiment analysis task." Upon receiving these improvement prompts, the evaluation task construction module will adjust its task generation strategy, update its task template library, and modify the difficulty grading criteria accordingly.

[0059] In a possible implementation of the present invention, the system first establishes a dynamic task priority queue to manage all pending task instances. When each task enters the queue, the system will assign it a priority score according to the priority calculation formula Priority = α × Importance + β × (1 / WaitTime) + γ × ResourceMatch. In actual operation, the task importance Importance is determined by the task type and the evaluation target. For example, the importance of the core capability assessment task is set to 1.0, the extended capability assessment task is set to 0.7, and the boundary test task is set to 0.5. The waiting time WaitTime is updated in real time in seconds. As the task stays in the queue for an increasing amount of time, the contribution of the 1 / WaitTime item gradually increases, ensuring that tasks that wait for a long time will not be delayed indefinitely. The resource matching degree ResourceMatch reflects the degree of match between the task requirements and the currently available resources. When the computing resources and memory capacity required for the task are highly consistent with the current state of the system, the value is close to 1.0, and vice versa.

[0060] The weight parameters α, β, and γ reflect the system's scheduling strategy. In the default configuration, α is set to 0.5 to ensure that important tasks are prioritized, β is set to 0.3 to prevent task starvation, and γ is set to 0.2 to improve resource utilization efficiency. These weights are dynamically adjusted based on the system's operating status. For example, if a large task backlog is detected, the system automatically increases β to speed up the processing of older tasks. When system resources are sufficient, γ is increased to optimize resource allocation.

[0061] The system continuously monitors the operational status of each agent, particularly its response time. When the response time of an agent processing a task exceeds a preset threshold (e.g., 1.5 times the average response time), the system determines that the agent may be under heavy load or experiencing performance degradation. At this point, the task migration mechanism is automatically triggered. The system first evaluates the migration cost of each task in the agent's task queue, prioritizing tasks that have not yet begun processing or have a low processing progress. Migration targets are selected based on the current load and historical performance data of other agents, ensuring that tasks are migrated to the most appropriate execution environment.

[0062] To further improve scheduling efficiency, the system implements a task execution time prediction function based on historical data. The system maintains a task execution history database that records the actual execution time of different types of tasks under different configurations. Through time series analysis and regression models, the system can predict the execution time of new tasks. For example, for tasks involving complex reasoning chains, the system will predict that they will require a longer processing time based on characteristics such as the number of reasoning steps and the knowledge domain involved. Based on these predictions, the system will reserve sufficient resources for compute-intensive tasks in advance, including pre-allocating GPU memory, pre-loading necessary model parameters, and pre-establishing computational graphs, thereby reducing task startup delays.

[0063] In this implementation, to enhance the structural relevance and responsiveness of evaluation tasks, a task dependency graph is constructed for the evaluation task set. Specifically, the system first defines the logical order of tasks based on task design requirements, assigning some tasks as predecessors and others as successors. When the score of the large model to be evaluated on a predecessor task falls below a preset threshold, the system automatically adjusts the difficulty level of the corresponding successor task, reducing the evaluation intensity to avoid invalid evaluations. To further improve semantic coherence and verification capabilities within the task set, the system organizes tasks with similar semantics and capability requirements into task clusters. During execution, a cross-validation mechanism is introduced between task scores within a cluster, allowing the evaluation results of a single task to serve as a reference for the evaluation of other related tasks. Furthermore, to achieve a more flexible task control mechanism, the system sets task trigger conditions based on the real-time performance of the large model to be evaluated in the current task. For example, advanced tasks are activated only when the score of a certain task type exceeds a certain threshold, or specific task paths are skipped when low-quality output occurs. This allows for dynamic optimization of the evaluation process and conserves computing resources.

[0064] See also Figure 2 The second embodiment of the present invention provides a large model autonomous evaluation device based on multi-agent technology, comprising: The evaluation task set generation unit 201 is used to obtain the application scenarios and capability boundaries of the large model to be evaluated, extract evaluation tasks from different fields and configure difficulty gradients, and assign basic indicators and extended indicators by setting dimension weights to generate an evaluation task set containing task descriptions, difficulty levels, and evaluation dimensions; An evaluation system construction unit 202 is configured to construct a multi-agent evaluation system based on the evaluation task set, including setting a questioner agent, an evaluator agent, and a reference agent; A multi-dimensional comparative scoring unit 203 is configured to interactively evaluate through the multi-agent evaluation system, wherein the questioner agent generates a task instance from the task set and simultaneously sends it to the large model to be evaluated and the reference agent through a row interaction mechanism to obtain output results of both. The evaluator agent then performs a multi-dimensional comparative scoring on the output results of both based on a scoring standard formulated by the evaluator agent based on the basic indicators and extended indicators. The structured data set construction unit 204 is used to construct a structured data set based on task instances, output results and scoring data, wherein the structured data set includes task description, output of the large model to be evaluated, output of the reference intelligent agent, and scores of each dimension. The structured data set is used to analyze and optimize the construction of the next round of task sets based on the analysis results.

[0065] The third embodiment of the present invention provides a large-model autonomous evaluation device based on multi-agent technology, including a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a large-model autonomous evaluation method based on multi-agent technology as described in any one of the above.

[0066] The fourth embodiment of the present invention provides a computer-readable storage medium, characterized in that it stores a computer program, and the computer program can be executed by a processor of the device where the computer-readable storage medium is located to implement a large-model autonomous evaluation method based on multi-agent technology as described in any one of the above items.

[0067] Based on the large-scale autonomous evaluation method, device, equipment and storage medium based on multi-agent technology provided by the present invention, an evaluation task set covering multiple scenarios and with multiple levels of difficulty is generated by setting the dimensional weights of basic indicators and extended indicators; a multi-agent evaluation system of questioners, evaluators and reference agents is constructed based on the task set, and a parallel interaction mechanism is adopted to simultaneously initiate tasks to the model to be evaluated and the reference agent. The evaluator agent compares the outputs of the two according to multi-dimensional scoring criteria, and constructs the task instances, output results and scores into a structured data set; through the analysis of the data set, the subsequent task sets are dynamically optimized to achieve a comprehensive and iterative evaluation of the large-scale model capabilities, thereby solving the problem that the existing evaluation method cannot fully reflect the true capability level of the large model.

[0068] For example, the computer programs described in the third and fourth embodiments of the present invention may be divided into one or more modules, which are stored in the memory and executed by the processor to implement the present invention. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which are used to describe the execution process of the computer program in the device for implementing a large-scale autonomous model assessment based on multi-agent technology. For example, the device described in the second embodiment of the present invention.

[0069] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the large-scale model autonomous evaluation method based on multi-agent technology, and utilizes various interfaces and lines to connect the various parts of the large-scale model autonomous evaluation method based on multi-agent technology.

[0070] The memory can be used to store the computer programs and / or modules. The processor implements various functions of a large-scale autonomous assessment method based on multi-agent technology by running or executing the computer programs and / or modules stored in the memory and accessing data stored in the memory. The memory may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function (such as a sound playback function or a text conversion function); the data storage area may store data generated based on the use of the mobile phone (such as audio data and text message data). In addition, the memory may include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0071] If the implementation module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can also implement all or part of the process steps in the above-mentioned method embodiments by using a computer program to instruct the relevant hardware. The computer program can be stored in a computer-readable storage medium. When executed by a processor, the computer program can implement the steps of each of the above-mentioned method embodiments. The computer program includes computer program code, which can be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content of the computer-readable medium can be appropriately increased or decreased based on the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0072] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0073] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A large model autonomous evaluation method based on multi-agent technology, characterized by: include: Obtain the application scenarios and capability boundaries of the large model to be evaluated, extract evaluation tasks from different fields and configure difficulty gradients, and assign basic indicators and extended indicators by setting dimension weights to generate an evaluation task set that includes task descriptions, difficulty levels, and evaluation dimensions; Constructing a multi-agent evaluation system based on the evaluation task set, including setting a questioner agent, an evaluator agent and a reference agent; Interactive evaluation is performed through the multi-agent evaluation system, wherein the questioner agent generates a task instance from the task set and simultaneously sends it to the large model to be evaluated and the reference agent through a row interaction mechanism to obtain output results of both. The evaluator agent performs a multi-dimensional comparative scoring of the output results of the two based on a scoring standard, wherein the scoring standard is formulated by the evaluator agent based on the basic indicators and extended indicators; A structured dataset is constructed based on task instances, output results, and scoring data. The structured dataset includes task descriptions, outputs of the large model to be evaluated, outputs of reference agents, and scores for each dimension. The structured dataset is used to analyze and optimize the construction of the next round of task sets based on the analysis results.

2. A large-scale model autonomous evaluation method based on multi-agent technology according to claim 1, characterized in that: The evaluation task set adopts a hierarchical task generation strategy, where: The first level is the domain division, including the natural language processing domain, code generation domain and reasoning and decision-making domain; the second level is the ability dimension, with four dimensions of understanding ability, generation ability, reasoning ability and innovation ability under each domain; the third level is the difficulty gradient, with each ability dimension containing three difficulty levels: elementary, intermediate and advanced; and the task matrix is ​​generated through the combination of the three levels.

3. The large-scale model autonomous evaluation method based on multi-agent technology according to claim 1 is characterized in that: It also includes constructing a task dependency graph of the evaluation task set, specifically: Define the logical relationship between the preceding and subsequent tasks. When the score of the large model to be evaluated on the preceding task is lower than the threshold, the difficulty level of the subsequent task is automatically reduced. Establish task clusters, where semantically related tasks are organized into task clusters, and task scoring results within the same task cluster are mutually verified by users; Set task trigger conditions, where specific tasks are dynamically activated or skipped based on the real-time performance of the large model to be evaluated.

4. The large-scale model autonomous evaluation method based on multi-agent technology according to claim 1 is characterized in that: The questioner agent receives task metadata of a set of evaluation tasks, generates task instances in batches in sequence according to difficulty levels, and refines the instances in the same batch according to task description templates and indicator weights, wherein the task metadata includes task ID, description template and expected output format.

5. The large-scale model autonomous evaluation method based on multi-agent technology according to claim 1 is characterized in that: The evaluator agent performs a multi-dimensional comparative scoring of the two output results based on the scoring criteria, specifically: Output the large model to be evaluated and the reference agent output Convert to vector representation and calculate cosine similarity , converted into accuracy scores according to preset mapping rules ; Build a logical dependency graph based on the two outputs and calculate the logical coherence score ,in, is the number of valid logical relations, is the number of all possible logical relations; Compare the n-grams in the output of the large model to be evaluated with the corpus and calculate the innovation score ,in, is the number of innovative n-grams, is the total number of n-grams; Calculate the comprehensive score based on the dimension weight of the task category ,in, is the weight of the accuracy score, is the weight of the logical coherence score, is the weight of the innovation score.

6. The large-scale model autonomous evaluation method based on multi-agent technology according to claim 1 is characterized in that: After the questioner agent generates a task instance from the task set, the method further includes: Establish a task priority queue and calculate the task priority according to the formula Priority = α × Importance + β × (1 / WaitTime) + γ × ResourceMatch, where Importance is the task importance, WaitTime is the waiting time, ResourceMatch is the resource matching degree, and α, β, and γ are the corresponding weights; When it is detected that the response time of an agent exceeds the preset threshold, some tasks in its task queue are automatically migrated to idle agents. Based on historical data, the execution time of different types of tasks is predicted, and resources are reserved in advance for computationally intensive tasks.

7. A large-scale model autonomous evaluation device based on multi-agent technology, characterized in that: include: The evaluation task set generation unit is used to obtain the application scenarios and capability boundaries of the large model to be evaluated, extract evaluation tasks from different fields and configure difficulty gradients, and allocate basic indicators and extended indicators by setting dimension weights to generate an evaluation task set that includes task descriptions, difficulty levels, and evaluation dimensions; An evaluation system construction unit, configured to construct a multi-agent evaluation system based on the evaluation task set, including setting a questioner agent, an evaluator agent, and a reference agent; a multi-dimensional comparative scoring unit, configured to interactively evaluate through the multi-agent evaluation system, wherein the questioner agent generates task instances from the task set and simultaneously sends them to the large model to be evaluated and the reference agent through a row interaction mechanism, obtaining output results of both, and the evaluator agent performs a multi-dimensional comparative scoring of the output results of the two based on a scoring standard formulated by the evaluator agent based on the basic indicators and extended indicators; A structured dataset construction unit is used to construct a structured dataset based on task instances, output results and scoring data, wherein the structured dataset includes task description, output of the large model to be evaluated, output of the reference intelligent agent, and scores of each dimension. The structured dataset is used to analyze and optimize the construction of the next round of task sets based on the analysis results.

8. A large-scale autonomous evaluation device based on multi-agent technology, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program, and the computer program can be executed by the processor to implement a large-model autonomous evaluation method based on multi-agent technology as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that A computer program is stored, and the computer program can be executed by a processor of the device where the computer-readable storage medium is located to implement a large-model autonomous evaluation method based on multi-agent technology as described in any one of claims 1 to 6.

Images