A large model automatic evaluation method based on multi-agent cooperation

Abstract

This invention discloses a method for automated evaluation of large-scale models based on multi-agent collaboration, comprising: First, automatically mining potential evaluation tasks from domain text through a task exploration module and constructing a task knowledge base; Second, using a teacher-student multi-agent structure to semantically expand and optimize task expression, generating multi-level task corpora; Third, introducing a teacher-student-reviewer multi-agent mechanism in the test case generation stage, generating high-quality test cases through iterative questioning and feedback; Finally, performing zero-human evaluation of the model's performance under each task based on an automatic scoring module, and outputting a comprehensive capability report. This invention enables automatic discovery and dynamic expansion of evaluation tasks, and can complete multi-dimensional capability assessment and weakness identification of large language models without human intervention, possessing advantages such as high automation, strong interpretability, and broad evaluation coverage.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence model evaluation technology, and in particular to an automated evaluation method for large models based on multi-agent collaboration. Background Technology

[0002] As the application of large language models in tasks such as natural language understanding, text generation, and reasoning continues to expand, evaluating model capabilities has become a crucial step in ensuring model reliability and driving technological progress. Most existing model evaluation methods rely on manually designed fixed task sets or benchmark tests. These evaluation methods typically involve experts manually constructing datasets and question templates, resulting in limited coverage, long update cycles, and difficulty in reflecting the model's true performance across different domains and task types.

[0003] As model performance improves, traditional static evaluation benchmarks gradually exhibit a "saturation" phenomenon, meaning that while the model can achieve high scores on existing test sets, its performance becomes unstable on new or more complex tasks. This indicates that fixed evaluation tasks are insufficient to comprehensively measure the model's overall capabilities and cannot accurately reveal the differences in the model across different cognitive levels (such as comprehension, reasoning, and creativity).

[0004] To reduce reliance on manual intervention, some automated evaluation methods have been proposed to automatically generate evaluation tasks or questions based on preset templates. However, these methods generally suffer from the following shortcomings: 1) The task type depends on the predefined template and cannot discover new evaluation tasks independently; 2) The generated test samples lack semantic diversity and hierarchy, making it difficult to cover the multidimensional capabilities of the model; 3) The evaluation process still requires manual screening and correction, making it difficult to achieve true zero-manual evaluation; 4) Most methods use static evaluation indicators, which cannot dynamically reflect the evolution of the model's capabilities and the distribution of weaknesses.

[0005] Furthermore, existing assessment systems mostly focus on superficial performance metrics at the task level, lacking a systematic analytical mechanism from a cognitive perspective, and thus failing to support the precise identification of model weaknesses and the hierarchical description of capability structures.

[0006] Therefore, there is an urgent need for a new automated model evaluation technology that can automatically discover tasks from unstructured text, build a task knowledge base, generate diverse test cases, and, combined with an automatic scoring mechanism, dynamically and finely evaluate the overall capabilities of the model without human intervention. This invention addresses these issues by proposing an adaptive, scalable, zero-human-intervention, and weakness-detection-capable automatic evaluation method for general-purpose large language models. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing technologies by proposing an automated evaluation method for large models based on multi-agent collaboration.

[0008] The objective of this invention is achieved through the following technical solution: an automatic evaluation method for large models based on multi-agent cooperation, the method comprising: Semantic fragments are extracted from unstructured text in the target domain, and these semantic fragments are sampled. Cognitive abilities are divided into six levels and several sub-abilities. Based on the weighted results of the target distribution, current distribution, and context-related distribution, the target ability label corresponding to the task is determined. The large model to be evaluated is then used to generate the task based on the target ability label and semantic fragments. The Socratic question-and-answer mechanism is used to conduct multiple rounds of interaction with the generated task, and key information from the multiple rounds of dialogue is extracted for knowledge expansion. Multi-agent collaboration generates high-quality task instances as an evaluation set; The evaluation set is categorized into tasks, and the model's performance score is calculated for each task.

[0009] Furthermore, the step of extracting semantic fragments from unstructured text in the target domain and sampling the semantic fragments includes: collecting unstructured text data from academic papers, reports, and technical documents in the target domain; performing sentence segmentation and semantic fragmentation processing on the text; and sampling the semantic fragments using the K-Center-Greedy algorithm.

[0010] Furthermore, the division of cognitive abilities into six levels and several sub-abilities specifically involves: constructing a hierarchical distribution of abilities based on an improved Bloom cognitive classification model, dividing the cognitive process into six levels and 27 fine-grained sub-abilities: memory, understanding, application, analysis, evaluation, and creation.

[0011] Furthermore, determining the target capability label corresponding to the task based on the weighted result of the target distribution, the current distribution, and the context-related distribution includes: The target distribution is obtained by normalizing the capability weight vectors defined by domain experts. The frequency vector maintained during the iterative construction of the task set is normalized and used as an existing distribution. The ranking vector of each capability in the large model, sorted by relevance from high to low, is transformed into a probability distribution using exponential decay and normalization. ; The calculation of the chosen distribution is as follows:

[0012] in To balance the trade-off between context alignment and coverage underrepresentation, the following was calculated: Then, the two items with the highest probabilities are selected as the target capability labels for this domain scenario, and the frequency vector is updated.

[0013] Furthermore, the task of generating data using the large model to be evaluated based on the target capability label and semantic fragments includes: By combining semantic fragments and their target capability labels, the large language model is input to generate task names, task descriptions, and question types; semantic similarity is calculated on the generated task set, and duplicate or semantically overlapping tasks are removed; a preliminary task list is output to provide input for subsequent context expansion.

[0014] Furthermore, the multi-round interaction with the generated task using the Socratic question-and-answer mechanism includes: Constructing teacher and student intelligent agents; The teacher agent poses higher-level questions based on the task description, and the student agent provides preliminary answers. The teacher's intelligent agent dynamically generates more in-depth follow-up questions based on the students' answers, forming a recursive question-and-answer chain; The intelligent agent extracts and summarizes the content of multiple rounds of question-and-answer sessions to form a set of key concepts.

[0015] Furthermore, the extraction of key information from multi-turn dialogues for knowledge expansion includes: converting the key information into vector representations; Based on the Sentence-BERT semantic retrieval model, text fragments with semantic similarity to concepts are retrieved from an external corpus; Add the search results to the task context to form an expanded set of task knowledge; stop expanding if the number of new knowledge points is below a set threshold or the dialogue enters a repetitive phase.

[0016] Furthermore, the multi-agent collaboration in generating high-quality task instances as an evaluation set includes: Construct a multi-agent system including teacher agents, student agents, review agents, and meta-detection agents; The teacher agent generates test questions based on the task and context, and the student agent generates the answers. The reviewing agent evaluates the answers based on four criteria: correctness, logic, clarity, and difficulty. If the score reaches the threshold, the review agent will provide feedback to the teacher agent indicating an increased difficulty level, which will guide the generation of more challenging questions. If the score is not up to standard, the review agent will trigger the context extension module to guide a new semantic retrieval through feedback information; The meta-detection agent monitors the convergence of multi-agent dialogue and the deviation from the task objective, and stops the interaction when the convergence condition is met. All question-answer pairs are quality-scored and deduplicated to construct the final evaluation instance set.

[0017] Furthermore, the task classification of the evaluation set and the calculation of the model's performance score on each task include: For multiple-choice and true / false questions, the accuracy rate is used to evaluate the model's basic recognition ability; for classification tasks, the macro-average F1 score is used to mitigate the bias caused by label imbalance; for open-ended question-answering tasks, binary keypoint matching and a 5-point Likert score are combined. The final score of the question-answering task is a weighted fusion of the two parts. Based on the cognitive ability labels of the task, the score of each task is mapped to the corresponding ability label. The score of each cognitive ability is calculated by weighted averaging to obtain the model's ability vector. The ability vector is then transformed into an ability distribution histogram and an ability radar chart.

[0018] On the other hand, a large model automatic evaluation device based on multi-agent cooperation is also provided, including a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it implements the large model automatic evaluation method based on multi-agent cooperation.

[0019] The beneficial effects of this invention are: A task automatic discovery mechanism based on a cognitive hierarchy model is proposed. By introducing a capability hierarchy distribution and a distribution-guided probability selection method, the mechanism realizes the automatic identification of potential task types from unstructured domain text, significantly reducing human intervention and improving the systematicness and automation of task generation. A context extension method integrating Socratic question answering and semantic retrieval was constructed, which realizes multi-round semantic expansion and knowledge enrichment of task corpus, makes up for the lack of context information in traditional task generation, and ensures the semantic hierarchy and knowledge relevance of task content. 3. A multi-agent collaborative evaluation instance generation framework was established. Through multiple rounds of interaction among teacher agents, student agents, review agents, and meta-detection agents, a high-quality, progressive task instance generation process was achieved, ensuring the rationality and diversity of the generated data. 4. A fine-grained evaluation mechanism based on dual-ability mapping was constructed. By accurately labeling and mapping tasks through cognitive level labels, the multidimensional cognitive ability distribution of the model can be meticulously characterized, effectively overcoming the problem of limited coverage of traditional evaluation tasks. 5. The evaluation framework has been made adaptable to different domains. By replacing the input corpus and capability templates, it can be quickly migrated to knowledge-intensive domains such as law, medicine, and education, enabling the efficient construction and continuous optimization of automated evaluation benchmarks in multiple domains. Attached Figure Description

[0020] Figure 1 A flowchart illustrating a dynamic knowledge-guided summary generation method for implementing this invention; Figure 2 A diagram illustrating the method for selecting knowledge based on a dynamic selection mechanism provided by this invention; Figure 3 A diagram illustrating a summary generation method based on dynamic knowledge guidance provided by this invention; Figure 4 This is a schematic diagram of an automated evaluation device for large models based on multi-agent collaboration, provided by the present invention. Detailed Implementation

[0021] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0022] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, those skilled in the art will understand that the invention may be practiced in other ways than those described herein, and similar extensions and modifications may be made without departing from the spirit of the invention. Therefore, the invention is not limited to the following specific embodiments.

[0023] like Figure 1 As shown, this invention provides an automatic evaluation method, system, and apparatus for large language models based on multi-agent collaboration and cognitive hierarchy guidance, with the following steps: Step one, task exploration, mainly involves automatically mining potential evaluation tasks from domain text, including the following steps: 1.1 Text sampling; a) Collect unstructured text data from academic papers, reports, and technical documents in the target field (such as law, medicine, education, etc.); b) Perform sentence segmentation and semantic fragmentation on the text; c) The K-Center-Greedy algorithm is used to sample semantic segments, so that the sampled segments are distributed as evenly as possible in the semantic space, thus ensuring the diversity of task discovery.

[0024] Based on the improved Bloom cognitive classification model, a hierarchical distribution of abilities was constructed, dividing the cognitive process into six levels (memory, understanding, application, analysis, evaluation, and creation) and 27 fine-grained sub-abilities (as shown in Table 2). 1.2 Cognitive Ability Selection: To ensure both context alignment and target ability coverage during task generation, this invention employs the following distributed guided selection mechanism: a) Construction of the target distribution: A preference-weighted distribution is given by domain experts. Let the total number of fine-grained capability units be n, and the domain experts assign weights to the i-th capability. The weight vector Normalization yields the target distribution

[0025] b) Calculation of the existing distribution: The existing distribution This reflects the coverage of each capability by the currently generated task set. A frequency vector is maintained during the iterative construction of the task set. ,in Let be the number of times ability i has been selected. The existing distribution is as follows:

[0026] c) Context distribution estimation: For the current semantic context, prompts are used to allow the large language model to rank the 27 capabilities from highest to lowest relevance, resulting in a ranking vector. ,in Let i be the rank of ability i (1 indicates most relevant). The rank is converted to probability using exponential decay:

[0027] And normalization yields:

[0028] hyperparameters Control the decay rate, The higher the value, the more concentrated the weight of the abilities ranked higher.

[0029] d) The calculation of the chosen distribution is as follows:

[0030] in To balance the trade-off between context alignment and coverage underrepresentation by adjusting the balancing factor; This capability selection mechanism ensures that the task is highly aligned with the current context, while also being able to compensate for [the limitations of the task]. Prioritize coverage of capabilities that are not yet fully represented in the target, thereby achieving a balance between contextual relevance and capability coverage.

[0031] e) To Sort the data by probability and select the two items with the highest probability values ​​as the target capability label force for the semantic scene. Update the frequency vector f for calculation in the next round of task generation. f) Parameter Recommendation: In the embodiments of this invention, the following parameters can be empirically set: (Or, as adjusted by experts / developers based on the needs of the field).

[0032] 1.3 Task Generation; a) Combine semantic fragments and their target ability labels, input into a large language model to generate task name, task description and question type (such as question-and-answer, selection, judgment, etc.). b) Calculate semantic similarity for the generated task set and remove duplicate or semantically overlapping tasks; c) Output a preliminary task list to provide input for subsequent context expansion.

[0033] Step two involves expanding the knowledge base of the task to include sufficient domain semantic information. This includes the following steps: 2.1 Socratic question-and-answer mechanism; triggering higher-level cognitive exploration through a recursive Prompt mechanism, promoting the generation of more logical and innovative contextual content by the model.

[0034] a) Construct teacher and student intelligent agents; b) The teacher agent poses higher-level questions based on the task description, and the student agent provides preliminary answers. c) The teacher agent dynamically generates more in-depth follow-up questions based on the students' answers, forming a recursive question-and-answer chain; d) Summarize and extract the content of multiple rounds of question and answer by the intelligent agent to form a set of key concepts.

[0035] 2.2 Knowledge retrieval and supplementation; a) Transform key concepts into vector representations; b) Based on the Sentence-BERT semantic retrieval model, retrieve text fragments with semantic similarity to concepts from an external corpus; c) Add the search results to the task context; form a task knowledge expansion set; stop expansion if the number of new knowledge points is below the set threshold or the dialogue enters a repetitive phase; Ultimately, a semantically rich contextual knowledge base with strong task relevance is formed, providing knowledge support for subsequent instance generation.

[0036] Step 3: Multi-agent collaboration generates high-quality task instances. The system includes the following agents: 1) Teacher agent: poses task-related questions; 2) Student agent: generates answers; 3) Review agent: evaluates the quality of the answers; 4) Meta-detection agent: detects the convergence state and consistency of multi-turn interactions. The instance generation steps are as follows: 3.1 The teacher agent generates an initial question based on the problem description and the given context, and the student agent generates the answer; 3.2 The review agent is scored based on four indicators: correctness, logic, completeness, and difficulty; 3.3 If the score is higher than the threshold, the review agent sends a "difficulty increase instruction" to the teacher agent; the teacher agent then generates more complex questions based on the feedback. 3.4 If the score is below the threshold, the review agent sends a "knowledge supplementation instruction" to the context module, triggering a new retrieval and question answering; 3.5 Meta-detection agents monitor semantic drift and redundancy in dialogues, and terminate the interaction when there is a lot of redundant or invalid information in the question or answer; 3.6 The system filters all generated question-answer pairs, retaining high-quality instances to form the final evaluation set.

[0037] This step ensures the semantic accuracy and progressive difficulty of generated instances through quality control of the review agent and the meta-detection agent.

[0038] Step four: Calculate the model's performance across different cognitive dimensions based on the generated assessment examples. The steps are as follows: 4.1 Based on task type (question-answering, multiple choice, reasoning, judgment) and cognitive ability labels, the automated scoring module calculates the model's performance score for each task. Specifically, multiple choice and judgment questions use accuracy to assess the model's basic recognition ability; classification tasks use macro-average F1 scores to mitigate bias caused by label imbalance; open-ended question-answering tasks combine binary keypoint matching and a 5-point Likert score to comprehensively evaluate the quality of responses in terms of accuracy, clarity, and reasoning depth. The final score for question-answering tasks is a weighted fusion of the two parts. Subsequently, based on the cognitive ability labels of the tasks, this invention aggregates and calculates task scores according to dimensions such as memory, comprehension, analysis, reasoning, and creativity. Each ability score is the average score of its corresponding task, and this is used to construct a model ability profile, thereby achieving visualized analysis and quantitative comparison of the model's multidimensional cognitive abilities.

[0039] 4.2 Constructing a Capability Matrix Where n represents the number of tasks and m represents the number of cognitive abilities, the score of each task is mapped to the corresponding ability label, and the score of each cognitive ability is calculated by weighted average to obtain the model's ability vector; 4.3 The capability vectors are transformed into capability distribution bar charts and capability radar charts, which can realize comparative analysis between different models, tracking of single-model capability vulnerabilities, and the output results can be used for the evaluation, optimization and monitoring of large language models.

[0040] This invention further selects the legal field as the verification scenario. This field is characterized by complex reasoning chains, a vast conceptual system, and tight semantic dependencies, requiring a high level of knowledge understanding and logical reasoning ability from the language model. Based on the automatic task exploration, context expansion, and high-quality use case generation framework proposed in this invention, the system automatically constructs an evaluation set covering various legal task types without human intervention, including legal text understanding, case fact comparison, crime prediction, and judicial reasoning.

[0041] Based on this, this invention systematically evaluated several mainstream large language models, and the test results are shown in Table 1. It also focused on comparing the performance differences of the Qwen2.5 series models across six cognitive ability levels, with the results as follows: Figure 2 As shown. Experimental results demonstrate that the assessment framework constructed in this invention can clearly distinguish the model's ability levels at different levels, such as memory, comprehension, analysis, and creation. Figure 3 As shown, this invention further establishes a capability profile for the Qwen2.5-7B-Instruct model, demonstrating its advantages in legal reasoning and complex semantic analysis. The results verify the applicability and evaluation accuracy of this invention in knowledge-intensive fields, providing a systematic and quantitative basis for performance optimization of legal language models. Furthermore, the framework of this invention has good versatility and can be extended to other knowledge-intensive fields such as medicine, education, and finance, achieving cross-domain automated capability assessment.

[0042] Table 1. Performance Evaluation Results of Mainstream Large Language Models under the Automatic Evaluation Framework

[0043] Corresponding to the aforementioned embodiment of an automated evaluation method for large models based on multi-agent collaboration, the present invention also provides an embodiment of an automated evaluation device for large models based on multi-agent collaboration.

[0044] See Figure 4 The present invention provides an automated evaluation device for large models based on multi-agent collaboration, comprising a memory and one or more processors. The memory stores executable code, and when the processor executes the executable code, it is used to implement an automated evaluation method for large models based on multi-agent collaboration as described in the above embodiments.

[0045] Table 2. Description of 27 sub-abilities based on the improved Bloom cognitive classification system

[0046] The embodiment of the large-scale automated evaluation device based on multi-agent collaboration provided by this invention can be applied to any device with data processing capabilities, such as a computer. The device embodiment can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data-processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution. From a hardware perspective, such as... Figure 4 The diagram shown is a hardware structure diagram of any device with data processing capabilities, which is the automated evaluation device for large models based on multi-agent collaboration provided by the present invention. (Except for...) Figure 4 In addition to the processor, memory, network interface, and non-volatile memory shown, any data processing device in the embodiment may also include other hardware depending on the actual function of the data processing device, which will not be described in detail here.

[0047] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0048] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0049] This invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements an automated evaluation method for large models based on multi-agent collaboration as described in the above embodiments.

[0050] The computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0051] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the aforementioned method for automated evaluation of large models based on multi-agent cooperation.

[0052] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0053] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. This application is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A method for automatic evaluation of large models based on multi-agent collaboration, characterized in that, The method includes: Semantic fragments are extracted from unstructured text in the target domain, and these semantic fragments are sampled. Cognitive abilities are divided into six levels and several sub-abilities. Based on the weighted results of the target distribution, current distribution, and context-related distribution, the target ability label corresponding to the task is determined. The large model to be evaluated is then used to generate the task based on the target ability label and semantic fragments. The Socratic question-and-answer mechanism is used to conduct multiple rounds of interaction with the generated task, and key information from the multiple rounds of dialogue is extracted for knowledge expansion. Multi-agent collaboration generates high-quality task instances as an evaluation set; The evaluation set is categorized into tasks, and the model's performance score is calculated for each task.

2. The automatic evaluation method for large models based on multi-agent cooperation according to claim 1, characterized in that, The step of extracting semantic fragments from unstructured text in the target domain and sampling the semantic fragments includes: collecting unstructured text data from academic papers, reports, and technical documents in the target domain; performing sentence segmentation and semantic fragmentation on the text; and sampling the semantic fragments using the K-Center-Greedy algorithm.

3. The automatic evaluation method for large models based on multi-agent collaboration according to claim 1, characterized in that, The specific division of cognitive abilities into six levels and several sub-abilities is as follows: Based on the improved Bloom cognitive classification model, a hierarchical distribution of abilities is constructed, and the cognitive process is divided into six levels and 27 fine-grained sub-abilities: memory, understanding, application, analysis, evaluation and creation.

4. The automatic evaluation method for large models based on multi-agent cooperation according to claim 1, characterized in that, The step of determining the target capability label corresponding to the task based on the weighted result of the target distribution, the current distribution, and the context-related distribution includes: The target distribution is obtained by normalizing the capability weight vectors defined by domain experts. The frequency vector maintained during the iterative construction of the task set is normalized and used as an existing distribution. The ranking vector of each capability in the large model, sorted by relevance from high to low, is transformed into a probability distribution using exponential decay and normalization. ; The calculation of the chosen distribution is as follows: in To balance the trade-off between context alignment and coverage underrepresentation, the following was calculated: Then, the two items with the highest probabilities are selected as the target capability labels for this domain scenario, and the frequency vector is updated.

5. The automatic evaluation method for large models based on multi-agent cooperation according to claim 1, characterized in that, The task of generating data from the target capability label and semantic fragments using the large model to be evaluated includes: By combining semantic fragments and their target capability labels, the large language model is input to generate task names, task descriptions, and question types; semantic similarity is calculated on the generated task set, and duplicate or semantically overlapping tasks are removed; a preliminary task list is output to provide input for subsequent context expansion.

6. The automatic evaluation method for large models based on multi-agent cooperation according to claim 1, characterized in that, The use of the Socratic question-and-answer mechanism to perform multi-round interactions with the generated task includes: Constructing teacher and student intelligent agents; The teacher agent poses higher-level questions based on the task description, and the student agent provides preliminary answers. The teacher's intelligent agent dynamically generates more in-depth follow-up questions based on the students' answers, forming a recursive question-and-answer chain; The intelligent agent extracts and summarizes the content of multiple rounds of question-and-answer sessions to form a set of key concepts.

7. The automatic evaluation method for large models based on multi-agent collaboration according to claim 1, characterized in that, The process of extracting key information from multi-turn dialogues for knowledge expansion includes: converting key information into vector representations; Based on the Sentence-BERT semantic retrieval model, text fragments with semantic similarity to concepts are retrieved from an external corpus; Add the search results to the task context to form an expanded set of task knowledge; stop expanding if the number of new knowledge points is below a set threshold or the dialogue enters a repetitive phase.

8. The automatic evaluation method for large models based on multi-agent cooperation according to claim 1, characterized in that, The multi-agent collaboration generates high-quality task instances as an evaluation set, including: Construct a multi-agent system including teacher agents, student agents, review agents, and meta-detection agents; The teacher agent generates test questions based on the task and context, and the student agent generates the answers. The reviewing agent evaluates the answers based on four criteria: correctness, logic, clarity, and difficulty. If the score reaches the threshold, the review agent will provide feedback to the teacher agent indicating an increased difficulty level, which will guide the generation of more challenging questions. If the score is not up to standard, the review agent will trigger the context extension module to guide a new semantic retrieval through feedback information; The meta-detection agent monitors the convergence of multi-agent dialogue and the deviation from the task objective, and stops the interaction when the convergence condition is met. All question-answer pairs are quality-scored and deduplicated to construct the final evaluation instance set.

9. The automatic evaluation method for large models based on multi-agent cooperation according to claim 1, characterized in that, The process of classifying the evaluation set into tasks and calculating the model's performance score on each task includes: For multiple-choice and true / false questions, the accuracy rate is used to evaluate the model's basic recognition ability; for classification tasks, the macro-average F1 score is used to mitigate the bias caused by label imbalance; for open-ended question-answering tasks, binary keypoint matching and a 5-point Likert score are combined. The final score of the question-answering task is a weighted fusion of the two parts. Based on the cognitive ability labels of the task, the score of each task is mapped to the corresponding ability label. The score of each cognitive ability is calculated by weighted averaging to obtain the model's ability vector. The ability vector is then transformed into an ability distribution histogram and an ability radar chart.

10. An automatic evaluation device for large models based on multi-agent cooperation, comprising a memory and one or more processors, wherein the memory stores executable code, characterized in that, When the processor executes the executable code, it implements a method for automatic evaluation of large models based on multi-agent cooperation as described in any one of claims 1-9.

Images