A large model evaluation method, system and computer device

Abstract

The method comprises the following steps: constructing a large model evaluation workflow; obtaining a large model test set corresponding to a target evaluation field through a test set acquisition node; determining a test dimension meeting a preset evaluation quality index according to the large model test set through a test dimension determination node; sampling evaluation samples in the large model test set according to the test dimension to obtain a target test set through a target test set determination node; generating an evaluation script through an evaluation code generation node; and evaluating a to-be-evaluated large model by using the target test set to obtain an evaluation result. The method provided by the present disclosure selects a corresponding target test set based on a target evaluation field, thereby avoiding the problem of insufficient evaluation of the to-be-evaluated large model; reduces the waste of computing power and improves the evaluation efficiency; and evaluates the to-be-evaluated large model through multiple test dimensions to obtain an evaluation result with higher credibility.

Description

Technical Field

[0001] This disclosure relates to the field of large model evaluation, and more particularly to a method, system and computer equipment for large model evaluation. Background Technology

[0002] Currently, in the research and application of large language models, multimodal large models, and embodied intelligence models, the mainstream evaluation paradigm relies on manually constructed evaluation datasets. This involves pre-defining task dimensions, selecting samples, and establishing standard answers, then scoring the model's output through comparison. However, with the further development of large models, existing evaluation methods have revealed significant limitations in practical applications: Firstly, it is difficult to balance evaluation efficiency and cost control. A single evaluation dataset can usually only cover a part of the model's capabilities. If multiple datasets are used for joint evaluation, the problem of overlapping capability dimensions and homogeneous sample distribution often arises, leading to a waste of computing power. In multimodal and embodied intelligence scenarios, the mainstream video uniform sampling or full-frame input strategy results in high inference costs. Especially in long videos or complex interaction scenarios, the surge in the number of visual tokens will bring high computational overhead, resulting in high inference costs and low evaluation efficiency.

[0003] Secondly, the evaluation system relies on human experience in terms of dimensional design and evaluation indicators. The existing system is overly dependent on single indicators such as accuracy or success rate, lacks fine-grained analysis of the model's sub-capabilities, and is unable to provide accurate feedback for the optimization of large models through attribution analysis.

[0004] In summary, existing evaluation techniques have significant shortcomings in terms of efficiency, cost, fine-grained diagnostics, and consistency with real-world scenarios, making it difficult to support the in-depth evaluation needs of large models evolving into complex application scenarios. Therefore, there is an urgent need for a comprehensive evaluation method that can significantly reduce labor and computational costs while ensuring reproducibility and stability of the evaluation, and provide evaluation conclusions with high credibility and consistency. Summary of the Invention

[0005] This disclosure provides a method, system, and computer device for evaluating large models, which addresses the issues of cost and reliability in existing evaluation methods.

[0006] In view of the above problems, firstly, this disclosure provides a large model evaluation method, including: A large model evaluation workflow is constructed, comprising: a test set construction sub-workflow and an evaluation execution sub-workflow; the test set construction sub-workflow includes: a test set acquisition node, a test dimension determination node, and a target test set determination node; the evaluation execution sub-workflow includes: an evaluation code generation node and an evaluation execution node. The test set acquisition node is used to obtain a large model test set corresponding to the target evaluation domain; Based on the large model test set, the test dimensions that meet the preset evaluation quality indicators are determined by the test dimensions. The target test set is obtained by sampling the evaluation samples in the large model test set according to the test dimension, based on the node determined by the target test set; The large model to be tested and the target test set are used as inputs to the evaluation execution sub-workflow, and the evaluation script is generated through the evaluation code generation node. The evaluation script is run through the evaluation execution node, and the target test set is used to evaluate the large model to be tested, so as to obtain the evaluation results.

[0007] In conjunction with the first aspect, in one possible implementation, obtaining a large model test set corresponding to the target evaluation domain through the test set acquisition node includes: The description of the target evaluation domain is used as the input to the test set acquisition node; the retrieval model is invoked to encapsulate the description of the target evaluation domain into a retrieval instruction, and a large model test set associated with the description of the target evaluation domain is retrieved from the preset data source; and / or Input a pre-determined large model test set associated with the target evaluation domain into the test set acquisition node.

[0008] In conjunction with the first aspect, in one possible implementation, the test dimension determination node includes: a candidate dimension determination node and a test dimension evaluation node; The step of determining nodes through the test dimensions, based on the large model test set, involves determining test dimensions that meet preset evaluation quality indicators, including: The large model test set is used as input to determine the nodes for candidate dimensions; Nodes are determined using the candidate dimensions, and a predetermined number of evaluation samples are extracted from the large model test set as sample examples; and The large language model is invoked, and based on the metadata, target evaluation domain, and sample examples of the large model test set, the test dimensions covered by the large model test set are identified to obtain a set of candidate test dimensions. The test dimensions include at least one of the following: dimension name, dimension explanation, typical sample features, exclusion conditions, and suggested sample size quota or priority. The test dimension evaluation node calls the dimension evaluation model to filter the candidate test dimension set and obtain test dimensions that meet the preset evaluation quality indicators.

[0009] In conjunction with the first aspect, in one possible implementation, the step of using the test dimension evaluation node to call the dimension evaluation model to filter the candidate test dimension set and obtain test dimensions that meet preset evaluation quality indicators includes: Verify the semantic relevance of each test dimension to the target evaluation domain, and remove test dimensions that are outside the coverage of the target evaluation domain; Calculate the semantic overlap between each test dimension, and perform aggregation and / or deduplication operations on test dimensions with an overlap higher than the first threshold; Evaluate the executable logic of each test dimension and eliminate test dimensions that cannot produce execution results; Detect the distribution density of each test dimension under each category, and adjust the proportion of test dimensions clustered under the preset category; Based on the mapping relationship between test dimensions and evaluation samples in the large model test set, test dimensions with fewer than the second threshold are removed.

[0010] In conjunction with the first aspect, in one possible implementation, the target test set determination node includes: a sample dimension mapping node and a dimension-based sampling node; The step of determining nodes through the target test set and sampling evaluation samples in the large model test set according to the test dimension to obtain the target test set includes: The test dimensions and large model test set that meet the preset evaluation quality indicators are used as the inputs to the sample dimension mapping nodes; The large language model is invoked through the sample dimension mapping node. Based on the semantics of each evaluation sample in the large model test set, the test dimension corresponding to each evaluation sample is determined. For evaluation samples with multi-dimensional labels, the corresponding test dimension is limited to a preset number, or the main dimension is determined as the final dimension. By sampling nodes by dimension, clustering and / or deduplication operations are performed on the evaluation samples of each test dimension based on the similarity between the evaluation samples corresponding to each test dimension, resulting in a collection of evaluation samples corresponding to each test dimension. According to the preset quota, the evaluation samples in each evaluation sample set are stratified and sampled, and the sampled evaluation samples are determined as the target test set; the preset quota includes at least one of the following: dimension quota, difficulty quota and source quota.

[0011] In conjunction with the first aspect, in one possible implementation, the step of using the large model to be tested and the target test set as inputs to the evaluation execution sub-workflow, and generating an evaluation script through the evaluation code generation node, includes: Call the large language model to generate evaluation criteria for the large model under test; the evaluation criteria shall include at least one of the following: the scoring criteria for each class of samples, the fault tolerance rules, the output parsing strategy, the aggregation method, and the error handling principles; Based on the evaluation criteria, an evaluation script is generated to call the large model under test and load the target test set for evaluation.

[0012] In conjunction with the first aspect, in one possible implementation, the step of running the evaluation script through the evaluation execution node, evaluating the large model to be tested using the target test set, and obtaining evaluation results includes: The evaluation script is executed in a pre-defined isolated environment. Based on the model information of the large model to be tested in the evaluation script, the large model to be tested is called in a preset isolation environment; In the preset isolation environment, the evaluation samples in the target test set are loaded and executed one by one to obtain the output results corresponding to each evaluation sample; Based on the obtained output results, and using the evaluation criteria in the evaluation script, the output results of the large model under test are scored for each evaluation sample. Based on the correspondence between the evaluation samples and the test dimensions, the scores of the large model under test on each test dimension are summarized to obtain the evaluation results of the large model under test.

[0013] In conjunction with the first aspect, in one possible implementation, the test set construction sub-workflow further includes at least one of the following: test set quality control points, dimension quality control points, and sample quality control points; The method further includes: The test set quality control point determines whether the number of large model test sets acquired by the test set acquisition node is lower than a preset first control value; if the number of large model test sets is lower than the preset first control value, a test set expansion instruction is generated to control the test set acquisition node to increase the number of large model test sets acquired. The dimensional quality control points determine whether the coverage of the test dimension that meets the preset evaluation quality indicators is lower than the preset second control value for the target evaluation domain; if the coverage of the test dimension is lower than the preset second control value, a test dimension expansion instruction is generated to control the test dimension determination node to call the large language model to generate a new test dimension; The sample quality control point determines whether the number of samples corresponding to each test dimension is lower than the preset third control value; if the number of evaluation samples corresponding to any test dimension is lower than the preset third control value, an evaluation sample expansion instruction is generated to control the target test set determination node to call the large language model to generate new evaluation samples.

[0014] In conjunction with the first aspect, in one possible implementation, the evaluation execution sub-workflow further includes: a sample compression node; The method further includes: The evaluation samples of the target test set are compressed using the sample compression node, including: Perform keyframe selection operations on video samples in the target test set; and / or Constraints are imposed on the resolution and sampling number of the image samples in the target test set; and / or Perform truncation on long text samples in the target test set to preserve key semantic paragraphs; and / or Perform turn merging and / or turn compression operations on the multi-turn dialogue samples in the target test set.

[0015] In conjunction with the first aspect, in one possible implementation, the evaluation execution sub-workflow further includes: evaluation script quality control points; The method further includes: The output of the evaluation execution node is detected by the quality control point of the evaluation script; if erroneous data is detected in the output of the evaluation execution node, an evaluation script error report is generated based on the erroneous data, and the evaluation script error report is output to the evaluation code generation node; The evaluation code generation node adjusts the evaluation script based on the evaluation script error report, and outputs the adjusted evaluation script to the evaluation execution node. The current evaluation script will be fixed until no error data is detected in the output of the evaluation execution node.

[0016] A second aspect of this disclosure provides a large model evaluation system, including: Build modules are used to construct large model evaluation workflows; The acquisition module is used to acquire a large model test set corresponding to the target evaluation domain through the test set acquisition node; The determination module is used to determine the test dimensions that meet the preset evaluation quality indicators based on the large model test set. The sampling module is used to sample the evaluation samples in the large model test set according to the test dimension by determining the nodes through the target test set, so as to obtain the target test set; The generation module is used to take the large model to be tested and the target test set as inputs to the evaluation execution sub-workflow, and generate the evaluation script through the evaluation code generation node; The evaluation module is used to run the evaluation script through the evaluation execution node, evaluate the large model to be tested using the target test set, and obtain the evaluation results.

[0017] This disclosure provides a computer device including a processor, a memory, and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor communicates with the memory via the bus, and when the machine-readable instructions are executed by the processor, they perform the steps of a large model evaluation method as described in any of the first aspects.

[0018] The beneficial effects of the embodiments disclosed herein include: This disclosure provides a method, system, and computer device for evaluating large models. The method includes: constructing a large model evaluation workflow; the large model evaluation workflow includes a test set construction sub-workflow and an evaluation execution sub-workflow; the test set construction sub-workflow includes a test set acquisition node, a test dimension determination node, and a target test set determination node; the evaluation execution sub-workflow includes an evaluation code generation node and an evaluation execution node; the test set acquisition node acquires a large model test set corresponding to the target evaluation domain; the test dimension determination node determines test dimensions that meet preset evaluation quality indicators based on the large model test set; the target test set determination node samples evaluation samples in the large model test set according to the test dimensions to obtain a target test set; the large model to be tested and the target test set are used as inputs to the evaluation execution sub-workflow, and an evaluation script is generated through the evaluation code generation node; the evaluation script is run through the evaluation execution node, and the large model to be tested is evaluated using the target test set to obtain evaluation results. The large model evaluation method disclosed herein, from determining the large model test set and selecting test dimensions to running the evaluation script to test the large model under test, is implemented through an automated pipeline of the large model evaluation workflow, without relying on human experience. When determining the large model test set, the test set acquisition node in the automated pipeline automatically selects the corresponding target test set based on the target evaluation domain, avoiding the problem of insufficient assessment of the large model's capabilities using a single evaluation dataset in existing technologies. When selecting test dimensions, the test dimension determination node in the automated pipeline determines test dimensions that meet preset evaluation quality indicators, avoiding the problems of overlapping capability dimensions and homogeneous sample distribution faced by joint evaluation using multiple datasets, reducing computational waste and improving evaluation efficiency. Furthermore, the method adopted in this disclosure evaluates the large model under test through multiple test dimensions, thereby obtaining more reliable evaluation results. Attached Figure Description

[0019] Figure 1 A flowchart illustrating a large model evaluation method provided in this embodiment of the disclosure; Figure 2 This is a schematic diagram of the large model evaluation workflow structure provided in the embodiments of this disclosure; Figure 3 This is a schematic diagram of the structure of a large model evaluation system provided in an embodiment of the present disclosure. Detailed Implementation

[0020] This disclosure provides a large model evaluation method, system, and computer device. Preferred embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustrative and explanatory purposes only and are not intended to limit this disclosure. Furthermore, the embodiments and features described in this application can be combined with each other unless otherwise specified.

[0021] This disclosure provides a method for evaluating large models, such as... Figure 1 As shown, it can be implemented as follows: S101. Construct a large-scale model evaluation workflow; The large model evaluation workflow includes: a test set construction sub-workflow and an evaluation execution sub-workflow; the test set construction sub-workflow includes: a test set acquisition node, a test dimension determination node, and a target test set determination node; the evaluation execution sub-workflow includes: an evaluation code generation node and an evaluation execution node; S102. Obtain a large model test set corresponding to the target evaluation domain through the test set acquisition node; S103. Determine the nodes through the test dimensions. Based on the large model test set, determine the test dimensions that meet the preset evaluation quality indicators. S104. The target test set is obtained by sampling the evaluation samples in the large model test set according to the test dimension through the node determined by the target test set. S105. The large model to be tested and the target test set are used as inputs to the evaluation execution sub-workflow, and the evaluation script is generated through the evaluation code generation node; S106. Run the evaluation script through the evaluation execution node, and use the target test set to evaluate the large model to be tested, and obtain the evaluation results.

[0022] In this embodiment of the disclosure, the large model evaluation workflow can be an executable program deployed on a computer system. This executable program encapsulates a series of automated procedures from data input to obtaining the evaluation results of the large model. According to the program settings within the large model evaluation workflow, various inputs set by the program can be invoked during the execution of the workflow to evaluate the large model under test, and finally, the evaluation results are output.

[0023] It should be noted that the large model to be tested in this disclosure can refer to a large-scale parameterized pre-trained model based on a neural network architecture. The large model has the ability to extract semantic features, perform logical reasoning, and generate instructions from multimodal data or text data. It can generate output results that meet specific business logic by loading a preset instruction template.

[0024] For example, a large model that processes only text data is called a Large Language Model (LLM). A Large Language Model can refer to a parameterized computational model based on a deep neural network architecture and pre-trained on a large-scale natural language corpus. Large Language Models can capture semantic dependencies in text sequences through attention mechanisms and can be configured to understand natural language instructions, perform logical reasoning, and execute various natural language processing tasks, including text generation, summarization, and language translation.

[0025] For example, a large model capable of processing multimodal data can be called a multimodal large model. A multimodal large model can refer to an integrated large model that can simultaneously process, understand, and associate two or more different types of modal data (e.g., text, images, video, and audio). It typically contains multiple modality-specific encoders and a cross-modal alignment layer, and can be configured to map heterogeneous features from different modalities to a unified semantic feature space, thereby enabling cross-modal joint reasoning and content generation.

[0026] The large model to be tested referred to in this disclosure may refer to, but is not limited to, a multimodal large model to be tested or a large language model to be tested.

[0027] In this disclosure, nodes are functional units in the architecture of large model evaluation workflow. Each node encapsulates a specific program that is responsible for calling the corresponding input, processing the data, and passing the processed data to the next node.

[0028] Specifically, such as Figure 2 As shown, the large model evaluation workflow can be divided into two sub-workflows. The first is the test set construction sub-workflow, which takes a description of the target evaluation domain as input and generates a target test set for evaluating the large model under test by calling various models. The evaluation domain refers to the coverage of the large model under test in a specific knowledge vertical classification or application scenario. The description of the target evaluation domain can be the name of the domain to be evaluated (e.g., embodied family tasks, medical question answering, and traffic scenario understanding), or it can be a more granular task description (e.g., multi-step task planning and execution feasibility in an indoor family scenario).

[0029] The second is the evaluation execution sub-workflow, which takes the large model to be tested and the target test set as input, and calls the program in the sub-workflow to execute the target test set on the large model to be tested, thereby obtaining the evaluation results.

[0030] Each sub-workflow contains multiple nodes to achieve its task objectives. The test set construction sub-workflow includes: a test set acquisition node, a test dimension determination node, and a target test set determination node. The test set acquisition node retrieves a large model test set corresponding to the target evaluation domain by executing a program based on the input target evaluation domain. This large model test set is designed to evaluate model performance and includes a collection of multiple evaluation samples and corresponding standard outputs. The test dimension determination node extracts all test dimensions from the selected large model test set. Test dimensions refer to the model's various sub-capabilities within the target evaluation domain (e.g., in the medical question-answering domain, capabilities in clinical diagnosis, pharmacology, medical record writing, and medical ethics). The target test set determination node samples the evaluation samples from the large model test set based on the extracted test dimensions to obtain the final target test set.

[0031] The evaluation execution sub-workflow includes an evaluation code generation node and an evaluation execution node. The evaluation code generation node can generate a targeted evaluation script based on the input large model to be tested and the target test set. The evaluation execution node can execute the generated evaluation script, call the large model to be tested and traverse the evaluation samples in the target test set according to the content of the evaluation script, evaluate the large model to be tested based on the output results of the large model to be tested, and obtain the evaluation results.

[0032] Based on the above, this disclosure achieves the entire large model evaluation process through an automated pipeline of the large model evaluation workflow, without relying on human experience. When determining the large model test set, the test set acquisition node in the automated pipeline automatically selects the corresponding target test set based on the target evaluation domain, avoiding the problem of insufficient assessment of the capabilities of the large model under test using a single evaluation dataset in existing technologies. When selecting test dimensions, the test dimension determination node in the automated pipeline determines test dimensions that meet preset evaluation quality indicators, avoiding the problems of overlapping capability dimensions and homogeneous sample distribution faced by joint evaluation using multiple datasets, reducing computational waste and improving evaluation efficiency. Furthermore, the method adopted in this disclosure evaluates the large model under test through multiple test dimensions, thereby obtaining more reliable evaluation results.

[0033] It should be noted that the workflows in this disclosure can be executed by intelligent agents deployed on computer devices (including but not limited to servers, terminal devices, or cloud computing platforms). The intelligent agents in this disclosure can be configured to invoke hardware resources and specific models according to the logic defined in the workflow, enabling them to autonomously complete specific tasks without external continuous instructions.

[0034] The method provided in this disclosure can be implemented using a centralized intelligent agent or a distributed implementation. For example, the test set construction sub-workflow and the evaluation execution sub-workflow can be deployed as independent services, or the tasks carried by different nodes can be executed separately. As long as the functional division and process logic of each implementation method are maintained, it can still achieve the technical effect of reducing labor costs and improving evaluation efficiency.

[0035] In another embodiment provided in this disclosure, the above step S102, "obtaining a large model test set corresponding to the target evaluation domain through the test set acquisition node," can be implemented as follows: Step 1: Use the description of the target evaluation domain as input to the test set acquisition node; call the retrieval model to encapsulate the description of the target evaluation domain into a retrieval instruction, and retrieve the large model test set associated with the description of the target evaluation domain from the preset data source; and / or Step 2: Input the pre-determined large model test set associated with the target evaluation domain into the test set acquisition node.

[0036] In this embodiment, a Large Language Model Retrieval API (LLMRetrieval API) can be set for the test set acquisition node. This API allows the retrieval of a large model test set from a preset data source. The API serves as a programming interface for the large language model to access, search, and retrieve external knowledge bases. The API accepts input descriptions of the target evaluation domain in natural language form. Within the large language model, these descriptions are vectorized and encapsulated into vectors that the computer system can understand. The system then retrieves content spatially similar to these vectors from the preset data source, thus enabling the retrieval of the corresponding large model test set based on the semantics of the target evaluation domain.

[0037] The preset data source here can be an external public data source or a pre-maintained internal asset repository. These asset repositories or public data sources store a large number of test sets (Benchmarks) for the model. Based on the description of the target evaluation domain, the retrieval API can be used to retrieve test sets that are semantically related to the target testing domain and obtain the metadata feedback of the retrieved test sets. The test set metadata here can include at least one of the following: Benchmark name / version, task type, input / output format, sample examples, and accessible sample paths or indexes, and may also include descriptions of known capability dimensions.

[0038] Among them, the Benchmark name / version refers to the identifier of different test sets within the preset data source. These test sets are usually designed manually by domain experts or researchers based on experience, and samples are selected from real data, simulation environments or public videos. Then, standard answers are generated through manual annotation or rule-based generation, and finally, different test sets are collected.

[0039] Task type refers to the operational mode required to execute the model in this test set (e.g., single-choice, extractive question answering, generative summarization, or code completion). Input / output format defines a structured template for the sample input and output data of the model. The large model under test can read samples based on this format and generate output results that conform to the format. Sample example refers to one or more specific evaluation samples extracted from the test set, used to assist the large model in understanding the content of the test set through its API retrieval. Accessible sample path or index refers to the address of the evaluation samples in the test set on the physical storage medium, based on which these evaluation samples can be accessed. Known capability dimension description refers to the pre-determined test dimensions for the evaluation samples within this test set. If these test dimensions are already included in the metadata, the evaluation samples can be determined as these pre-determined test dimensions in subsequent processes.

[0040] It should be noted that in some implementations, the test set acquisition node can also acquire test sets provided by the user or use data sources specified by the user. In this case, the test set acquisition node can only perform validation and standardization on the user-provided test set within the scope specified by the user to obtain a large model test set, or it can mix the user-provided test set with the large model test set acquired by the test set acquisition node and use them in subsequent tests.

[0041] Alternatively, a combination of traditional information retrieval systems and large-scale models can be used to obtain a large-scale model test set. For example, multiple candidate test sets related to the retrieval target can be recalled from a large-scale data warehouse through keyword retrieval, vector retrieval, or rule filtering. Then, the large-scale model performs semantic reordering and filtering on the recalled results to finally obtain a large-scale model test set that is highly relevant to the target evaluation domain.

[0042] The above method can reduce the cost of manual screening and overcome the problem of relying on manual selection of test sets in existing technologies.

[0043] In yet another embodiment provided in this disclosure, the test dimension determination node includes: a candidate dimension determination node and a test dimension evaluation node; In step S103 above, "determining the test dimension that meets the preset evaluation quality index based on the large model test set" can be implemented as follows: Step 1: Using the large model test set as input for the candidate dimension determination node, and extracting a predetermined number of evaluation samples as sample examples from the large model test set using the candidate dimension determination node; and Step 2: Invoke the large language model, and based on the metadata, target evaluation domain, and sample examples of the large model test set, identify the test dimensions covered by the large model test set to obtain a set of candidate test dimensions; each test dimension shall include at least one of the following: dimension name, dimension explanation, typical sample features, exclusion conditions, and suggested sample size quota or priority. Step 3: Use the test dimension evaluation node to call the dimension evaluation model to filter the candidate test dimension set and obtain the test dimensions that meet the preset evaluation quality indicators.

[0044] In this embodiment, the candidate dimension determination node can receive the large model test set obtained by the test set acquisition node as input, and set a large oracle model for the candidate dimension determination node. First, a predetermined number of evaluation samples are extracted from each large model test set as sample examples. Then, the metadata of the large model test set, the extracted evaluation samples, and the target evaluation domain are packaged into a collection and input into the large language model. Based on this content, the large language model identifies the test dimensions covered by the large model test set and outputs the test dimensions in a predetermined format. The extracted sample examples are used to assist the large language model in understanding the type of evaluation samples and the dimensions involved in each large model test set. The predetermined number can be set as needed; a larger number of sample examples is more beneficial for the large language model to identify the test dimensions of the large model test set, but it requires more computing power.

[0045] After receiving the above content, the large language model can output the identified test dimensions according to the format constraints of the prompt engineering (Prompt) and output the definition of each test dimension according to the format. In one possible implementation, the definition of the test dimension may include the following: dimension name, dimension explanation, typical example features, exclusion conditions, suggested sample size quota or priority, etc. Here, the dimension name refers to the representative label of the test dimension. The dimension explanation is a detailed definition of the test dimension, which may include the model capabilities examined by the test dimension. Typical example features refer to the typical features of the samples of the test dimension, which can be a specific feature that can be recognized by the computer system. Exclusion conditions are negative constraints of the test dimension to clearly define the boundary between the test dimension and similar test dimensions. Suggested sample size quota or priority refers to the sampling weight of the evaluation samples given according to the importance of the test dimension in the target evaluation domain. These test dimensions identified and extracted by the large language model can be used as a set of candidate test dimensions as input to the test dimension evaluation nodes. The test dimension evaluation node calls the dimension evaluation model to filter the test dimensions in the candidate test dimension set according to the pre-set evaluation quality indicators.

[0046] This disclosure sets up a dimension evaluation model for the test dimension evaluation nodes. By introducing a test dimension as an intermediate abstraction and using the dimension evaluation model to filter and optimize the test dimensions, the structure of the large model test set is no longer just a simple collection of samples, but a hierarchical structure with clearly defined semantic coverage capabilities. The preset evaluation quality indicators include multiple pre-determined standards, which aim to determine whether the test dimensions in the candidate test dimension set can support a comprehensive evaluation of the module under test.

[0047] In another embodiment provided in this disclosure, step three above, "using the dimension evaluation model through the test dimension evaluation node to filter the candidate test dimension set and obtain test dimensions that meet the preset evaluation quality indicators," can be implemented as follows: Step 1: Verify the semantic relevance of each test dimension to the target evaluation domain, and remove test dimensions that are outside the coverage of the target evaluation domain; Step 2: Calculate the semantic overlap between each test dimension, and perform aggregation and / or deduplication operations on test dimensions with an overlap higher than the first threshold; Step 3: Evaluate the executable logic of each test dimension and eliminate test dimensions that cannot produce execution results; Step 4: Detect the distribution density of each test dimension under each category, and adjust the proportion of test dimensions clustered under the preset category; Step 5: Based on the mapping relationship between the test dimensions and the evaluation samples in the large model test set, remove test dimensions whose number of evaluation samples is lower than the second threshold.

[0048] In this embodiment of the disclosure, the dimensional evaluation model can be a small architecture model specifically trained for judging test dimensions; it can also be a large language model with strong language capabilities; or it can be a combination of preset rules and models, in which the candidate test dimension set is filtered by using preset rules with methods such as keywords or regular expressions, and then the model judges the filtered content.

[0049] In practical applications, the dimensional evaluation model can also be implemented as a collaborative scoring system composed of multiple models. For example, multiple different large models can screen the candidate test dimension set separately, and then determine the final test dimension that meets the preset evaluation quality indicators through voting, consistency detection or cross-review.

[0050] Among the methods described above, using a small-scale model for discrimination offers higher screening efficiency and lower cost compared to using a large language model, but its generalization ability is lower, and it cannot cover multiple target evaluation domains without training. Using a large language model for screening offers advantages such as strong comprehension and high flexibility, adapting to prompts, but its invocation cost is higher. A method combining rules and models requires specially designed architecture and filtering rules. In practical applications, different dimensions of the evaluation model can be selected as needed.

[0051] The preset evaluation quality indicators should at least test the following: the relevance of the test dimensions to the target evaluation domain, the overlap between test dimensions, the executability of the test dimensions, the coverage of the test dimensions in the target evaluation domain, and the evaluation sample size of each test dimension in the large model test set.

[0052] The relevance of test dimensions to the target evaluation domain can be determined by verifying the semantic relevance of each test dimension in the candidate test dimension set to the target evaluation domain. The overlap between test dimensions can be determined based on the semantic independence of each test dimension; test dimensions with an overlap exceeding a first threshold can be considered the same test dimension, allowing for aggregation or removal of one of them. The executability of a test dimension can be determined by judging whether the large model under test for that test dimension can output quantifiable results. The coverage of test dimensions in the target evaluation domain can be determined by determining the clustering of each test dimension within different categories of the target evaluation domain, avoiding clustering of test dimensions into undefined categories. The number of evaluation samples for each test dimension in the large model test set can be determined based on the mapping relationship between the number of evaluation samples for each test dimension in the large model test set; test dimensions with a number of evaluation samples less than a second threshold can be considered to have too few samples, and effective evaluation results cannot be obtained under this dimension.

[0053] The above process allows for the selection of test dimensions that meet preset evaluation quality metrics from the candidate test dimension set. In practical applications, the rejected test dimensions and their reasons can be output as logs for subsequent iterations. This method effectively avoids the problems of capability duplication and low information density between different test sets in existing technologies. It enables the selection of large model test sets by test dimensions to cover a more comprehensive model capability space with the same or smaller sample size, thereby improving the information content and discriminative power of the evaluation results.

[0054] In yet another embodiment provided in this disclosure, the target test set determination node includes: a sample dimension mapping node and a dimension-based sampling node; In step S104 above, "the nodes are determined by the target test set, and the evaluation samples in the large model test set are sampled according to the test dimension to obtain the target test set," which can be implemented as follows: Step 1: Take the test dimensions that meet the preset evaluation quality indicators and the large model test set as input to the sample dimension mapping node. Call the large language model through the sample dimension mapping node and determine the test dimension corresponding to each evaluation sample according to the semantics of each evaluation sample in the large model test set. For the evaluation samples of multi-dimensional labels, the corresponding test dimensions are limited to a preset number, or the main dimension is determined as the final dimension. Step 2: Based on the similarity between the evaluation samples corresponding to each test dimension, perform clustering and / or deduplication operations on the evaluation samples of each test dimension by sampling nodes according to the dimensions, to obtain the set of evaluation samples corresponding to each test dimension; Step 3: Perform stratified sampling on the evaluation samples in each evaluation sample set according to the preset quota, and determine the sampled evaluation samples as the target test set. The preset quota includes at least one of the following: dimension quota, difficulty quota, and source quota.

[0055] In this embodiment, the sample dimension mapping node can take a large model test set and determined test dimensions as input. A large language model is set up for the sample dimension mapping node. By calling the large language model, the mapping relationship between test dimensions and evaluation samples can be determined based on the semantics of each evaluation sample in the large model test set. It should be noted that if an evaluation sample can correspond to multiple test dimensions, a priority rule can be used to select the primary dimension to avoid statistical and sampling mismanagement.

[0056] After mapping is complete, the mapping relationship between evaluation samples and test dimensions can be used as input to the dimensional sampling nodes. In these nodes, the similarity of evaluation samples can be analyzed based on clustering features, and highly similar samples can be clustered or deduplicated, thereby reducing semantically repetitive evaluation samples and increasing the evaluation information density. After clustering and / or deduplication, a set of evaluation samples corresponding to each test dimension can be obtained. The clustering features can come from text embeddings, image embeddings, or multimodal joint embeddings.

[0057] For each testing dimension, stratified sampling can be performed based on dimension quota, difficulty quota, and source quota to obtain the target test set. Dimension quota refers to the proportion of testing samples corresponding to that dimension within the overall target test set. Difficulty quota refers to the proportion of questions of different difficulty levels within the testing sample set; this difficulty can be determined based on the metadata tags of each testing sample, its length, reasoning steps, etc. Source quota can refer to the proportion of each testing sample in the original test set. Controlling the source proportion maintains the diversity of testing samples and prevents overfitting.

[0058] The target test set can be stored in a uniform format, which may include at least one of the following: sample identification (ID), dimension labels, raw input, reference answer or scoring criteria, data source, difficulty level or confidence level, and necessary additional information (e.g., units, tolerances, and option sets). Specifically, the sample ID is a unique identifier assigned to each evaluation sample within the target test set; the dimension label refers to the test dimension corresponding to the evaluation sample; the raw input refers to the content that needs to be input into the large model to be tested; the reference answer or scoring criteria refers to the standard output used to judge the output result of the large model to be tested; the data source refers to the original test set to which the evaluation sample belongs; and the difficulty level or confidence level refers to the difficulty level of the evaluation sample.

[0059] It should be noted that in practical applications, similarity based on statistical features and clustering methods based on graph structures can also be used, or algorithms such as hierarchical clustering and density clustering can be introduced to reduce semantically repetitive evaluation samples, control the distribution of evaluation samples, and ensure capability coverage. This overcomes the problems of high redundancy and low evaluation information density in existing technologies.

[0060] In another embodiment provided in this disclosure, step S105 above, "taking the large model to be tested and the target test set as inputs to the evaluation execution sub-workflow, and generating the evaluation script through the evaluation code generation node," can be implemented as follows: Step 1: Call the large language model to generate evaluation criteria for the large model under test; The evaluation criteria include at least one of the following: the scoring criteria for each type of sample, the fault tolerance rules, the output parsing strategy, the aggregation method, and the error handling principles; Step 2: Based on the evaluation criteria, generate an evaluation script for calling the large model to be tested and loading the target test set for evaluation.

[0061] In this embodiment of the disclosure, the input to the evaluation code generation node can be the large model to be tested and the target test set. The evaluation code generation node can extract the information of the large model to be tested and the information of the target test set, generate evaluation criteria for the large model to be tested through the large language model set for the evaluation code generation node, and generate an evaluation script for calling the large model to be tested and loading the target test set for evaluation in combination with the evaluation criteria.

[0062] Specifically, the information of the large model under test can include at least one of the following: model invocation method, input format, output format, context constraints and token constraints, inference parameters, and authentication method. The model invocation method defines the location where the large model under test runs (which can include calling the large model from the cloud via HyperText Transfer Protocol / API or local inference) and the protocol for interacting with the large model evaluation workflow. The input format refers to the modality and structure of the data received by the large model under test; depending on the model, it can be set to plain text, images, or video frames as input. The output format refers to the data structure of the results generated by the large model under test, which generally includes free text, structured output in JavaScript Object Notation (JSON), or function calls. Context constraints and token constraints define the maximum amount of information that the interaction with the large model under test can handle; inference parameters can define the generation behavior of the large model under test, for example, controlling the randomness of the model's generation through temperature. The authentication method can define security mechanisms such as keys for model access.

[0063] Evaluation criteria may include at least one of the following: scoring criteria, fault tolerance rules, output parsing strategies, aggregation methods, and error handling principles for each type of sample. Scoring criteria are differentiated scoring logics defined based on the task type of the evaluation samples. For example, for objective evaluation samples, a score is awarded if the result matches the correct output, and no score is awarded if they do not match. For generative evaluation samples, multiple scoring angles can be defined and introduced into the large language model for scoring. Fault tolerance rules refer to the tolerance between the output of the model under test and the standard output result, which may include defining numerical tolerance for numerical precision and defining synonymous expressions for semantic similarity. Output parsing strategies refer to converting the unstructured text output by the model into recognizable structured data; this process can be achieved through JSON field parsing and option mapping. Aggregation methods combine the scattered scores for the evaluation samples into a single metric, which can be achieved using dimensional evaluation, macro averaging, and weighted averaging to integrate the scores of each evaluation sample. Error handling principles refer to the handling logic when the output of the model under test does not conform to the specifications; these erroneous outputs may include defects such as missing fields, incorrect formatting, or empty output.

[0064] An evaluation script is an executable script generated based on evaluation criteria. It typically includes: loading the test set, calling the model inference interface, parsing the model output, calculating scores according to the evaluation criteria, and saving logs and results summaries. Loading the test set is the starting point for the evaluation script execution, guiding the computer system to read data from the target test set to obtain the corresponding evaluation samples. Calling the model inference interface sends the read evaluation samples to the location of the large model under test through a defined interface. Parsing the model output cleans and extracts the output from the large model under test and generates output in a predetermined format. Calculating scores according to the evaluation criteria allows for multi-faceted scoring of the corresponding output of each evaluation sample using the scoring criteria and error tolerance rules within the evaluation criteria.

[0065] It should be noted that the evaluation criteria and evaluation scripts in this disclosure can be generated using a large language model based on the large model to be tested and the target test set. Different large models to be tested correspond to different target test sets, evaluation scripts, and evaluation criteria. In addition, the evaluation scripts can also be generated using templates and large language model completion methods or multi-agent collaborative generation methods to improve the stability of the generation and reduce the workload of manually writing and maintaining the evaluation scripts.

[0066] In another embodiment provided in this disclosure, step S106 above, "running the evaluation script through the evaluation execution node, evaluating the large model to be tested using the target test set, and obtaining the evaluation result," can be implemented as follows: Step 1: Execute the evaluation script in the preset isolation environment; Step 2: Based on the model information of the large model to be tested in the evaluation script, call the large model to be tested in a preset isolation environment; Step 3: In the preset isolation environment, load and execute the evaluation samples in the target test set one by one to obtain the output results corresponding to each evaluation sample; Step 4: Based on the obtained output results and the evaluation criteria in the evaluation script, score the output results of the large model under test for each evaluation sample. Step 5: Based on the correspondence between the evaluation samples and the test dimensions, summarize the scores of the large model under test on each test dimension to obtain the evaluation results of the large model under test.

[0067] In this embodiment of the disclosure, the preset isolation environment can be a sandbox running in a computer environment. The sandbox can provide an isolation environment for the evaluation scripts running in it to control security risks and provide a unified testing environment to maintain the consistency of evaluation results.

[0068] After loading the target test set into the sandbox according to the evaluation script, the large model under test can be called to load the evaluation samples in the target test set one by one, and the large model under test can give the corresponding output results for each evaluation sample.

[0069] Furthermore, each output result is scored based on the evaluation criteria in the evaluation script. Specifically, for objective evaluation samples, scoring is based on the standard results corresponding to the evaluation script. For generated evaluation samples, scoring is based on the relevant scoring angles in the evaluation script by calling the large language model.

[0070] After traversing all the evaluation samples, the scores of the large model under test on that test dimension can be summarized based on the scores of each evaluation sample mapped to each test dimension, and then the evaluation results of the large model under test can be obtained.

[0071] The evaluation results output by the sandbox can include the following: sample score, dimensional summary score, overall score, list of failed samples, and error log. The list of failed samples summarizes evaluation samples whose scores are below a pre-set low score (e.g., zero). Typically, the reason for each failure can also be noted in the list. The error log records abnormal situations in the evaluation script during sandbox operation, which can be used for subsequent adjustments to the evaluation script.

[0072] In yet another embodiment provided in this disclosure, the test set construction sub-workflow further includes at least one of the following: test set quality control points, dimension quality control points, and sample quality control points; Large model evaluation methods also include the following steps: Step 1: Determine whether the number of large model test sets acquired by the test set acquisition node is lower than a preset first control value through the test set quality control point; if the number of large model test sets is lower than the preset first control value, generate a test set expansion instruction to control the test set acquisition node to increase the number of large model test sets acquired. Step 2: Determine whether the coverage of the test dimension that meets the preset evaluation quality index is lower than the preset second control value through the dimensional quality control point; if the coverage of the test dimension is lower than the preset second control value, generate a test dimension expansion instruction to control the test dimension determination node to call the large language model to generate a new test dimension; Step 3: Determine whether the number of samples corresponding to each test dimension is lower than the preset third control value through the sample quality control point; if the number of evaluation samples corresponding to any test dimension is lower than the preset third control value, generate an evaluation sample expansion instruction to control the target test set determination node to call the large language model to generate new evaluation samples.

[0073] In this embodiment of the disclosure, multiple control points can also be set in the workflow to perform quality detection and processing on the acquired test set, the identified test dimensions and evaluation samples.

[0074] The test set quality control point can be set after the test set acquisition node. It determines the acquisition quality by checking whether the number of large model test sets is lower than a pre-set first control value. If the number of large model test sets is lower than the first control value, a test set expansion command can be generated to control the test set acquisition node to perform expanded retrieval, thereby increasing the number of large model test sets acquired.

[0075] Dimension quality control points can be set after the test dimension is determined. The quality of the generated test dimension is judged by checking whether its coverage in the target evaluation domain is lower than a pre-set second control value. If the coverage is lower than the second control value, a test dimension expansion instruction can be generated to control the test dimension determination node to call the large language model to generate new test dimensions, thereby improving the coverage of the test dimension in the target evaluation domain.

[0076] Sample quality control points can be set after the target test set is determined. The clustering of evaluation samples is judged by checking whether the number of evaluation samples clustered in each test dimension is lower than a pre-set third control value. If the number of evaluation samples corresponding to any test dimension is lower than the third control value, an evaluation sample expansion instruction can be generated to control the target test set determination node to call the large language model to generate new evaluation samples. Alternatively, new evaluation samples can be retrieved through the test set acquisition node.

[0077] In another embodiment provided in this disclosure, the evaluation execution sub-workflow further includes: a sample compression node; Large model evaluation methods also include the following steps: The compression operation performed on the evaluation samples of the target test set through the sample compression node can be implemented in the following steps: Step 1: Perform keyframe selection operation on the video samples in the target test set; and / or Step 2: Constrain the resolution and sampling quantity of the image samples in the target test set; and / or Step 3: Perform truncation on the long text samples in the target test set to preserve key semantic paragraphs; and / or Step 4: Perform turn merging and / or turn compression operations on the multi-turn dialogue samples in the target test set.

[0078] In this embodiment, the sample compression node can be set before the evaluation code generation node. This node can reduce inference costs and failure rates without compromising evaluation effectiveness. Specifically, this node can perform compression operations on the evaluation samples in the input target test set. The video sample keyframe selection operation can extract the most representative image frames from the video, preserving the original completion process of the video while reducing the video's data volume. Furthermore, input compression methods based on event detection, action slicing, semantic summarization, or attention can be used to compress the long sequences of original video samples in the target test set into a more compact form.

[0079] Constraining image samples can involve reducing the resolution of images in the evaluation samples or, for multi-image samples, selecting representative images for retention. For long text samples, semantic search can be used to extract key information and truncate secondary content, reducing the data volume while preserving the original important semantics. For multi-turn dialogue samples, summarization can be used to condense the multi-turn dialogue into a single sentence, or the number of dialogue turns can be reduced by discarding non-essential dialogue.

[0080] In addition, the unit formats involved in the samples can be standardized to avoid the large model under test consuming computing power to standardize the unit formats and other content.

[0081] It should be noted that the evaluation samples processed by the sample compression node can maintain the format of the original target test set, compressing only the form of the large input model to be tested. This node can be turned off when computing power is sufficient or when the testing process needs to be completely comparable or consistent with historical benchmarks.

[0082] In another embodiment provided in this disclosure, the evaluation execution sub-workflow further includes: evaluation script quality control points; Large model evaluation methods also include the following steps: Step 1: Detect the output of the evaluation execution node through the evaluation script quality control point; if erroneous data is detected in the output of the evaluation execution node, generate an evaluation script error report based on the erroneous data, and output the evaluation script error report to the evaluation code generation node; Step 2: Based on the error report of the evaluation script, the evaluation script is adjusted by the evaluation code generation node, and the adjusted evaluation script is output to the evaluation execution node; Step 3: Fix the current evaluation script until no error data is detected in the output of the evaluation execution node.

[0083] In this embodiment, the evaluation script quality control point can be set after the evaluation execution node to detect the output of the evaluation execution node. When erroneous data is detected in the output of the evaluation execution node, an evaluation script error report can be generated. The error data may include the error log output by the sandbox and the context of the evaluation script corresponding to the error in the error log.

[0084] Error log entries can include reasons for errors such as syntax errors, missing dependencies, output parsing errors, and crashes caused by boundary samples. Syntax errors refer to code in the evaluation script that does not conform to the rules of the programming language, causing the interpreter or compiler in the sandbox to be unable to understand the script's content. Missing dependencies refer to the sandbox lacking necessary external libraries or toolkits, preventing the evaluation script from running. Output parsing errors refer to the output format of the large model under test not conforming to the script's settings. Crashes caused by boundary samples refer to situations where, upon encountering special evaluation samples, the evaluation script's flawed code logic prevents the sandbox from continuing to execute the script.

[0085] The context of the evaluation script can include the evaluation sample associated with the error, the evaluation criteria performed on the evaluation sample, and the code snippet in which the error occurred.

[0086] The aforementioned error logs and context are output by the evaluation script quality control point to the evaluation code generation node, triggering the evaluation script adjustment process. The current evaluation script is adjusted by calling the large language model according to certain repair principles, which may include a minimum modification strategy. Specifically, the above-mentioned syntax errors and parsing anomalies need to be fixed first; then, dependency missing issues need to be fixed based on the sandbox environment; furthermore, the sandbox's calling issues with the large model under test need to be fixed; finally, the script is input into the sandbox again for execution until the evaluation script can run stably and no error logs are output. During the above process, the number of iterations and rollback strategies can be set (e.g., rolling back to a more conservative script generation rule or rolling back to a pre-defined baseline script) to avoid endless loops. After the evaluation script is stably executed in the sandbox, the current evaluation script can be solidified and saved. The saving format may include saving the final evaluation standard text, the final evaluation code version, runtime environment information, and random seeds and configurations.

[0087] The key technical point of this invention is: (1) The “test dimensions” are changed from being manually determined to objects that can be automatically generated and filtered, and the dimension evaluation model is used to control the dimension quality; (2) Automate the "evaluation sample distribution control" (e.g., sample coverage of the target evaluation domain, sample difficulty, and removal of redundant samples) instead of manually selecting samples; (3) Transform the “evaluation script production and maintenance” into a closed loop of “dynamic generation by large language model according to task, sandbox execution verification and automatic repair”, thereby reducing manual debugging; (4) Introduce optional sample compression nodes (such as selecting / compressing / normalizing keyframes) for multimodal evaluation of the large model under test to ensure that the evaluation process is balanced between cost and stability.

[0088] Compared with the prior art, the present invention has at least the following multi-faceted and hierarchical technical effects. These technical effects are not generated in isolation by a single technical feature, but are achieved by the automated testing data mechanism, the automated evaluation mechanism, and the collaborative closed loop between the two.

[0089] First, at the test data construction level, this invention utilizes "domain semantic parsing—related testing" to... tryThe overall process of "automatic retrieval—automatic generation of test dimensions—control of dimension quality—classification, clustering, and sampling by dimension" transforms the construction of the evaluation dataset (i.e., the target test set in this disclosure) from a process highly dependent on human experience and manual operation to an automated process driven by large models. Compared with existing methods of manually constructing test sets, this invention can quickly generate structured and reusable test sets according to different target evaluation domains without significantly increasing human input, thereby significantly reducing the time and human cost of test set construction and improving the adaptability of the evaluation system to new domains, new tasks, and new models.

[0090] Secondly, this invention introduces the intermediate abstraction of "test dimension" at the test data level, and uses a dimension evaluation model to filter and optimize the test dimension. This makes the structure of the test set no longer just a simple collection of samples, but a hierarchical structure with clearly defined capability coverage semantics. By performing redundancy control and coverage analysis on the test dimension, the problems of capability duplication and low information density between different test sets in existing technologies can be effectively avoided. This allows the target test set to cover a more comprehensive model capability space with the same or fewer sample sizes, thereby improving the information content and discriminative power of the evaluation results.

[0091] Furthermore, this invention performs sample classification at the dimensional level and performs clustering and redundancy removal sampling within each test dimension, ensuring semantic diversity and statistical balance in the evaluation samples. The advantages of this technique are twofold: firstly, it reduces the interference of highly similar samples on the evaluation results, preventing the model from obtaining disproportionate scores on certain repetitive patterns due to a "practice-oriented advantage"; secondly, it ensures that different ability dimensions and difficulty levels have representative samples in the test set, thereby improving the generalization significance of the evaluation conclusions to real-world usage scenarios.

[0092] Furthermore, in multimodal and embodied intelligence evaluation scenarios, this invention introduces an optional "sample compression node" during the evaluation execution phase, enabling the test input to be regularized and compressed according to the evaluation target before being fed into the large model under test. For example, by using keyframe selection, input truncation, or structured regularization, the number of visual or text tokens can be reduced without compromising the effectiveness of the evaluation. This technique can significantly reduce the computational overhead during model inference, allowing multimodal evaluation tasks that were originally computationally expensive and difficult to scale to be completed under controllable resource conditions, thereby improving the deployability of the evaluation system in practical engineering environments.

[0093] Furthermore, this invention automatically generates evaluation criteria and test scripts from large models, and combines sandbox execution and error regeneration mechanisms to enable the large model evaluation process to have adaptive and self-healing capabilities. Compared with existing methods that require manual maintenance of test scripts, this invention can automatically update the corresponding evaluation logic when test data structures change, model output formats change, or evaluation rules are adjusted, reducing evaluation failures or manual intervention caused by script errors. This mechanism not only reduces the maintenance cost of the evaluation system but also improves the stability and robustness of the evaluation process in long-term operation.

[0094] Furthermore, because this invention solidifies the evaluation outputs such as evaluation criteria, evaluation code, operating environment, and configuration parameters during the evaluation process, each evaluation result is reproducible and auditable. This technical effect allows the evaluation results not only to be used for model ranking and comparison, but also to retrospectively analyze the reasons for changes in model performance, providing a reliable basis for model development, optimization, and regression testing.

[0095] From the perspective of evaluation result quality, this invention, through a combination of techniques including explicit modeling of test dimensions, control of sample distribution, and dynamically generated evaluation rules, enables automated evaluation results to more realistically reflect the model's performance across different capability dimensions and difficulty levels. Compared to evaluation methods that rely solely on a single score or test set, the evaluation results obtained by this invention are more interpretable and understandable, and are more likely to align with the subjective judgments of human experts or actual users regarding the model's capabilities, thereby enhancing the credibility of the evaluation conclusions in decision-making scenarios.

[0096] Finally, because the overall technical solution of this invention adopts a modular and process-oriented design, each of its sub-nodes (such as the test set acquisition node, test dimension determination node, target test set determination node, evaluation code generation node, and evaluation execution node) can be replaced or tailored according to specific engineering conditions. Therefore, this invention is not only applicable to the current evaluation of large language models and multimodal large models, but can also be extended to new large model forms or new evaluation tasks in the future, and has good scalability and engineering implementation value.

[0097] This disclosure also provides a large model evaluation system, such as Figure 3 As shown, it includes: Module 301 is used to build a large model evaluation workflow; The large model evaluation workflow includes: a test set construction sub-workflow and an evaluation execution sub-workflow; the test set construction sub-workflow includes: a test set acquisition node, a test dimension determination node, and a target test set determination node; the evaluation execution sub-workflow includes: an evaluation code generation node and an evaluation execution node. The acquisition module 302 is used to acquire a large model test set corresponding to the target evaluation domain through the test set acquisition node; The determination module 302 is used to determine the test dimensions that meet the preset evaluation quality indicators based on the large model test set. The sampling module 304 is used to sample the evaluation samples in the large model test set according to the test dimension through the node determined by the target test set to obtain the target test set; The generation module 305 is used to take the large model to be tested and the target test set as inputs to the evaluation execution sub-workflow, and generate an evaluation script through the evaluation code generation node; Evaluation module 306 is used to run the evaluation script through the evaluation execution node, evaluate the large model to be tested using the target test set, and obtain evaluation results.

[0098] In another embodiment provided in this disclosure, the acquisition module 302 is further configured to take the description of the target evaluation domain as the input of the test set acquisition node; call the retrieval model to encapsulate the description of the target evaluation domain into a retrieval instruction, and retrieve a large model test set associated with the description of the target evaluation domain from a preset data source; and / or Input a pre-determined large model test set associated with the target evaluation domain into the test set acquisition node.

[0099] In another embodiment provided in this disclosure, the determining module 302 is further configured to: use the large model test set as input to the candidate dimension determining node; extract a preset number of evaluation samples as sample examples from the large model test set through the candidate dimension determining node; call a large language model to identify the test dimensions covered by the large model test set based on the metadata, target evaluation domain, and sample examples of the large model test set, thereby obtaining a set of candidate test dimensions; and call a dimension evaluation model through the test dimension evaluation node to filter the set of candidate test dimensions, thereby obtaining test dimensions that meet preset evaluation quality indicators.

[0100] In another embodiment provided in this disclosure, the determining module 302 is further configured to verify the semantic relevance of each test dimension to the target evaluation domain, remove test dimensions that exceed the coverage of the target evaluation domain; calculate the semantic overlap between each test dimension, and perform aggregation and / or deduplication operations on test dimensions with an overlap higher than a first threshold; evaluate the executable logic of each test dimension, and eliminate test dimensions that cannot obtain execution results; detect the distribution density of each test dimension under each category, and adjust the proportion of test dimensions clustered under the preset category; and remove test dimensions with a number of evaluation samples lower than a second threshold according to the mapping relationship between the test dimensions and the evaluation samples in the large model test set.

[0101] In another embodiment provided in this disclosure, the sampling module 304 is further configured to take the test dimensions that meet the preset evaluation quality indicators and the large model test set as inputs to the sample dimension mapping node; call the large language model through the sample dimension mapping node, and determine the test dimensions corresponding to each evaluation sample according to the semantics of each evaluation sample in the large model test set; wherein, for evaluation samples with multi-dimensional labels, the corresponding test dimensions are limited to a preset number, or the main dimension is determined as the final dimension; perform clustering and / or deduplication operations on the evaluation samples of each test dimension according to the similarity between the evaluation samples corresponding to each test dimension through the dimension sampling node, and obtain the set of evaluation samples corresponding to each test dimension; perform stratified sampling of the evaluation samples in each set of evaluation samples according to the preset quota, and determine the sampled evaluation samples as the target test set.

[0102] In another embodiment provided in this disclosure, the generation module 305 is further configured to call a large language model to generate an evaluation standard for the large model to be tested; and, based on the evaluation standard, generate an evaluation script for calling the large model to be tested and loading a target test set for evaluation.

[0103] In another embodiment provided in this disclosure, the evaluation module 306 is further configured to call a preset isolation environment to execute the evaluation script; based on the model information of the large model to be tested in the evaluation script, call the large model to be tested in the preset isolation environment; in the preset isolation environment, load and execute the evaluation samples in the target test set one by one to obtain the output results corresponding to each evaluation sample; based on the obtained output results and the evaluation criteria in the evaluation script, score the output results of the large model to be tested for each evaluation sample; based on the correspondence between the evaluation samples and the test dimensions, summarize the scores of the large model to be tested on each test dimension to obtain the evaluation result of the large model to be tested.

[0104] In yet another embodiment provided in this disclosure, such as Figure 3 As shown, the large model evaluation system also includes: The first control module 307 is used to determine whether the number of large model test sets acquired by the test set acquisition node is lower than a preset first control value through the test set quality control point; if the number of large model test sets is lower than the preset first control value, it generates a test set expansion instruction to control the test set acquisition node to increase the number of large model test sets acquired. The second control module 308 is used to determine whether the coverage of the test dimension that meets the preset evaluation quality index to the target evaluation domain is lower than the preset second control value through the dimension quality control point; if the coverage of the test dimension is lower than the preset second control value, a test dimension expansion instruction is generated to control the test dimension determination node to call the large language model to generate a new test dimension. The third control module 309 is used to determine whether the number of samples corresponding to each test dimension is lower than the preset third control value through the sample quality control point; if the number of evaluation samples corresponding to any test dimension is lower than the preset third control value, an evaluation sample expansion instruction is generated to control the target test set determination node to call the large language model to generate new evaluation samples.

[0105] In yet another embodiment provided in this disclosure, such as Figure 3 As shown, the large model evaluation system also includes: Compression module 310 is used to perform compression operations on the evaluation samples of the target test set through the sample compression node; perform keyframe selection operations on video samples in the target test set; and / or constrain the resolution and sampling number of image samples in the target test set; and / or perform truncation operations on long text samples in the target test set to retain key semantic paragraphs; and / or perform turn merging and / or turn compression operations on multi-turn dialogue samples in the target test set.

[0106] In yet another embodiment provided in this disclosure, such as Figure 3 As shown, the large model evaluation system also includes: The fourth control module 311 is used to detect the output of the evaluation execution node through the evaluation script quality control point; when erroneous data is detected in the output of the evaluation execution node, an evaluation script error report is generated based on the error data, and the evaluation script error report is output to the evaluation code generation node; The adjustment module 312 is used to adjust the evaluation script according to the evaluation script error report through the evaluation code generation node, and output the adjusted evaluation script to the evaluation execution node; until no error data can be detected in the output of the evaluation execution node, the current evaluation script is fixed.

[0107] This disclosure also provides a computer device including a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of a large model evaluation method as described in any of the above embodiments.

[0108] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments of this disclosure can be implemented in hardware or by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions of the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, mobile hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this disclosure.

[0109] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes in the drawings are not necessarily essential for implementing this disclosure.

[0110] Those skilled in the art will understand that the modules in the system of the embodiments can be distributed in the system of the embodiments as described in the embodiments, or they can be located in one or more systems different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0111] The sequence numbers of the embodiments disclosed above are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0112] Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from its spirit and scope. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A method for evaluating large models, characterized in that, include: Build a large-scale model evaluation workflow; The large model evaluation workflow includes: a test set construction sub-workflow and an evaluation execution sub-workflow; the test set construction sub-workflow includes: a test set acquisition node, a test dimension determination node, and a target test set determination node; the evaluation execution sub-workflow includes: an evaluation code generation node and an evaluation execution node; The test set acquisition node is used to obtain a large model test set corresponding to the target evaluation domain; Based on the large model test set, the test dimensions that meet the preset evaluation quality indicators are determined by the test dimensions. The target test set is obtained by sampling the evaluation samples in the large model test set according to the test dimension, based on the node determined by the target test set; The large model to be tested and the target test set are used as inputs to the evaluation execution sub-workflow, and the evaluation script is generated through the evaluation code generation node. The evaluation script is run through the evaluation execution node, and the target test set is used to evaluate the large model to be tested, so as to obtain the evaluation results.

2. The method as described in claim 1, characterized in that, The step of obtaining a large model test set corresponding to the target evaluation domain through the test set acquisition node includes: The description of the target evaluation domain is used as the input to the test set acquisition node; the retrieval model is invoked to encapsulate the description of the target evaluation domain into a retrieval instruction, and a large model test set associated with the description of the target evaluation domain is retrieved from the preset data source; and / or Input a pre-determined large model test set associated with the target evaluation domain into the test set acquisition node.

3. The method as described in claim 1, characterized in that, The test dimension determination nodes include: candidate dimension determination nodes and test dimension evaluation nodes; The step of determining nodes through the test dimensions, based on the large model test set, involves determining test dimensions that meet preset evaluation quality indicators, including: The large model test set is used as input to determine the nodes for candidate dimensions; Nodes are determined using the candidate dimensions, and a predetermined number of evaluation samples are extracted from the large model test set as sample examples; and The large language model is invoked, and based on the metadata, target evaluation domain, and sample examples of the large model test set, the test dimensions covered by the large model test set are identified to obtain a set of candidate test dimensions. The test dimensions include at least one of the following: dimension name, dimension explanation, typical sample features, exclusion conditions, and suggested sample size quota or priority. The test dimension evaluation node calls the dimension evaluation model to filter the candidate test dimension set and obtain test dimensions that meet the preset evaluation quality indicators.

4. The method as described in claim 3, characterized in that, The step of using the test dimension evaluation node to call the dimension evaluation model to filter the candidate test dimension set and obtain test dimensions that meet the preset evaluation quality indicators includes: Verify the semantic relevance of each test dimension to the target evaluation domain, and remove test dimensions that are outside the coverage of the target evaluation domain; Calculate the semantic overlap between each test dimension, and perform aggregation and / or deduplication operations on test dimensions with an overlap higher than the first threshold; Evaluate the executable logic of each test dimension and eliminate test dimensions that cannot produce execution results; Detect the distribution density of each test dimension under each category, and adjust the proportion of test dimensions clustered under the preset category; Based on the mapping relationship between test dimensions and evaluation samples in the large model test set, test dimensions with fewer than the second threshold are removed.

5. The method as described in claim 1, characterized in that, The target test set determination nodes include: sample dimension mapping nodes and dimension-based sampling nodes; The step of determining nodes through the target test set and sampling evaluation samples in the large model test set according to the test dimension to obtain the target test set includes: The test dimensions and large model test set that meet the preset evaluation quality indicators are used as the inputs to the sample dimension mapping nodes; The large language model is invoked through the sample dimension mapping node. Based on the semantics of each evaluation sample in the large model test set, the test dimension corresponding to each evaluation sample is determined. For evaluation samples with multi-dimensional labels, the corresponding test dimension is limited to a preset number, or the main dimension is determined as the final dimension. By sampling nodes by dimension, clustering and / or deduplication operations are performed on the evaluation samples of each test dimension based on the similarity between the evaluation samples corresponding to each test dimension, resulting in a collection of evaluation samples corresponding to each test dimension. According to the preset quota, the evaluation samples in each evaluation sample set are stratified and sampled, and the sampled evaluation samples are determined as the target test set; the preset quota includes at least one of the following: dimension quota, difficulty quota and source quota.

6. The method as described in claim 1, characterized in that, The step of using the large model to be tested and the target test set as inputs to the evaluation execution sub-workflow, and generating an evaluation script through the evaluation code generation node, includes: Call the large language model to generate evaluation criteria for the large model under test; the evaluation criteria shall include at least one of the following: the scoring criteria for each class of samples, the fault tolerance rules, the output parsing strategy, the aggregation method, and the error handling principles; Based on the evaluation criteria, an evaluation script is generated to call the large model under test and load the target test set for evaluation.

7. The method as described in claim 1, characterized in that, The evaluation script is run through the evaluation execution node, and the target test set is used to evaluate the large model to be tested, obtaining evaluation results, including: The evaluation script is executed in a pre-defined isolated environment. Based on the model information of the large model to be tested in the evaluation script, the large model to be tested is called in a preset isolation environment; In the preset isolation environment, the evaluation samples in the target test set are loaded and executed one by one to obtain the output results corresponding to each evaluation sample; Based on the obtained output results, and using the evaluation criteria in the evaluation script, the output results of the large model under test are scored for each evaluation sample. Based on the correspondence between the evaluation samples and the test dimensions, the scores of the large model under test on each test dimension are summarized to obtain the evaluation results of the large model under test.

8. The method as described in claim 1, characterized in that, The test set construction sub-workflow also includes at least one of the following: test set quality control points, dimension quality control points, and sample quality control points; The method further includes: The test set quality control point determines whether the number of large model test sets acquired by the test set acquisition node is lower than a preset first control value; if the number of large model test sets is lower than the preset first control value, a test set expansion instruction is generated to control the test set acquisition node to increase the number of large model test sets acquired. The dimensional quality control points determine whether the coverage of the test dimension that meets the preset evaluation quality indicators is lower than the preset second control value for the target evaluation domain; if the coverage of the test dimension is lower than the preset second control value, a test dimension expansion instruction is generated to control the test dimension determination node to call the large language model to generate a new test dimension; The sample quality control point determines whether the number of samples corresponding to each test dimension is lower than the preset third control value; if the number of evaluation samples corresponding to any test dimension is lower than the preset third control value, an evaluation sample expansion instruction is generated to control the target test set determination node to call the large language model to generate new evaluation samples.

9. The method as described in claim 1, characterized in that, The evaluation execution sub-workflow also includes: a sample compression node; The method further includes: The evaluation samples of the target test set are compressed using the sample compression node, including: Perform keyframe selection operations on video samples in the target test set; and / or Constraints are imposed on the resolution and sampling number of the image samples in the target test set; and / or Perform truncation on long text samples in the target test set to preserve key semantic paragraphs; and / or Perform turn merging and / or turn compression operations on the multi-turn dialogue samples in the target test set.

10. The method as described in claim 1, characterized in that, The evaluation execution sub-workflow also includes: evaluation script quality control points; The method further includes: The output of the evaluation execution node is detected by the quality control point of the evaluation script; if erroneous data is detected in the output of the evaluation execution node, an evaluation script error report is generated based on the erroneous data, and the evaluation script error report is output to the evaluation code generation node; The evaluation code generation node adjusts the evaluation script based on the evaluation script error report, and outputs the adjusted evaluation script to the evaluation execution node. The current evaluation script will be fixed until no error data is detected in the output of the evaluation execution node.

11. A large model evaluation system, characterized in that, include: Build modules are used to construct large model evaluation workflows; The large model evaluation workflow includes: a test set construction sub-workflow and an evaluation execution sub-workflow; the test set construction sub-workflow includes: a test set acquisition node, a test dimension determination node, and a target test set determination node; the evaluation execution sub-workflow includes: an evaluation code generation node and an evaluation execution node. The acquisition module is used to acquire a large model test set corresponding to the target evaluation domain through the test set acquisition node; The determination module is used to determine the test dimensions that meet the preset evaluation quality indicators based on the large model test set. The sampling module is used to sample the evaluation samples in the large model test set according to the test dimension by determining the nodes through the target test set, so as to obtain the target test set; The generation module is used to take the large model to be tested and the target test set as inputs to the evaluation execution sub-workflow, and generate the evaluation script through the evaluation code generation node; The evaluation module is used to run the evaluation script through the evaluation execution node, evaluate the large model to be tested using the target test set, and obtain the evaluation results.

12. A computer device, characterized in that, The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions that can be executed by the processor. When the computer device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of a large model evaluation method as described in any one of claims 1-10.

Images