Model evaluation method and device, and computer device

Abstract

This application relates to a model evaluation method, apparatus, and computer equipment. The method includes: acquiring a model to be evaluated, a standard analysis model, a dataset of literature to be processed, and a set of tools; inputting the literature topics and the set of tools into the model to be evaluated to determine a first operational step; inputting the first operational step, multiple original documents, and the set of tools into the standard analysis model to determine the first operational result and construct a simulation operation set; based on the simulation operation set, literature topics, the set of tools, multiple original documents, the model to be evaluated, and the standard analysis model, determining multiple operational steps and the operational result corresponding to each operational step, and updating the simulation operation set; inputting the updated simulation operation set and literature topics into the standard analysis model to determine the simulation conclusion; and inputting the simulation conclusion and the literature conclusion into the standard analysis model to perform a quality evaluation of the model to be evaluated. This method can be used to evaluate the performance of experimental design of a model.

Description

Technical Field

[0001] This application relates to the field of model evaluation technology, and in particular to a model evaluation method, apparatus and computer equipment. Background Technology

[0002] In recent years, large language models have seen rapid development in areas such as tool invocation, code generation, and multimodal understanding. In the scientific research field, large language models can be used to design relevant experiments to verify research topics. Therefore, large language models are widely used in scientific research scenarios such as chemical synthesis and biological experiments.

[0003] While existing technologies can design experiments to verify research topics, they often rely heavily on static text benchmarks, resulting in a disconnect from real-world experimental environments. Furthermore, due to variations in training datasets and methods, different large language models exhibit varying performance. Experiments designed for lower-performing large language models are prone to flaws. Therefore, a solution is urgently needed to evaluate the performance of experimental design for these models. Summary of the Invention

[0004] Therefore, it is necessary to provide a model evaluation method, apparatus, and computer equipment that can evaluate the performance of experimental design of models to address the above-mentioned technical problems.

[0005] Firstly, this application provides a model evaluation method. The method includes: acquiring a model to be evaluated, a standard analysis model, a set of literature data to be processed, and a set of tools corresponding to the set of literature data; the set of literature data to be processed includes: a literature topic, an operation procedure, and literature conclusions; inputting the literature topic and the set of tools into the model to be evaluated to determine a first operation step; inputting the first operation step, multiple original documents, and the set of tools into the standard analysis model to determine a first operation result corresponding to the first operation step, and constructing a simulated operation set; the multiple original documents include the original documents corresponding to the set of literature data to be processed; based on the simulated operation set, literature topic, set of tools, multiple original documents, the model to be evaluated, and the standard analysis model, determining multiple operation steps and the operation result corresponding to each operation step, and updating the simulated operation set; inputting the updated simulated operation set and literature topic into the standard analysis model to determine simulation conclusions; and inputting the simulation conclusions and literature conclusions into the standard analysis model to perform a quality evaluation on the model to be evaluated.

[0006] In one embodiment, the method further includes: acquiring a standard analysis model and multiple original documents; inputting the target original document into the standard analysis model to determine the document data set corresponding to the target original document and the document type corresponding to the target original document; inputting the target original document and / or the document data set corresponding to the target original document into the standard analysis model to determine the tool corresponding to the target original document and the input and output parameters of the tool; constructing a document data set based on the document data sets corresponding to all the original documents; and constructing multiple tool sets based on the document types, tools, and input and output parameters corresponding to all the original documents.

[0007] In one embodiment, constructing multiple tool sets based on the corresponding document types, tools, and input / output parameters of all the original documents includes: merging the tools and corresponding input / output parameters of the original documents of biological type to determine a biological tool set; merging the tools and corresponding input / output parameters of the original documents of physical type to determine a physical tool set; and merging the tools and corresponding input / output parameters of the original documents of chemical type to determine a chemical tool set.

[0008] In one embodiment, the step of determining multiple operation steps and corresponding operation results based on the simulation operation set, literature topics, toolset, multiple original documents, model to be evaluated, and standard analysis model, and updating the simulation operation set includes: inputting the literature topics, toolset, and updated simulation operation set into the model to be evaluated to determine a second operation step; inputting the updated simulation operation set, the second operation step, multiple original documents, and toolset into the standard analysis model to determine a second operation result corresponding to the second operation step, and updating the simulation operation set; cyclically determining operation steps and operation results until the preset conditions are met to obtain the final updated simulation operation set.

[0009] In one embodiment, inputting the simulation conclusions and literature conclusions into a standard analysis model to perform a quality assessment of the model to be evaluated includes: inputting the simulation conclusions and literature conclusions into a standard analysis model to determine the quality assessment results of the literature data set to be processed on the model to be evaluated; obtaining the quality assessment results of all literature data sets to be processed on the model to be evaluated; and determining the quality parameters of the model to be evaluated based on the quality assessment results of all literature data sets to be processed on the model to be evaluated.

[0010] In one embodiment, determining the quality parameters of the model to be evaluated based on the quality assessment results of all the literature data sets to be processed includes:

[0011] Based on the first quality assessment results of all the literature data sets to be processed for the model to be evaluated, the first number of matching literature data sets to be processed is determined; the first quality assessment result is obtained by performing a quality assessment once for each of the literature data sets to be processed; the total number of literature data sets to be processed is counted; based on the first number and the total number, the first accuracy parameter of the model to be evaluated is determined.

[0012] In one embodiment, determining the quality parameters of the model to be evaluated based on the quality assessment results of all the literature data sets to be processed includes: determining a second number of matching literature data sets based on a second quality assessment result of all the literature data sets to be processed; the second quality assessment result is obtained by performing multiple quality assessments for each of the literature data sets to be processed; counting the total number of literature data sets to be processed; and determining a second accuracy parameter of the model to be evaluated based on the second number and the total number.

[0013] In one embodiment, determining the quality parameters of the model to be evaluated based on the quality assessment results of all unprocessed literature data sets includes: obtaining a simulation operation set corresponding to the target unprocessed literature data set; the second quality assessment result of the target unprocessed literature data set is a match; determining a third number of operation steps based on the simulation operation set; obtaining a fourth number of operation steps in the operation process of the target unprocessed literature data set; determining the initial efficiency parameters of the target unprocessed literature data set for the model to be evaluated based on the third and fourth numbers; and determining the target efficiency parameters of the model to be evaluated based on the initial efficiency parameters of all unprocessed literature data sets for the model to be evaluated.

[0014] Secondly, this application also provides a model evaluation apparatus. The apparatus includes:

[0015] The acquisition module is used to acquire the model to be evaluated, the standard analysis model, the literature data set to be processed, and the tool set corresponding to the literature data set to be processed; the literature data set to be processed includes: literature topic, operation process, and literature conclusion.

[0016] The step determination module is used to input the literature topic and the toolset into the model to be evaluated and determine the first operation step.

[0017] The construction module is used to input the first operation step, multiple original documents, and toolset into a standard analysis model, determine the first operation result corresponding to the first operation step, and construct a simulation operation set; the multiple original documents include the original documents corresponding to the data set of documents to be processed.

[0018] The update module is used to determine multiple operation steps and the operation results corresponding to each operation step based on the simulation operation set, literature topics, tool set, multiple original documents, model to be evaluated, and standard analysis model, and update the simulation operation set.

[0019] The conclusion determination module is used to input the updated simulation operation set and literature topics into the standard analysis model to determine the simulation conclusions.

[0020] The evaluation module is used to input the simulation results and literature results into the standard analysis model to evaluate the quality of the model to be evaluated.

[0021] Thirdly, this application also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement any of the methods in the first aspect above.

[0022] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the methods in the first aspect described above.

[0023] The aforementioned model evaluation method, apparatus, and computer equipment acquire the model to be evaluated, a standard analysis model, a set of literature data to be processed, and a set of tools corresponding to the set of literature data to be processed. The set of literature data to be processed includes: literature topics, operation procedures, and literature conclusions. Then, the literature topics and the set of tools are input into the model to be evaluated to determine the first operation step. The first operation step, multiple original documents, and the set of tools are then input into the standard analysis model to determine the first operation result corresponding to the first operation step, and a simulation operation set is constructed. The multiple original documents include the original documents corresponding to the set of literature data to be processed. Then, based on the simulation operation set, literature topics, set of tools, multiple original documents, the model to be evaluated, and the standard analysis model, multiple operation steps and the operation result corresponding to each operation step are determined. The simulation operation set is updated, and the updated simulation operation set and literature topics are input into the standard analysis model to determine the simulation conclusion. Finally, the simulation conclusion and the literature conclusion are input into the standard analysis model to perform a quality evaluation on the model to be evaluated, thereby achieving the performance evaluation of experimental design of the model and solving the problem that unreasonable aspects are prone to occur in the experimental process designed by large language models with poor performance. Attached Figure Description

[0024] Figure 1 This is a diagram illustrating the application environment of the model evaluation method in one embodiment;

[0025] Figure 2This is a flowchart illustrating a model evaluation method in one embodiment;

[0026] Figure 3 This is a flowchart illustrating the process of building multiple tool sets in one embodiment;

[0027] Figure 4 This is a schematic diagram of the process for quality assessment of the model to be evaluated in one embodiment;

[0028] Figure 5 A block diagram of a model evaluation method in another embodiment;

[0029] Figure 6 This is a schematic diagram of the evaluation results in one embodiment;

[0030] Figure 7 This is a structural block diagram of the model evaluation device in one embodiment;

[0031] Figure 8 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0032] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0033] In recent years, large language models have been widely used in tool invocation, code generation, and multimodal understanding. They have evolved from playing a role in assisting in understanding literature to being able to autonomously design and execute experiments, and have been widely applied in scientific research scenarios such as chemical synthesis and biological experiments.

[0034] However, in related technologies, model evaluation heavily relies on static text benchmarks, which are disconnected from real-world wet experimental environments. Mainstream static text benchmarks, such as SciKnowEval, LLM-SRBench, and ResearchBench, primarily focus on single-question-answering and fixed-path inference. Domain-specific static text benchmarks, such as BioPlanner and BioProBench, directly generate experimental plans, but do not support step-by-step verification and dynamic adjustments. These methods cannot capture the iterative decisions, resource constraints, dynamic error correction, and failure recovery in real-world wet experiments, nor can they measure the model's performance reliability under real physical constraints. Therefore, related technologies lack a systematic evaluation tool covering the entire lifecycle of wet experiments, failing to objectively reflect the model's capability boundaries in real-world research scenarios.

[0035] Current related technologies can be broadly categorized into two types: The first type is static knowledge-driven benchmarks, including SciKnowEval, LLM-SRBench, and ResearchBench. These benchmarks primarily focus on single-round question answering, fixed-path reasoning, and offline knowledge assessment, without involving any experimental execution procedures or dynamic interactive capabilities. The second type is protocol-driven benchmarks, including BioPlanner, BioProBench, and SGI-Bench. These benchmarks can generate experimental steps based on the task, but they output the complete experimental process directly at once, without supporting step-by-step execution, intermediate result feedback, exception handling, equipment constraints, or reagent limitations.

[0036] In summary, the relevant technologies suffer from static evaluation paradigms that deviate from the logic of real experiments. All static text benchmarks employ offline, single-round, and answer-driven models, failing to simulate iterative reasoning, dynamic observation, anomaly handling, and closed-loop decision-making in real wet labs. This leads to significant discrepancies between evaluation results and the model's real-world implementation capabilities. Furthermore, the lack of physical resource constraints easily generates illusionary solutions: these technologies do not establish real physical constraints such as hardware equipment, analysis software, experimental reagents, and consumable inventory. Models often generate illusionary solutions where equipment is nonexistent, reagents are incompatible, or procedures are infeasible. Additionally, there is the issue of one-time protocol generation, unable to perform step-by-step verification: these technologies only output the complete experimental process, without supporting step-by-step execution and verification. They cannot detect logical breaks, skipped steps, incorrect operation sequences, or omissions of critical steps in long-running experiments. Moreover, they cannot assess dynamic reprogramming and error recovery capabilities: real experiments frequently encounter equipment failures, reagent shortages, and abnormal phenomena. Existing benchmarks are completely unable to assess the model's adjustment, reprogramming, and fault recovery capabilities in such scenarios. Furthermore, the evaluation metrics are too simplistic and cannot comprehensively measure the model's performance: existing metrics only focus on whether the conclusions are correct, without measuring the rationality of the steps, execution efficiency, operational standardization, scientific rigor, or resource utilization, and thus cannot distinguish the differences in the model's true capabilities.

[0037] Neither of the two existing technologies mentioned above possesses interactive closed-loop simulation, physical resource constraints, multi-round step-by-step execution, dynamic replanning, or experimental fault recovery capabilities. They cannot realistically simulate the entire wet experiment process, nor can they accurately evaluate the actual performance of the model in a real laboratory.

[0038] The model evaluation method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Server 104 is used to execute the model evaluation method. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0039] To address the aforementioned problems, in one embodiment of this application, such as Figure 2 As shown, a model evaluation method is provided, including the following steps:

[0040] Step 201: Obtain the model to be evaluated, the standard analysis model, the set of literature data to be processed, and the set of tools corresponding to the set of literature data to be processed.

[0041] The model to be evaluated is the model that needs quality assessment to verify its performance in experimental design; in this embodiment, it is a large language model. The standard analysis model is a pre-set model that has been pre-evaluated and deemed to have excellent performance and reliability; in this embodiment, it is a large language model. The literature data set to be processed is the literature data set obtained after extracting information from the original literature using the standard analysis model. The literature data set to be processed includes: the literature topic, the operation process, and the literature conclusion. The original literature is the pre-set literature material, presented in the form of a paper. The literature topic is the scientific problem or technical task to be studied or demonstrated in the original literature. The operation process is the overall process of studying or demonstrating the literature topic in the original literature, including multiple steps and the results corresponding to each step. The literature conclusion is the final result obtained after executing the operation process for the literature topic in the original literature.

[0042] It should be noted that in this embodiment, multiple original documents are prepared in advance, and information is extracted from each original document to obtain multiple sets of document data to be processed. In this embodiment, the set of document data to be processed is any one of all the sets of document data to be processed. In this embodiment, the relevant steps are performed on the set of document data to be processed as an example. In actual practice, the relevant steps in this embodiment need to be performed on each set of document data to be processed.

[0043] For example, the literature data set to be processed is: {"problem": "Does this solution contain ammonium ions (NH4+)?"}+ )? ", "answer": "The solution contains ammonium ions (NH4+)?" + The reaction produces ammonia (NH3), which turns moist red litmus paper blue. The steps are: ["Add sodium hydroxide (NaOH) solution to the solution to be tested", "Heat the test tube", "Place moist red litmus paper at the mouth of the test tube and observe the color change"]. The literature question is whether the solution contains ammonium ions (NH4+). + The procedure is as follows: ["Add sodium hydroxide (NaOH) solution to the solution to be tested", "Heat the test tube", "Place moistened red litmus paper at the mouth of the test tube and observe the color change of the paper"]; the literature concludes that the solution contains ammonium ions (NH4+). + ).

[0044] The toolset is a collection of tools pre-extracted from multiple original documents using a standard analysis model and then merged. It includes multiple tools and their corresponding input / output parameters. Tools are those used in the operational process of the original documents, and their forms include hardware and software. Input / output parameters are the inputs and outputs of the tools in the operational process. For example, if a tool is an algorithm, its corresponding input / output parameters are the data input to the algorithm and the result output by the algorithm. In this embodiment, the toolset is a single set of tools corresponding to the group of document data to be processed, including tools from multiple original documents of the same document type and their corresponding input / output parameters. The document type refers to the field to which the original documents belong; for example, document types include biological, chemical, and physical types.

[0045] In this embodiment, after obtaining the model to be evaluated, a standard analysis model, a set of literature data to be processed, and a set of tools corresponding to the set of literature data to be processed are obtained from a preset database.

[0046] Step 202: Input the literature topics and toolset into the model to be evaluated to determine the first operation step.

[0047] In this embodiment, the literature topics and toolset are input into the model to be evaluated, and the model is designed by using preset prompts, and the first operation step is output.

[0048] The preset prompts are pre-defined prompt texts. These prompts are natural language instructions, questions, or context input to the model being evaluated, guiding it to generate specific outputs. For example, in this embodiment, the prompts guide the model to generate a first operation step based on the literature review topic and toolset.

[0049] The first operational step is a practical step designed to validate the literature topic for the model to be evaluated.

[0050] Step 203: Input the first operation step, multiple original documents and toolset into the standard analysis model, determine the first operation result corresponding to the first operation step, and construct the simulation operation set.

[0051] In this embodiment, the first operation step, multiple original documents, and toolset are input into the standard analysis model, and the standard analysis model is deduced by preset prompt words to obtain the first operation result. The first operation result and the first operation step are used as a simulation operation set.

[0052] The multiple original documents include the original documents corresponding to the document data group to be processed. In this embodiment, the multiple original documents include all the original documents prepared in advance.

[0053] The first operation result is the result of executing the first operation step, derived from the standard analysis model, and corresponds to the first operation step.

[0054] Step 204: Based on the simulation operation set, literature topics, tool set, multiple original documents, model to be evaluated, and standard analysis model, determine multiple operation steps and the operation results corresponding to each operation step, and update the simulation operation set.

[0055] In this embodiment, the literature topics, toolset, and simulation operation set are input into the model to be evaluated. Preset prompts guide the model to design and output a second operation step. Then, the simulation operation set, the second operation step, multiple original documents, and the toolset are input into a standard analysis model. Preset prompts guide the standard analysis model to derive a second operation result. The second operation step and the second operation result are added to the simulation operation set. The operation steps and corresponding operation results are then iteratively determined until preset conditions are met, resulting in an updated simulation operation set.

[0056] The second operation step is the next operation step after the latest operation step in the simulation operation set. The second operation result is the result of executing the second operation step, derived from the standard analysis model, corresponding to the second operation step. The preset condition is a pre-set condition for stopping the determination of the operation step and the corresponding operation result. In this embodiment, it is that the number of times the operation step and the corresponding operation result are determined is greater than a preset number of loops, such as 15 times. The updated simulation operation set is the simulation operation set after the loop stops.

[0057] Step 205: Input the updated simulation operation set and literature topics into the standard analysis model to determine the simulation conclusions.

[0058] In this embodiment, the updated simulation operation set and literature topics are input into the standard analysis model, and the standard analysis model summarizes and obtains simulation conclusions through preset prompts.

[0059] The simulation conclusions are derived from the standard analytical model based on the literature research topic, the operational steps and results in the updated simulation operation set. The simulation conclusions correspond to the literature research topic.

[0060] Step 206: Input the simulation results and literature results into the standard analysis model to evaluate the quality of the model to be evaluated.

[0061] In this embodiment, the simulation results and literature results are input into the standard analysis model. The standard analysis model determines whether the simulation results and literature results match through preset prompts. If they match, the evaluation model is deemed to have passed the quality assessment. If they do not match, the evaluation model is deemed to have failed the quality assessment.

[0062] It should be noted that in this embodiment, by determining multiple operation steps and the corresponding operation results for each operation step based on the simulation operation set, literature topics, tool sets, multiple original documents, the model to be evaluated, and the standard analysis model, and updating the simulation operation set, the problem of static evaluation paradigms in related technologies, which deviate from the logic of real experiments, is solved. Furthermore, by setting a tool set corresponding to the literature data set to be processed as a constraint on physical resources, the problem of lacking physical resource constraints and easily generating illusionary solutions in related technologies is addressed. Moreover, in this embodiment, multiple operation steps and the corresponding operation results for each operation step are obtained iteratively, solving the problem of one-time protocol generation and inability to perform step-by-step verification in related technologies.

[0063] In the above model evaluation method, the following steps are taken: The model to be evaluated, a standard analysis model, a set of literature data to be processed, and a set of tools corresponding to the set of literature data to be processed are obtained. The set of literature data to be processed includes: literature topic, operation procedure, and literature conclusion. The literature topic and the set of tools are then input into the model to be evaluated to determine the first operation step. The first operation step, multiple original documents, and the set of tools are then input into the standard analysis model to determine the first operation result corresponding to the first operation step, and a simulation operation set is constructed. The multiple original documents include those corresponding to the set of literature data to be processed. Based on the simulation operation set, literature topic, set of tools, multiple original documents, the model to be evaluated, and the standard analysis model, multiple operation steps and the operation result corresponding to each operation step are determined. The simulation operation set is then updated, and the updated simulation operation set and literature topic are input into the standard analysis model to determine the simulation conclusion. Finally, the simulation conclusion and the literature conclusion are input into the standard analysis model to perform a quality evaluation of the model to be evaluated, thereby achieving the performance evaluation of the model for experimental design.

[0064] In other embodiments of this application, such as Figure 3 As shown, the method also includes:

[0065] Step 301: Obtain the standard analysis model and multiple original documents.

[0066] The standard analysis model is a pre-set model that has been evaluated beforehand and deemed to be of excellent performance and reliable. In this embodiment, it is a large language model. The original documents are pre-set literature materials, which are presented in the form of academic papers.

[0067] In this embodiment, when it is necessary to obtain a standard analysis model and multiple original documents, the standard analysis model and multiple original documents will be directly obtained from a preset database.

[0068] Step 302: Input the target original document into the standard analysis model to determine the document data group and document type corresponding to the target original document.

[0069] The target original document can be any one of multiple original documents. In this embodiment, the steps are described using the target original document as an example. In practice, the steps in this embodiment need to be performed on each original document.

[0070] Document type refers to the type of field to which the original document belongs. For example, document types include biological, chemical, and physical types.

[0071] A literature data set is a set of literature data obtained after extracting information from original documents using a standard analysis model. The literature data set includes: the literature topic, the operational process, and the literature conclusion. The original documents are pre-defined literature materials, presented in the form of papers. The literature topic is the scientific problem or technical task to be researched or demonstrated in the original documents. The operational process is the overall process of researching or demonstrating the literature topic in the original documents, including multiple steps and the corresponding results for each step. The literature conclusion is the final result obtained after executing the operational process for the literature topic in the original documents.

[0072] In this embodiment, the target original document is input into the standard analysis model, and the standard analysis model extracts information through preset prompts, thereby obtaining the document data group corresponding to the target original document and the document type corresponding to the target original document.

[0073] Step 303: Input the target original document and / or the document data set corresponding to the target original document into the standard analysis model to determine the tool corresponding to the target original document and the input and output parameters of the tool.

[0074] In this embodiment, the target original document and / or the corresponding document data set will be input into the standard analysis model, and the standard analysis model will extract information through preset prompts to obtain the tools used in the target original document and the input and output parameters of the tools.

[0075] Preferably, the target original document is input into the standard analysis model, and the standard analysis model extracts information through preset prompts to obtain the tools used in the target original document and the input and output parameters of the tools.

[0076] Tools are the tools used in the operational process of the original literature, and their forms include hardware and software. Input and output parameters are the inputs and outputs of the tools in the operational process. For example, if the tool is an algorithm, its corresponding input and output parameters are the data input to the algorithm and the results output by the algorithm.

[0077] Step 304: Construct a literature data set based on the literature data groups corresponding to all original documents.

[0078] In this embodiment, all the original documents' corresponding document data groups will be used as the document data set.

[0079] The literature dataset includes all the literature datasets corresponding to the original documents.

[0080] Step 305: Based on the corresponding document types, tools, and input / output parameters of all original documents, construct multiple tool sets.

[0081] In this embodiment, based on the corresponding document types, tools, and input / output parameters of all original documents, multiple tool sets are constructed, including:

[0082] Step 1: Merge the tools and their corresponding input / output parameters from the original documents of the biological type to determine the biological tool set.

[0083] In this embodiment, for all original documents of the biological type, the corresponding tools and their corresponding input and output parameters will be merged to obtain a biological tool set.

[0084] The biological toolset includes tools for all documents of biological type, the original documents, and the corresponding input and output parameters for each tool.

[0085] Step 2: Merge the tools and their corresponding input / output parameters for the original documents of the physical type to determine the physical tool set.

[0086] In this embodiment, for all original documents of the physical type, the corresponding tools and their corresponding input and output parameters will be merged to obtain a set of physical tools.

[0087] The physical toolset includes tools for all documents of the physical type, the original documents, and the corresponding input and output parameters for each tool.

[0088] Step 3: Merge the tools and their corresponding input / output parameters from the original documents of the chemistry type to determine the chemistry tool set.

[0089] In this embodiment, for all original documents of the chemical type, the corresponding tools and their corresponding input and output parameters will be merged to obtain a chemical tool set.

[0090] The chemical toolset includes tools for all documents of chemical type, the original documents, and the corresponding input and output parameters for each tool.

[0091] It should be noted that in this embodiment, the deduplication of each tool set is also performed. For a certain tool set, if the input and output parameters of a certain tool in the tool set are repeated, only one repeated input and output parameter of that tool is retained.

[0092] For example, in this embodiment, a certain biological tool cell culture chamber and its input / output parameters are as follows: {"item":"cell culture chamber", "examples": [{"input": "culturing shRNA-transduced MDDCs at 37°C and 5% CO2", "output": "stable shRNA-expressing MDDCs cells, maintaining cell viability and normal physiological state"}, {"input": "culturing Jurkat E6.1 cells and HEK293T cells to the required number", "output": "the number of Jurkat E6.1 cells reaches 5×10^6, and the number of HEK293T cells reaches 1×10^6"}, {"input": "culturing fibroblasts at 37°C, 5% CO2, and 90% humidity", "output": "fibroblasts cultured to 3...{"input": "HeLa, DT40, RPE1 and other cell lines were cultured at 37°C and 5% CO2 for Blastidin selection and single-cell clone expansion", "output": "Cells grew normally at a constant temperature of 37°C, humidity of 95%, and CO2 concentration of 5%, with a passage cycle of 2-3 days"}, {"input": "H1299-dGFP cells were cultured and incubated under conditions maintained after DD-siRNA conjugate transduction", "output": "Stable-growing H1299-dGFP cells were cultured at 37°C and 5% CO2"}, {"input": "HEK293 cells were cultured in a 37°C, 5% CO2 incubator, and cells were collected 48 hours after transfection", "output": "Cells showed good adherence and growth, with a density of 70-80% confluence and cell viability >95%"}, {"input": "96-well plate single-cell clones were cultured undisturbed for 5 days in a 37°C, 5% CO2 incubator", "output": "Single-cell clones began to proliferate, and cell colonies were observed after 5 days.", {"input": "Embryosomes and brain organoids were cultured in a cell culture incubator at 37°C and 5% CO2", "output": "Stable temperature (37°C) and CO2 concentration (5%) were maintained to support normal cell growth and differentiation.", {"input": "HEK293T cells were cultured at 37°C and 5% CO2 and incubated for 20 hours after transfection", "output": "Cells adhered well and grew, reaching a density of 650,000 / well, with a cell viability >95%.", {"input": "Transfected HEK293T cells were cultured at 37°C and 5% CO2 for 20 hours", "output": "Cells adhered well and grew, with the culture temperature maintained at 37±0.5°C and the CO2 concentration stabilized at 5±0.2%.", {"input": "tRNA-modified mutant cells were cultured at 37°C and 5% CO2", "output": "The cell density reached 5×10⁻⁶". 5Cells / mL, cell viability >95%"}, {"input": "Culturing smNPCs and organoids at 37°C and 5% CO2", "output": "Maintaining a constant culture temperature of 37±0.5°C, CO2 concentration of 5±0.2%, and relative humidity >95% to support normal cell growth and metabolism"}, {"input": "Culturing BM-EPCs at 37°C and 5% CO2 and conducting cell adhesion co-culture experiments", "output": "After 7 days of culture, BM-EPCs formed a confluent monolayer. After 20 minutes of cell adhesion co-culture, calcein-AM positive cells adhered to the recipient cell layer"}, {"input": "Culturing virus production cells (NIH 3T3 or 293T cells) in a cell culture incubator", "output": "Cells grew to 80-90% confluence, producing a virus-containing supernatant"}, {"input": "Culturing ES-2 and OVCAR3 cells in 96-well ULA plates for 24 hours and 72 hours in a cell culture incubator", "output": "Maintaining culture conditions of 37°C and 5% CO2, cells formed a complete 3D spherical structure.", {"input": "After seeding cells at a density of 4×10^5, they were cultured at 37°C and 5% CO2 for 24 hours, followed by 48 hours of serum-free culture induction.", "output": "Cells successfully adhered, and approximately 150 detectable cilia were formed after cilia induction.", {"input": "Experimental cells were cultured in a cell culture incubator for siRNA transfection and TMG treatment.", "output": "Cells grew normally at 37°C and 5% CO2, and the transfection efficiency and drug treatment effect met the experimental requirements.", {"input": "Transfected HEK293T cells expressed fluorescently labeled proteins overnight.", "output": "Cells adhered well, and fluorescent protein expression was sufficient. Temperature: 37°C, CO2 concentration: 5%.", {"input": "HeLa cells were cultured to 70-80% confluence.", "output": "HeLa cells were cultured at 6..." The cells reached 70-80% confluence in the culture dish. One hour before injection, the cells with fresh culture medium were returned to the incubator. The cells were allowed to recover at 37°C and 5% CO2 for one hour. DRG neurons were cultured in a cell culture incubator at 37°C and 5% CO2 for 24 hours until day 3. The DRG neurons showed good adherence and growth, normal cell morphology, and a survival rate >80%. Cells were cultured for several generations in a medium containing 10% defatted serum, and proliferation was observed.{"output": "Cells can maintain proliferation for several generations, but the proliferation rate of some cell lines slows down, and the doubling time is prolonged by about 30-50%"}, {"input": "HeLa cells were cultured in a cell culture incubator to 80% confluence", "output": "HeLa cells have grown to 80% confluence, the cell condition is good, and subsequent labeling experiments can be performed"}]},

[0093] For example, in this embodiment, a benchtop microcentrifuge and its input / output parameters are as follows: {"item": "benchtop microcentrifuge", "examples": [{"input": "instantaneous centrifugation of plasmid solution and transfection reagent to avoid liquid adhesion to the tube wall", "output": "solution concentrated at the bottom of the tube, no adhesion to the tube wall"}, {"input": "sampling collection of samples using a microcentrifuge", "output": "liquid collected at the bottom of the tube after centrifugation at 12000-16000×g for easy sampling"}]}.

[0094] For example, in this embodiment, a physical tool, a light intensity measuring instrument, and its input and output parameters are: {"item":"light intensity measuring instrument", "examples": [{"input": "measures the light intensity of 460 nm blue light, 660 nm red light, and green light","output": "accurate light intensity values ​​(µmol m) of each wavelength light source".} -2 sec -1 ), ensure the experimental light intensity is controllable and repeatable"},{"input": "Measure the light intensity of the blue light pulse, ensuring it is 120 µmol m -2 sec -1 ", "output": "Light intensity value 120 µmol m -2 sec -1 To ensure that the intensity of each pulse is consistent.

[0095] It should be noted that in this embodiment, a standard analysis model and multiple original documents are obtained. Then, the target original document is input into the standard analysis model to determine the document data set and document type corresponding to the target original document. Next, the target original document and / or the document data set corresponding to the target original document are input into the standard analysis model to determine the tool and its input / output parameters. Based on the document data sets corresponding to all the original documents, a document data set is constructed. Finally, based on the document types, tools, and input / output parameters corresponding to all the original documents, multiple tool sets are constructed, laying the foundation for subsequent quality assessment of the model to be evaluated based on the tool sets.

[0096] In other embodiments of this application, based on a set of simulated operations, literature topics, a set of tools, multiple original documents, a model to be evaluated, and a standard analysis model, multiple operation steps and the corresponding operation results for each operation step are determined, and the set of simulated operations is updated as follows:

[0097] Step 1: Input the literature review topics, toolset, and updated simulation operation set into the model to be evaluated, and determine the second operation step.

[0098] In this embodiment, the literature topics, toolset, and simulation operation set are input into the model to be evaluated, and the model to be evaluated is designed by using preset prompts, and the second operation step is output.

[0099] The second operation step is the next operation step after the latest operation step in the simulation operation set.

[0100] Step 2: Input the updated simulation operation set, the second operation step, multiple original documents, and the toolset into the standard analysis model, determine the second operation result corresponding to the second operation step, and update the simulation operation set.

[0101] In this embodiment, the simulation operation set, the second operation step, multiple original documents, and the toolset are input into the standard analysis model. The standard analysis model is then deduced using preset prompts to obtain the second operation result. The second operation step and the second operation result are then added to the simulation operation set to update the simulation operation set.

[0102] The second operation result is the result of executing the second operation step, derived from the standard analysis model, and corresponds to the second operation step.

[0103] Step 3: Repeatedly determine the operation steps and the operation results until the preset conditions are met, and finally obtain the updated simulation operation set.

[0104] In this embodiment, the steps of determining the second operation step and the second operation result in this embodiment will be repeated to cyclically determine the operation steps and the corresponding operation results until the preset conditions are met, and the final updated simulation operation set is obtained.

[0105] The final updated set of simulation operations is the set of simulation operations after the loop stops.

[0106] In this embodiment, the preset condition is that the model to be evaluated believes that the current set of simulated operations has solved the problem in the literature, or that the number of times the operation steps and corresponding operation results are determined is greater than the preset number of loops.

[0107] For example, the preset number of loops can be 15 times. That is, when the number of times the operation steps and corresponding operation results are determined is greater than 15, the loop will be exited and the simulated operation set after the loop stops will be used as the final updated simulated operation set.

[0108] It should be noted that the model under evaluation believes that the current set of simulation operations has solved the literature's questions through preset prompts.

[0109] It should be noted that, in this embodiment, by inputting the literature topic, toolset, and updated simulation operation set into the model to be evaluated, the second operation step is determined. Then, the updated simulation operation set, the second operation step, multiple original documents, and toolset are input into the standard analysis model to determine the second operation result corresponding to the second operation step, and the simulation operation set is updated. Finally, the operation steps and operation results are determined iteratively until the preset conditions are met, and the final updated simulation operation set is obtained. This lays the foundation for subsequent determination of simulation conclusions based on the updated simulation operation set and literature topic input into the standard analysis model.

[0110] In other embodiments of this application, such as Figure 4 As shown, the simulation results and literature results are input into the standard analysis model, and the quality assessment of the model to be evaluated includes:

[0111] Step 401: Input the simulation results and literature results into the standard analysis model to determine the quality assessment results of the literature data set to be processed for the model to be evaluated.

[0112] In this embodiment, the simulation conclusions and literature conclusions are input into the standard analysis model. The standard analysis model determines whether the simulation conclusions and literature conclusions match through preset prompts, and the final judgment result is used as the quality assessment result of the literature data group to be processed for the model to be evaluated.

[0113] The quality assessment results characterize the results of judging whether the simulation conclusions and the literature conclusions match for the literature data set to be processed, including: the simulation conclusions and the literature conclusions match, or the simulation conclusions and the literature conclusions do not match.

[0114] It should be noted that the document data group to be processed can be any one of all document data groups to be processed. In this embodiment, the steps are described using a document data group to be processed as an example. In practice, the steps in this embodiment need to be performed on every document data group to be processed.

[0115] Step 402: Obtain the quality assessment results of the model to be evaluated from all the literature data sets to be processed.

[0116] In this embodiment, for each literature data set, the quality assessment result of the model to be evaluated for that literature data set will be determined, and then the quality assessment results of the model to be evaluated for all literature data sets will be obtained.

[0117] Step 403: Based on the quality assessment results of all the literature data sets to be processed, determine the quality parameters of the model to be evaluated.

[0118] Quality parameters are parameters used to characterize the performance of the model being evaluated in experimental design.

[0119] In this embodiment, based on the quality assessment results of all the literature data sets to be processed, the quality parameters of the model to be evaluated are determined as follows:

[0120] Step 1: Based on the first quality assessment results of all unprocessed literature data groups for the model to be evaluated, determine the first number of unprocessed literature data groups whose quality assessment results match.

[0121] The first quality assessment result was obtained by performing a quality assessment once for each group of literature data to be processed.

[0122] It should be noted that the process of performing a quality assessment is the same as the process described in the above embodiments of this application, which involves determining the simulation conclusions of the literature data set to be processed and inputting the simulation conclusions and literature conclusions into the standard analysis model to determine the quality assessment result of the literature data set to be processed on the model to be evaluated. This will not be elaborated further here. In this embodiment, the first quality assessment result is the quality assessment result of the literature data set to be processed on the model to be evaluated.

[0123] In this embodiment, based on the quality assessment results of all the literature data sets to be processed for the model to be evaluated, the number of literature data sets whose quality assessment results match is counted and used as the first number.

[0124] The first quantity refers to the number of matching groups of literature data to be processed based on the first quality assessment result.

[0125] Step 2: Count the total number of literature data sets to be processed.

[0126] In this embodiment, the number of all pending document data sets will be counted and used as the total number of pending document data sets.

[0127] The total number of document data sets to be processed is the same as the total number of all document data sets to be processed.

[0128] Step 3: Based on the first quantity and the total quantity, determine the first accuracy parameter of the model to be evaluated.

[0129] The first accuracy parameter of the model to be evaluated is used to characterize the performance of the model in experimental design when performing a quality assessment.

[0130] In this embodiment, the quotient of the first quantity divided by the total quantity is calculated, and the quotient is used as the first accuracy parameter of the model to be evaluated.

[0131] It should be noted that in this embodiment, by determining the first quality assessment result of the model to be evaluated based on all the literature data sets to be processed, the first number of matching literature data sets to be processed is determined, and the total number of literature data sets to be processed is counted. Finally, based on the first number and the total number, the first accuracy parameter of the model to be evaluated is determined, thereby realizing the evaluation of the performance of the experimental design of the model.

[0132] In other embodiments of this application, the quality parameters of the model to be evaluated are determined based on the quality evaluation results of all the literature data sets to be processed, including:

[0133] Step 1: Based on the second quality assessment results of all unprocessed literature data groups for the model to be evaluated, determine the second number of unprocessed literature data groups whose quality assessment results match.

[0134] The second quality assessment result was obtained by performing multiple quality assessments for each group of literature data to be processed.

[0135] It should be noted that performing multiple quality assessments specifically involves: performing a quality assessment once for a specific set of literature data to be processed, determining the quality assessment result for the model to be evaluated from this set of literature data. If the quality assessment result for this set of literature data is a mismatch, the quality assessment is repeated once, until the number of times the quality assessment is repeated equals the preset number of iterations, or until the quality assessment result for this set of literature data matches the model to be evaluated. If the loop ends and the quality assessment result for this set of literature data matches the model to be evaluated, then the match is taken as the second quality assessment result for the model to be evaluated. If the loop ends and the quality assessment result for this set of literature data matches the model to be evaluated, then the mismatch is taken as the second quality assessment result for the model to be evaluated.

[0136] The preset number of cycles is the number of times a quality assessment is performed, which is a preset value. For example, it can be 5 times.

[0137] In this embodiment, the number of unprocessed literature data groups that match the second quality assessment result will be counted, and this number will be used as the second quantity.

[0138] Step 2: Count the total number of literature data sets to be processed.

[0139] In this embodiment, the number of all pending document data sets will be counted and used as the total number of pending document data sets.

[0140] The total number of document data sets to be processed is the same as the total number of all document data sets to be processed.

[0141] Step 3: Based on the second quantity and the total quantity, determine the second accuracy parameter of the model to be evaluated.

[0142] The second accuracy parameter of the model to be evaluated is used to characterize the performance of the model in experimental design when performing multiple quality assessments.

[0143] In this embodiment, the quotient of the second quantity divided by the total quantity is calculated, and the quotient is used as the second accuracy parameter of the model to be evaluated.

[0144] It should be noted that in this embodiment, the second quality assessment result of the model to be evaluated is determined based on the second quality assessment result of all the literature data sets to be processed, the second number of matching literature data sets to be processed is determined, and the total number of literature data sets to be processed is counted. Finally, based on the second number and the total number, the second accuracy parameter of the model to be evaluated is determined, thereby realizing the evaluation of the performance of the model in experimental design.

[0145] In other embodiments of this application, the quality parameters of the model to be evaluated are determined based on the quality evaluation results of all the literature data sets to be processed, including:

[0146] Step 1: Obtain the simulation operation set corresponding to the target literature data group to be processed.

[0147] The second quality assessment result of the target document data group is a match, meaning the target document data group is any one of the document data groups whose second quality assessment result is a match. In this embodiment, the steps are explained using the target document data group as an example. In practice, the steps in this embodiment need to be performed on every document data group whose second quality assessment result is a match.

[0148] In this embodiment, the simulation operation set corresponding to the target document data group to be processed will be obtained through a preset database.

[0149] Step 2: Determine the third number of operation steps based on the simulation operation set.

[0150] In this embodiment, the number of operation steps in the simulated operation set of the target document data group to be processed will be counted and used as a third quantity.

[0151] The third quantity is the number of operation steps in the simulated operation set.

[0152] Step 3: Obtain the fourth step in the operation process of the target literature data group to be processed.

[0153] In this embodiment, the number of operation steps in the operation process of the target document data group to be processed will be counted and used as the fourth quantity.

[0154] The fourth quantity is the number of operation steps in the operation process of the target literature data group to be processed.

[0155] Step 4: Based on the third and fourth quantities, determine the initial efficiency parameters of the target literature data set for the model to be evaluated.

[0156] In this embodiment, the fourth quantity is divided by the third quantity, and the result is used as the initial efficiency parameter of the model to be evaluated for the target literature data set to be processed.

[0157] The initial efficiency parameter is the quotient of the fourth quantity divided by the third quantity for the target literature data set, used to characterize the performance of the model to be evaluated in experimental design for the target literature data set.

[0158] Step 5: Based on the initial efficiency parameters of the model to be evaluated from all the literature data sets to be processed, determine the target efficiency parameters of the model to be evaluated.

[0159] In this embodiment, the average value of the initial efficiency parameters of the model to be evaluated for all the literature data sets to be processed is calculated and used as the target efficiency parameter of the model to be evaluated.

[0160] The target efficiency parameter is used to characterize the performance of the model under evaluation in experimental design for all the literature data sets to be processed.

[0161] It should be noted that in this embodiment, the simulation operation set corresponding to the target literature data set to be processed is obtained. Then, based on the simulation operation set, the third number of operation steps is determined. Next, the fourth number of operation steps in the operation process of the target literature data set to be processed is obtained. Based on the third and fourth numbers, the initial efficiency parameters of the target literature data set to be processed for the model to be evaluated are determined. Finally, based on the initial efficiency parameters of all literature data sets to be processed for the model to be evaluated, the target efficiency parameters of the model to be evaluated are determined, thus realizing the evaluation of the performance of the experimental design of the model.

[0162] It should be noted that in this application, the first accuracy parameter, the second accuracy parameter, and the target efficiency parameter of the model to be evaluated can be selected to evaluate the performance of the experimental design of the model, which solves the problem that the evaluation index is singular in the related technology and cannot comprehensively measure the performance of the model.

[0163] It should be noted that the first accuracy parameter and the second accuracy parameter are used to measure the factual consistency between the simulation results and the literature results, and to assess the accuracy of the scientific findings of the model being evaluated. The target efficiency parameter is used to measure the degree of optimization of the discovery process by tracking the average number of interaction rounds required to solve the literature questions.

[0164] In one complete embodiment of this application, such as Figure 5 As shown, an interactive closed-loop simulation evaluation framework for the entire wet experiment capability of large language models is presented, specifically designed for evaluating the full-process wet experiment capability of large language models. This framework quantifies the autonomous scientific experiment capability of models by simulating the complex environment of a real physics laboratory. The framework can separate and measure two core capabilities: the strategy synthesis capability of experimental schemes, and the logical rigor required to derive experimental conclusions. The entire system is divided into two stages: a scientific task generation stage and an evaluation stage where the model under evaluation acts as the experimenter, completely simulating the entire process from knowledge absorption to autonomous experimental reasoning.

[0165] First, the scientific task generation phase includes structured protocol extraction, laboratory resource inventory construction, and extraction of laboratory hardware and software input / output information.

[0166] Structured protocol extraction specifically involves: for multiple pre-defined original documents... Where D represents a set of multiple original documents, D1, D2, and D... n For a given original document, a structured protocol, i.e. a document data set, will be extracted from each original document using a standard analysis model.

[0167] For example, structured protocol extraction can be represented as:

[0168] ;

[0169] in, The structured protocol for the original documents, i.e., document data sets; For standard analysis models; For a specific original document; These are preset prompts used to guide the model in identifying and capturing key experimental attributes.

[0170] The structured protocol of the original documents, i.e., the document data set, can be represented as:

[0171] ;

[0172] in, The structured protocol for the original documents, i.e., document data sets; The title refers to a specific research question. This is the operational procedure, including detailed experimental steps and operational procedures; This refers to the conclusions of the literature, i.e., the final experimental results obtained.

[0173] The construction of the laboratory resource inventory specifically involves: achieving automated experiment execution. In this embodiment, laboratory information and core assets are directly derived from the literature data set to construct a comprehensive laboratory resource inventory. Multiple tool sets are constructed based on the corresponding document types of all the original documents; that is, one tool set corresponds to one literature data set, but one tool set may correspond to multiple literature data sets.

[0174] For example, the construction of a laboratory resource inventory can be represented as:

[0175] ;

[0176] in, A set of tools; Document data groups of the same document type; These are preset prompts.

[0177] The extraction of laboratory hardware and software input / output information specifically involves: to bridge the gap between high-level experimental descriptions and executable laboratory operations, it is necessary to formally define the operational boundaries of available resources. Based on multiple pre-defined original documents, this embodiment transforms unstructured text into structured specifications of laboratory tool capabilities, constructing hardware or software input / output parameters. A large language model is used to extract the tool's input / output parameters from each original document; subsequently, all original documents are traversed, and the input / output parameters of the same tool are deduplicated and globally aggregated to synthesize the complete input / output parameters for each tool.

[0178] For example, a tool and its corresponding input / output parameters can be represented as:

[0179] ;

[0180] in, For a specific tool and its corresponding input / output parameters; The type of tool, including physical hardware or software systems; These are the input and output parameters of the tool, i.e., the functional mapping between specific inputs and outputs under different experimental scenarios.

[0181] The evaluation phase of the model to be evaluated as an experimenter includes: task initiation and strategy planning, single-step execution and experience feedback, and conclusion synthesis and multi-dimensional evaluation.

[0182] Specifically, the task initiation and strategy planning process begins with analyzing the literature topic. Once the model to be evaluated becomes the subject of the literature study, it serves as the core decision-making unit and must formulate a strategic experimental plan through multi-step CoT reasoning. Furthermore, to bridge the gap between high-level scientific hypotheses and the constraints of the physical laboratory, the model to be evaluated will also query the toolkit corresponding to the literature topic in real time to obtain the precise functional specifications and operational boundaries of all relevant tools within the toolkit.

[0183] For example, task initiation and strategy planning can be represented as:

[0184] ;

[0185] in, The operational steps obtained by simulating the model to be evaluated; For a certain literature topic; for; This refers to the set of tools corresponding to the topic in this literature. This is the current set of simulation operations. This step ensures that the model to be evaluated is calibrated based on the input-output parameters of the real tool, effectively transforming abstract research objectives into executable operational steps.

[0186] Single-step execution and experience feedback: To simulate the inherent "observation-adjustment" cycle in real scientific research, this embodiment employs a strict single-step execution protocol. In the continuously evolving experimental plan, each operational step is sequentially sent to a standard analysis model, serving as the high-fidelity operational and physical layer of the virtual laboratory environment.

[0187] Unlike the batch processing method of related technologies that blindly executes a series of instructions, the standard analysis model in this embodiment processes each operation step individually based on predefined environmental logic, real-time physical constraints, and the current state space, and returns the corresponding operation result.

[0188] For example, single-step execution and experience feedback can be represented as:

[0189] ;

[0190] in, The result of the operation; For standard analysis models; These are the operating steps; This includes the current state of the laboratory environment, historical operating procedures, all original documents, and the corresponding toolkit.

[0191] It should be noted that single-step execution and empirical feedback ensure that the model under evaluation receives immediate and realistic feedback at each operational step. The operational results are integrated into the simulation operation set in real time, transforming the raw data into structured empirical evidence to drive subsequent inferences of the standard analytical model, enabling real-time error detection and correction, dynamic hypothesis revision, and precise calibration of downstream experimental parameters based on intermediate results.

[0192] The conclusion synthesis and multi-dimensional evaluation process is as follows: When the model to be evaluated determines that it has collected sufficient evidence to address the research topic in the literature, the interactive loop terminates. The process then proceeds to the conclusion synthesis stage, where the standard analysis model performs in-depth analysis of the simulated operation set to generate simulated conclusions. This stage not only assesses the model's ability to interpret data but also its proficiency in extracting scientific principles from fragmented experimental phenomena. Finally, the standard analysis model compares the simulated conclusions with the literature conclusions to evaluate the quality of the model under evaluation.

[0193] For example, the conclusion synthesis and multidimensional evaluation can be expressed as:

[0194] ;

[0195] in, This is a simulation conclusion; This is a research topic; This is a set of simulated operations.

[0196] It should be noted that, to verify the effectiveness of the simulation evaluation framework, we constructed a high-fidelity benchmark set containing 150 structured experimental tasks. These tasks cover core scientific fields, were generated with the assistance of Claude-sonnet-4-5, and rigorously audited by domain experts to ensure their logical integrity. The evaluation environment includes five toolkits involving thousands of hardware, software, and reagent entities. Each tool defines dozens of expert-validated input and output parameters to rigorously test the tool-calling capabilities of the models under evaluation.

[0197] We selected eight advanced large language models for benchmarking, including the GPT-5.4 series, Gemini-3.1 series, and DeepSeek-v3.2. The temperature parameter was set to 0.0 to ensure repeatability, and the maximum number of dialogue rounds was limited to 20. Each task allowed the model a maximum of five independent attempts to evaluate its robustness to random failures.

[0198] We applied the aforementioned interactive closed-loop simulation evaluation framework to the eight large language models described above, such as... Figure 6As shown, the evaluation results of each model on 150 experimental task datasets are presented. Pass@1 represents the first accuracy parameter, Pass@5 the second accuracy parameter, and Efficiency the initial efficiency parameter. It can be seen that the evaluation results exhibit significant hierarchical differences. Top-tier models, represented by GPT-5.4, demonstrate superior analytical capabilities, capable of synthesizing complex biological mechanisms and accurately interpreting experimental feedback. The interactive closed-loop simulation evaluation framework successfully identified key deficiencies in existing models. For example, Gemini-3.1 exhibited operational redundancy bottlenecks in multiple tasks, easily getting bogged down in preparation cycles and failing to achieve core experimental milestones. In summary, the evaluation data not only validated the capability boundaries of different large language models in the process of autonomous scientific discovery but also fully demonstrated the superiority of the interactive closed-loop simulation evaluation framework, capable of capturing deep logical problems that traditional static evaluations cannot expose.

[0199] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0200] Based on the same inventive concept, this application also provides a model evaluation apparatus for implementing the model evaluation method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more model evaluation apparatus embodiments provided below can be found in the limitations of the model evaluation method described above, and will not be repeated here.

[0201] In one embodiment of this application, such as Figure 7 As shown, a model evaluation apparatus is provided, comprising:

[0202] The acquisition module 100 is used to acquire the model to be evaluated, the standard analysis model, the literature data set to be processed, and the tool set corresponding to the literature data set to be processed; the literature data set to be processed includes: literature topic, operation process, and literature conclusion.

[0203] The step determination module 200 is used to input the literature topic and the toolset into the model to be evaluated and determine the first operation step.

[0204] The construction module 300 is used to input the first operation step, multiple original documents, and toolset into a standard analysis model, determine the first operation result corresponding to the first operation step, and construct a simulation operation set; the multiple original documents include the original documents corresponding to the data set of documents to be processed.

[0205] The update module 400 is used to determine multiple operation steps and the operation results corresponding to each operation step based on the simulation operation set, literature topics, tool set, multiple original documents, model to be evaluated, and standard analysis model, and update the simulation operation set.

[0206] The conclusion determination module 500 is used to input the updated simulation operation set and literature topics into the standard analysis model to determine the simulation conclusions.

[0207] The evaluation module 600 is used to input the simulation conclusions and literature conclusions into the standard analysis model to evaluate the quality of the model to be evaluated.

[0208] In other embodiments of this application, the acquisition module 100 is further configured to acquire a standard analysis model and multiple original documents; input the target original document into the standard analysis model to determine the document data set corresponding to the target original document and the document type corresponding to the target original document; input the target original document and / or the document data set corresponding to the target original document into the standard analysis model to determine the tool corresponding to the target original document and the input and output parameters of the tool; construct a document data set based on the document data sets corresponding to all the original documents; and construct multiple tool sets based on the document types, tools, and input and output parameters corresponding to all the original documents.

[0209] In other embodiments of this application, the acquisition module 100 is further configured to merge the tools and corresponding input / output parameters of the original documents of biological type to determine a biological tool set; merge the tools and corresponding input / output parameters of the original documents of physical type to determine a physical tool set; and merge the tools and corresponding input / output parameters of the original documents of chemical type to determine a chemical tool set.

[0210] In other embodiments of this application, the update module 400 is further configured to input the literature topic, the toolset, and the updated simulation operation set into the model to be evaluated to determine the second operation step; input the updated simulation operation set, the second operation step, multiple original documents, and the toolset into a standard analysis model to determine the second operation result corresponding to the second operation step, and update the simulation operation set; cyclically determine the operation step and determine the operation result until the preset conditions are met to obtain the final updated simulation operation set.

[0211] In other embodiments of this application, the evaluation module 600 is further configured to input the simulation conclusions and literature conclusions into a standard analysis model to determine the quality evaluation results of the literature data set to be processed on the model to be evaluated; obtain the quality evaluation results of all literature data sets to be processed on the model to be evaluated; and determine the quality parameters of the model to be evaluated based on the quality evaluation results of all literature data sets to be processed on the model to be evaluated.

[0212] In other embodiments of this application, the evaluation module 600 is further configured to determine a first number of matching literature data groups based on the first quality evaluation result of all literature data groups to be processed for the model to be evaluated; the first quality evaluation result is obtained by performing a quality evaluation once for each literature data group to be processed; count the total number of literature data groups to be processed; and determine a first accuracy parameter of the model to be evaluated based on the first number and the total number.

[0213] In other embodiments of this application, the evaluation module 600 is further configured to determine a second number of matching literature data groups based on the second quality evaluation results of all literature data groups to be processed for the model to be evaluated; the second quality evaluation results are obtained by performing multiple quality evaluations for each literature data group to be processed; the total number of literature data groups to be processed is counted; and a second accuracy parameter of the model to be evaluated is determined based on the second number and the total number.

[0214] In other embodiments of this application, the evaluation module 600 is further configured to: obtain a simulated operation set corresponding to the target literature data set to be processed; the second quality evaluation result of the target literature data set to be processed is a match; determine a third number of operation steps based on the simulated operation set; obtain a fourth number of operation steps in the operation process of the target literature data set to be processed; determine the initial efficiency parameter of the target literature data set to be processed for the model to be evaluated based on the third number and the fourth number; and determine the target efficiency parameter of the model to be evaluated based on the initial efficiency parameters of all literature data sets to be processed for the model to be evaluated.

[0215] Each module in the aforementioned model evaluation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0216] In one embodiment of this application, a computer device is provided, which may be a server, and its internal structure diagram may be as follows. Figure 8 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores all relevant data for executing model evaluation methods. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements a model evaluation method.

[0217] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0218] In one embodiment of this application, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the model evaluation method in the above embodiment.

[0219] In one embodiment of this application, a computer-readable storage medium is provided, on which a computer program is stored, the computer program being executed by a processor to implement the steps of the model evaluation method in the above-described method embodiments.

[0220] In one embodiment of this application, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the model evaluation method in the above-described method embodiments.

[0221] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0222] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0223] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0224] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A model evaluation method, characterized in that, The method includes: The process involves acquiring the model to be evaluated, the standard analysis model, the set of literature data to be processed, and the corresponding set of tools; the set of literature data to be processed includes: the literature topic, the operation process, and the literature conclusions. Input the literature topics and the toolset into the model to be evaluated to determine the first operation step; The first operation step, multiple original documents, and toolset are input into the standard analysis model to determine the first operation result corresponding to the first operation step and to construct a simulation operation set; the multiple original documents include the original documents corresponding to the data set of documents to be processed. Based on the simulated operation set, literature topics, tool set, multiple original documents, model to be evaluated, and standard analysis model, multiple operation steps and the operation results corresponding to each operation step are determined, and the simulated operation set is updated. Input the updated set of simulation operations and the literature topics into the standard analysis model to determine the simulation conclusions; The simulation results and literature findings are input into a standard analysis model to evaluate the quality of the model to be evaluated.

2. The model evaluation method according to claim 1, characterized in that, The method further includes: Obtain the standard analytical model and multiple original documents; Input the target original document into the standard analysis model to determine the document data group and document type corresponding to the target original document. Input the target original document and / or the document data set corresponding to the target original document into the standard analysis model to determine the tool corresponding to the target original document and the input and output parameters of the tool; A literature data set is constructed based on the literature data groups corresponding to all the original documents; Based on the corresponding document types, tools, and input / output parameters of all the original documents, multiple tool sets are constructed.

3. The model evaluation method according to claim 2, characterized in that, The construction of multiple tool sets based on the corresponding document types, tools, and input / output parameters of all the original documents includes: The tools and their corresponding input and output parameters of the original documents of the biological type are merged to determine the biological tool set; The tools and their corresponding input and output parameters for original documents of the physics type are merged to determine the physics tool set; The tools and their corresponding input and output parameters of the original documents of the chemistry type are merged to determine the chemistry tool set.

4. The model evaluation method according to claim 1, characterized in that, The process of determining multiple operational steps and the corresponding operational results for each operational step based on the simulation operation set, literature topics, tool sets, multiple original documents, the model to be evaluated, and the standard analysis model, and updating the simulation operation set includes: Input the literature topic, the toolset, and the updated simulation operation set into the model to be evaluated to determine the second operation step; The updated set of simulation operations, the second operation step, multiple original documents, and the toolset are input into the standard analysis model to determine the second operation result corresponding to the second operation step, and the set of simulation operations is updated. The operation steps and results are determined iteratively until the preset conditions are met, resulting in the final updated set of simulated operations.

5. The model evaluation method according to claim 1, characterized in that, The simulation results and literature findings are input into a standard analysis model, and the quality assessment of the model to be evaluated includes: The simulation results and literature results are input into the standard analysis model to determine the quality assessment results of the literature data set to be processed on the model to be evaluated. Obtain the quality assessment results of the model to be evaluated from all the literature datasets to be processed; Based on the quality assessment results of all the literature data sets to be processed on the model to be evaluated, the quality parameters of the model to be evaluated are determined.

6. The model evaluation method according to claim 1, characterized in that, The quality parameters of the model to be evaluated, determined based on the quality assessment results of all the literature datasets to be processed, include: Based on the first quality assessment results of the model to be evaluated for all the literature data groups to be processed, the first number of matching literature data groups to be processed is determined; the first quality assessment results are obtained by performing a quality assessment once for each of the literature data groups to be processed. Count the total number of the document data sets to be processed; Based on the first quantity and the total quantity, a first accuracy parameter is determined for the model to be evaluated.

7. The model evaluation method according to claim 1, characterized in that, The quality parameters of the model to be evaluated, determined based on the quality assessment results of all the literature datasets to be processed, include: Based on the second quality assessment results of the model to be evaluated for all the literature data sets to be processed, the second number of matching literature data sets to be processed is determined; the second quality assessment results are obtained by performing multiple quality assessments for each literature data set to be processed. Count the total number of the document data sets to be processed; Based on the second quantity and the total quantity, a second accuracy parameter for the model to be evaluated is determined.

8. The model evaluation method according to claim 1, characterized in that, The quality parameters of the model to be evaluated, determined based on the quality assessment results of all the literature datasets to be processed, include: Obtain the simulation operation set corresponding to the target document data set to be processed; the second quality assessment result of the target document data set to be processed is a match. Based on the simulated operation set, determine the third number of operation steps; Obtain the fourth number of operation steps in the operation process of the target document data group to be processed; Based on the third and fourth quantities, the initial efficiency parameters of the target literature data set for the model to be evaluated are determined. Based on the initial efficiency parameters of the model to be evaluated from all the literature datasets to be processed, the target efficiency parameters of the model to be evaluated are determined.

9. A model evaluation device, characterized in that, The device includes: The acquisition module is used to acquire the model to be evaluated, the standard analysis model, the literature data set to be processed, and the tool set corresponding to the literature data set to be processed; the literature data set to be processed includes: literature topic, operation process, and literature conclusion. The step determination module is used to input the literature topic and the toolset into the model to be evaluated and determine the first operation step. The construction module is used to input the first operation step, multiple original documents, and toolset into a standard analysis model, determine the first operation result corresponding to the first operation step, and construct a simulation operation set; the multiple original documents include the original documents corresponding to the data set of documents to be processed. The update module is used to determine multiple operation steps and the operation results corresponding to each operation step based on the simulation operation set, literature topics, tool set, multiple original documents, model to be evaluated, and standard analysis model, and update the simulation operation set. The conclusion determination module is used to input the updated simulation operation set and literature topics into the standard analysis model to determine the simulation conclusions. The evaluation module is used to input the simulation results and literature results into the standard analysis model to evaluate the quality of the model to be evaluated.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images