Method, device and equipment for generating evaluation dataset, and storage medium

Abstract

The application provides a method and device for generating an evaluation data set, and a storage medium. For each evaluation question about earth science knowledge, a large language model is used to predict tool information of a target tool required to solve the evaluation question, a difficulty level of the evaluation question is determined based on the tool information of the target tool, and a corresponding strategy is used to generate a test case according to the difficulty level of the evaluation question. The test case includes the evaluation question, an evaluation reference answer, and evaluation reasoning steps. Based on each evaluation reasoning step and a corresponding scoring item, evaluation data corresponding to the test case is generated. The scoring item is used to verify the consistency between an actual reasoning step of an earth science large language model and the evaluation reasoning step. Based on the evaluation data corresponding to each test case, an evaluation data set is generated, which is used to evaluate the earth science large language model. The embodiments of the application can improve the evaluation quality of the evaluation data set.

Description

Technical Field

[0001] This application relates to the field of large model technology, and more specifically, to a method, apparatus, device, and storage medium for generating evaluation datasets. Background Technology

[0002] With the rapid development of artificial intelligence technology, large language models are being applied more and more widely in the field of Earth science. Earth science research usually involves multi-source data fusion, complex computational analysis, and the use of specialized tools (such as remote sensing image interpretation, seismic waveform processing, numerical simulation, etc.). Therefore, the evaluation of large language models in geosciences needs not only to examine their knowledge memorization ability, but also to assess their ability to use external tools to complete scientific research tasks. Summary of the Invention

[0003] In view of this, this application provides a method, apparatus, device and storage medium for generating evaluation datasets, so as to at least solve the problems existing in the related art.

[0004] Specifically, this application is implemented through the following technical solution: This application provides a method for generating an evaluation dataset, including: Obtain a set of assessment questions; the set of assessment questions includes multiple assessment questions about Earth science knowledge; For each evaluation question, a large language model is used to predict the tool information of the target tool required to solve the evaluation question, and the difficulty level of the evaluation question is determined based on the tool information of the target tool. Based on the difficulty level of the evaluation question, a test case generation strategy corresponding to the difficulty level is adopted to generate test cases; the test cases include the evaluation question, the evaluation reference answer to the evaluation question, and at least one evaluation reasoning step performed to generate the evaluation reference answer; For each evaluation reasoning step of each test case, a scoring item is constructed for the evaluation reasoning step, and evaluation data corresponding to the test case is generated based on each evaluation reasoning step and its corresponding scoring item; the scoring item is used to verify the consistency between the actual reasoning steps of the Earth Science Big Language Model and the evaluation reasoning steps. An evaluation dataset is generated based on the evaluation data corresponding to each test case. This evaluation dataset is used to evaluate the Earth Science Large Language Model.

[0005] This application also provides an apparatus for generating an evaluation dataset, comprising: The evaluation question acquisition module is used to acquire a set of evaluation questions; the set of evaluation questions includes multiple evaluation questions about earth science knowledge; The difficulty level determination module is used to predict the tool information of the target tool required to solve each evaluation problem using a large language model, and to determine the difficulty level of the evaluation problem based on the tool information of the target tool. The test case generation module is used to generate test cases based on the difficulty level of the evaluation question, using an evaluation case generation strategy corresponding to the difficulty level; the test cases include the evaluation question, the evaluation reference answer to the evaluation question, and at least one evaluation reasoning step performed to generate the evaluation reference answer; The evaluation data generation module is used to construct a scoring item for each evaluation reasoning step of each test case, and generate evaluation data corresponding to the test case based on each evaluation reasoning step and its corresponding scoring item; the scoring item is used to verify the consistency between the actual reasoning steps of the Earth Science Big Language Model and the evaluation reasoning steps. The evaluation dataset generation module is used to generate an evaluation dataset based on the evaluation data corresponding to each test case. The evaluation dataset is used to evaluate the Earth Science Large Language Model.

[0006] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the evaluation dataset generation method described in any of the foregoing embodiments.

[0007] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the evaluation dataset generation method described in any of the foregoing embodiments.

[0008] This application also provides a computer program product, including a computer program that, when run by a processor, performs the steps of any of the possible evaluation dataset generation methods described above.

[0009] The technical solutions provided by the embodiments of this application may include the following beneficial effects: In this embodiment, the difficulty level is determined based on the tool information of the target tools required to solve the evaluation problem. Then, according to the difficulty level, a corresponding evaluation case generation strategy is adopted to generate test cases. The test cases include the evaluation problem, the evaluation reference answer to the evaluation problem, and at least one evaluation reasoning step performed to generate the evaluation reference answer. In this way, by evaluating the model based on the evaluation dataset, it is possible to accurately distinguish whether the Earth Science Big Language Model is "rote memorization" or "possesses scientific research-level tool calling and reasoning capabilities". This makes the evaluation results more reflective of the usability of the Earth Science Big Language Model in real geoscience scientific research tasks.

[0010] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0011] Figure 1 This is a flowchart illustrating an exemplary embodiment of the present application of a method for generating an evaluation dataset; Figure 2 This is a flowchart illustrating a test case generation process according to an exemplary embodiment of this application; Figure 3 This is a schematic diagram illustrating the number of test cases and the distribution of sub-disciplines at the first level, as shown in an exemplary embodiment of this application; Figure 4 This is a schematic diagram illustrating the number of test cases and the distribution of sub-disciplines at the second level, as shown in an exemplary embodiment of this application; Figure 5 This is a schematic diagram illustrating the number of test cases and the distribution of sub-disciplines at the third level, as shown in an exemplary embodiment of this application; Figure 6 This is a schematic diagram of the structure of an evaluation dataset generation device shown in an exemplary embodiment of this application; Figure 7 This is a hardware structure diagram of a computer device illustrated in an exemplary embodiment of this application. Detailed Implementation

[0012] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0013] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0014] It should be understood that although the terms first, second, third, etc., may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0015] With the rapid development of artificial intelligence technology, large language models are being applied more and more widely in the field of Earth science. Earth science research usually involves multi-source data fusion, complex computational analysis, and the use of specialized tools (such as remote sensing image interpretation, seismic waveform processing, numerical simulation, etc.). Therefore, the evaluation of large language models in geosciences needs not only to examine their knowledge memorization ability, but also to assess their ability to use external tools to complete scientific research tasks.

[0016] Currently, the main technical problems with evaluation datasets for large language models are as follows: 1. The difficulty level is categorized by a single dimension, neglecting the ability to use the tools. Most existing evaluation datasets categorize difficulty levels based on the depth of knowledge points or the complexity of questions, such as distinguishing between fact-based questions and reasoning-based questions. However, many problems in Earth science, while involving simple knowledge points, require complex toolchains (such as code interpreters, spatial analysis tools, and visualization tools) to solve. Traditional evaluation methods cannot scientifically measure the model's real-world performance in tool-initiated scenarios, leading to a disconnect between evaluation results and the model's actual application capabilities.

[0017] 2. Lack of fine-grained evaluation of the problem-solving process. Current mainstream evaluation methods typically employ a "question-answer" pair format, focusing solely on the correctness of the model's final output while ignoring the crucial reasoning steps taken by the model during problem-solving. This includes factors such as whether the correct tools were chosen, whether the tool parameters were appropriate, and whether intermediate data underwent necessary validation. This "results-oriented" evaluation fails to diagnose process errors in complex research tasks and struggles to detect the illusion of "correct answer but flawed process."

[0018] 3. The cost of constructing highly complex evaluation data is high. Complex test cases that can examine the collaborative and multi-turn interaction capabilities of multiple tools (such as dialogue data containing complete tool call trajectories, intermediate outputs, and inference chains) require significant effort from domain experts for manual design and annotation, making them difficult to scale. This limits the coverage and update frequency of evaluation datasets, hindering the in-depth development of large-scale model applications in the geosciences.

[0019] 4. Lack of assessment of dynamic environmental adaptability In real-world scientific research, dynamic changes frequently occur, such as data version updates and the overturning of initial hypotheses, requiring models to have the ability to adjust their analytical approach and revise their conclusions in a timely manner. However, existing evaluation datasets are all static data, which cannot simulate such dynamic challenges, making it difficult to assess the robustness and adaptability of models.

[0020] Based on the above, it can be seen that in related technologies, most evaluation datasets are divided into difficulty levels according to the depth of knowledge points or the complexity of questions, such as distinguishing between fact-based questions and reasoning questions. This approach has a single dimension and only focuses on the evaluation results, making it difficult to achieve scientific measurement.

[0021] Furthermore, how to automatically generate evaluation datasets that can scientifically reflect the model's tool usage capabilities, process reasoning abilities, and dynamic adaptability has become an urgent problem to be solved in the field of large language model evaluation technology for Earth science.

[0022] Based on the above research, this disclosure provides a method for generating an evaluation dataset. The method first obtains a set of evaluation questions, including multiple evaluation questions about Earth science knowledge. Second, for each evaluation question, a large language model is used to predict the tool information of the target tool required to solve the evaluation question, and based on the tool information, the difficulty level of the evaluation question is determined. Next, according to the difficulty level of the evaluation question, an evaluation case generation strategy corresponding to the difficulty level is adopted to generate test cases. Each test case includes the evaluation question, the evaluation reference answer to the evaluation question, and at least one evaluation reasoning step performed to generate the evaluation reference answer. Then, for each evaluation reasoning step of each test case, a scoring item for the evaluation reasoning step is constructed, and based on each evaluation reasoning step and its corresponding scoring item, evaluation data corresponding to the test case is generated. The scoring item is used to verify the consistency between the actual reasoning steps of the Earth science large language model and the evaluation reasoning steps. Finally, based on the evaluation data corresponding to each test case, an evaluation dataset is generated, which is used to evaluate the Earth science large language model.

[0023] In this embodiment, the difficulty level is determined based on the tool information of the target tools required to solve the evaluation problem. Then, according to the difficulty level, a corresponding evaluation case generation strategy is adopted to generate test cases. The test cases include the evaluation problem, the evaluation reference answer to the evaluation problem, and at least one evaluation reasoning step performed to generate the evaluation reference answer. In this way, by evaluating the model based on the evaluation dataset, it is possible to accurately distinguish whether the Earth Science Big Language Model is "rote memorization" or "possesses scientific research-level tool calling and reasoning capabilities". This makes the evaluation results more reflective of the usability of the Earth Science Big Language Model in real geoscience scientific research tasks.

[0024] To facilitate understanding of this embodiment, a method for generating an evaluation dataset disclosed in this disclosure will first be described in detail. The execution entity of the evaluation dataset generation method provided in this disclosure is generally a computer device. This computer device can be a server, which can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud storage, big data, and artificial intelligence platforms. In other embodiments, the computer device can also be a terminal device, which can be a mobile device, terminal, handheld device, computing device, vehicle-mounted device, etc.

[0025] In other embodiments, the method can also be applied to an implementation environment consisting of computer equipment and servers, or an implementation environment consisting of terminal equipment and servers. Furthermore, the method for generating the evaluation dataset can also be implemented by a processor calling computer-readable instructions stored in memory.

[0026] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0027] Please see the appendix Figure 1 The above is a flowchart illustrating a method for generating an evaluation dataset, as shown in an exemplary embodiment of this application. Figure 1 As shown, the method for generating the evaluation dataset in this embodiment includes the following steps S101~S105: S101: Obtain a set of evaluation questions; the set of evaluation questions includes multiple evaluation questions about Earth science knowledge.

[0028] Here, the evaluation question set refers to the question bank used to test the Earth Science Large Language Model, which includes multiple evaluation questions about Earth Science knowledge. Earth Science includes multiple sub-disciplines, including but not limited to seismology, mineralogy, remote sensing, hydrology, oceanography, meteorology, geology, geochemistry, etc. Therefore, Earth Science knowledge includes sub-discipline knowledge of the above-mentioned sub-disciplines.

[0029] Each evaluation question can be generated through at least one of three methods: manual design, extraction from literature, and generation with the assistance of a large language model. No specific method is required here.

[0030] S102: For each evaluation problem, use a large language model to predict the tool information of the target tool required to solve the evaluation problem, and determine the difficulty level of the evaluation problem based on the tool information of the target tool.

[0031] In this implementation, "large language model" refers to a general pre-trained model with powerful natural language understanding and generation capabilities, such as GPT-4, Claude, and Wenxin Yiyan.

[0032] In this embodiment of the application, when using a large language model to predict the tool information of the target tool required to solve the evaluation problem, the prediction can be made in conjunction with a preset tool specification library. The tool specification library is used to provide tools of various types, including information retrieval, analysis and processing, visualization and document communication. Each type of tool has corresponding attribute information, which includes functional description, applicable difficulty level, typical use cases and calling parameter specifications, as shown in Table 1.

[0033] Table 1

[0034]

[0035] In this implementation, the difficulty levels include a first level, a second level, and a third level, with the difficulty increasing sequentially. The first level (L1) refers to a knowledge-based problem that can be solved without using any tools. The second level (L2) refers to a research-based problem that requires using a tool step-by-step to solve. The third level (L3) refers to an integrated problem that requires the collaborative use of multiple tools.

[0036] The configuration of each difficulty level is described in detail below.

[0037] Difficulty level configuration: Define the core parameters for three levels: L1 / L2 / L3, including allowed question types, maximum number of generated questions, answer types, dialogue types, and keyword features, as detailed below: L1 (Knowledge and Ability): Allows question types such as multiple choice, fill-in-the-blank, terminology definition, and short answer; maximum number of generated questions is 300; answer type is a concise and unique standard answer; dialogue type is single turn; keywords include "definition", "what is", "selection", "fill-in-the-blank", "identification", etc.

[0038] L2 (Research Ability): Allows question types such as discussion questions, calculation questions, coding questions, experimental design, and paper reproduction; maximum number of generated questions is 300; answer type is multi-faceted detailed reference answer; dialogue type is single-turn open-ended; keywords include "analysis", "calculation", "implementation", "design", "reproduction", "discussion", etc.

[0039] L3 (Multi-tool integration capability): Allows question types such as remote sensing image interpretation, geological map analysis, multi-source data fusion, and report generation; maximum number of generated questions: 50; response type: tool-assisted multi-turn dialogue; dialogue type: multiple rounds (≥2 rounds); keywords include "multi-source", "literature review", "image analysis", "data integration", "report generation", etc.

[0040] S103: Based on the difficulty level of the evaluation question, a test case generation strategy corresponding to the difficulty level is adopted to generate test cases; the test cases include the evaluation question, the evaluation reference answer to the evaluation question, and at least one evaluation reasoning step performed to generate the evaluation reference answer.

[0041] The evaluation reference answer refers to the ideal answer generated by the large language model that conforms to the norms of geosciences, and serves as the standard for measuring the model's output.

[0042] The evaluation reasoning steps can refer to the key logical steps (e.g., knowledge retrieval, data extraction, calculation verification, conclusion integration) that are arranged in sequence during the process of generating the reference answer. Each step corresponds to a standard action that can be verified.

[0043] In this step, for each evaluation question, the difficulty level (L1 / L2 / L3) is first determined using a large language model. Then, based on the difficulty level, different generation strategies are selected: for low-difficulty questions, the answer is generated directly; for high-difficulty (L3) questions, the "reverse thinking chain decomposition" technique is used to generate a detailed reference answer and each evaluation reasoning step involved in generating the answer (e.g., first retrieving background knowledge, then extracting data, performing calculations, and finally integrating the conclusions).

[0044] S104: For each evaluation reasoning step of each test case, construct a scoring item for the evaluation reasoning step, and generate evaluation data corresponding to the test case based on each evaluation reasoning step and its corresponding scoring item; the scoring item is used to verify the consistency between the actual reasoning steps of the Earth Science Big Language Model and the evaluation reasoning steps.

[0045] Among them, the Earth Science Big Language Model refers to a specialized model formed on the basis of the Big Language Model by fine-tuning Earth Science data, adding domain knowledge bases, or integrating professional tool interfaces.

[0046] Here, the scoring items refer to the quantifiable checking rules constructed for each evaluation inference step (such as judging whether the model calls a certain tool, whether the parameters are correct, and whether the intermediate results are reasonable), which are used to evaluate the actual performance of the model.

[0047] Evaluation data refers to structured data generated for a test case, which includes questions, reference answers, reasoning steps, and corresponding scoring items, and is usually stored in JSON format.

[0048] In this step, for each evaluation inference step in each test case, one or more quantifiable scoring checks are constructed (e.g., Did the model call the correct tool? Is the deviation of the calculation result less than 5%?). Then, these evaluation inference steps are combined with the corresponding scoring items to form a structured "evaluation data" for subsequent step-by-step verification of the model.

[0049] S105: Based on the evaluation data corresponding to each test case, generate an evaluation dataset, which is used to evaluate the Earth Science Large Language Model.

[0050] In this step, the evaluation data corresponding to all test cases are collected to form a standardized evaluation dataset (usually in JSONL format). This evaluation dataset can be directly used to automatically evaluate the Earth Science Large Language Model, outputting the model's reasoning ability scores in multiple dimensions such as knowledge memory, tool invocation, and logical reasoning.

[0051] In this embodiment, the difficulty level is determined based on the tool information of the target tools required to solve the evaluation problem. Then, according to the difficulty level, a corresponding evaluation case generation strategy is adopted to generate test cases. The test cases include the evaluation problem, the evaluation reference answer to the evaluation problem, and at least one evaluation reasoning step performed to generate the evaluation reference answer. In this way, by evaluating the model based on the evaluation dataset, it is possible to accurately distinguish whether the Earth Science Big Language Model is "rote memorization" or "possesses scientific research-level tool calling and reasoning capabilities". This makes the evaluation results more reflective of the usability of the Earth Science Big Language Model in real geoscience scientific research tasks.

[0052] Furthermore, in related technologies, the evaluation datasets are divided into difficulty levels only based on the depth of knowledge points, ignoring the key dimension of "tool usage". This application introduces for the first time a three-level difficulty grading standard (knowledge-based, research-based, and multi-tool integration-based) based on tool dependence in the evaluation of large language models in earth sciences. This standard can effectively distinguish between the model's memory ability and scientific reasoning ability, and scientifically measure the model's true level in complex toolchain calling scenarios.

[0053] In some implementations, the evaluation questions can be generated using a combination of manual design and LLM-assisted expansion. First, earth science experts design core questions based on corpus content, covering basic knowledge points and typical research scenarios across various disciplines. Then, LLM calls the question expansion templates in PromptTemplates to generate variant evaluation questions. For example, "Calculate the amplitude ratio of P-waves to S-waves" can be expanded to "How to calculate the amplitude ratio of P-waves to S-waves based on seismic waveform data? What steps and tools are needed?", ensuring the scale and diversity of the question set.

[0054] The corpus refers to a multimodal raw corpus in the field of Earth sciences, which is constructed through the following steps: 1. Corpus construction requirements: Subject coverage requirements: The evaluation data must cover at least 8 sub-disciplines in earth sciences, including but not limited to seismology, mineralogy, remote sensing, hydrology, oceanography, meteorology, geology, and geochemistry, to ensure the comprehensiveness of the evaluation data across disciplines.

[0055] Modal diversity requirements: Include at least five types of modal data, such as text, waveform files (e.g., SEED format seismic data), remote sensing images (e.g., satellite RGB images, SAR images), geological maps (e.g., regional geological maps, profile maps), geochemical diagrams, and chronological tables, to meet the diverse data type needs of geoscience research.

[0056] Data quality requirements: Text data must ensure that the accuracy of professional terminology is ≥95%, and the data source is traceable (such as authoritative journal articles, research institution datasets, standard test libraries); File data (waveforms, images, geological maps) must be in a standardized format, undamaged, and have clear metadata descriptions (such as sampling rate, resolution, coordinate system, scale, etc.).

[0057] 2. Multimodal data collection and integration: Basic knowledge collection: Integrating key terms and standard unit systems from branches of geology, meteorology, oceanography, hydrology, and remote sensing, such as "rock," "strata," and "fault" in geology, and "precipitation," "air pressure," and "front" in meteorology. The unit system covers temperature (°C, K), length (m, km), area (°C, K), etc. , ), concentration (mg / L, ppm), etc., provide support for identifying the subject attributes of the problem and verifying the rationality of the data.

[0058] Text data collection: Core content, including research background, scientific questions, experimental methods, data results, conclusions and viewpoints, was extracted from high-quality papers published in authoritative journals such as Nature Geoscience, Geology and Chinese Journal of Geophysics over the past 20 years to construct a text corpus; at the same time, knowledge points and exercises from standard textbooks, technical manuals and research reports in the field of geosciences were collected to supplement the basic theoretical text data.

[0059] File-based data collection: Waveform files: Download SEED format waveform data recorded by global seismic stations from the IRIS (Incorporated Research Institutions for Seismology) database, covering earthquake events of different magnitudes and epicentral distances to ensure data diversity.

[0060] Remote sensing image collection: Acquire remote sensing images of different regions and resolutions from satellite data platforms such as Landsat and Sentinel, including optical images, SAR images, and thermal infrared images, covering various research scenarios such as topography, vegetation cover, and water distribution.

[0061] Geological map collection: Collect regional geological maps at different scales, including key information such as stratigraphy, lithology, geological structure, and mineral resources.

[0062] Other document collection: Collect geochemical analysis tables, chronological dating data, numerical simulation input and output files, etc., to enrich the data modal types.

[0063] Data integration: Organize data from different modalities according to the "question-data-answer" relationship. For example, for the question "analyze the polarity of the P-wave initial motion recorded at a certain station", link the corresponding SEED format waveform files, regional geological background text data, standard analysis results, etc., to form structured data pairs.

[0064] Each evaluation question has initial attributes, including the relevant sub-discipline, data modality type, expected tool requirements, difficulty level (initially labeled and subsequently optimized through step S102), and keyword features. For example, the attributes of the question "Inverting vegetation cover of a certain area based on Landsat 8 remote sensing imagery" are labeled as follows: sub-discipline is remote sensing; data modality is remote sensing imagery; expected tool requirements include Image QA / Interpretation, Code Interpreter, and Data Visualization; initial difficulty level is L2; ​​and keywords include remote sensing imagery, inversion, and vegetation cover.

[0065] Question set selection: By combining automated scripts with manual review, questions with vague descriptions, mismatched data, or unreasonable difficulty levels are filtered out, ultimately forming an initial question set of no less than 650 questions (L1=300 questions, L2=300 questions, L3=50 questions).

[0066] In some implementations, for step S102, when predicting the tool information of the target tool required to solve the evaluation problem using a large language model for each evaluation problem, the following steps (1) to (2) can be used: (1) For each assessment question, determine the sub-discipline type to which the assessment question belongs.

[0067] Large Language Modeling (LLM) can be used to identify which branch of Earth science the evaluation problem belongs to. For example, if the evaluation problem is "analyze the trend of river runoff in a certain area", the corresponding branch is hydrology; if the evaluation problem is "retrieve vegetation cover based on Landsat imagery", the corresponding branch is remote sensing.

[0068] This is easy to understand because different sub-disciplines typically require different tools, and the sub-discipline type can help the system narrow down the range of tools to choose from.

[0069] (2) Based on the sub-discipline type of the evaluation problem and the corresponding text semantics, determine the tool information of the target tool to be called to solve the evaluation problem; the tool information includes type, quantity and calling order.

[0070] Here, the big oracle model can be used to extract the textual semantics of the evaluation question, including keywords, sentence structure, and task intent (such as "calculate", "analyze", "invert", "generate report" etc.).

[0071] Based on the determination of the sub-discipline type, the tool information of the target tools required to solve the evaluation problem can be determined by text semantic judgment. The type in the tool information is used to explain which category each tool belongs to (such as code execution, retrieval, etc.), the quantity is the total number of target tools required, and the calling order refers to the execution (or calling) order of each target tool.

[0072] For example, the evaluation question is: "Combining Landsat 9 remote sensing images of the Tarim Basin with 1:250,000 geological maps, invert vegetation cover and analyze its relationship with faults, and generate a research report."

[0073] Sub-disciplines: Remote sensing, geology.

[0074] Text semantic analysis: This involves image parsing, numerical calculation (NDVI, pixel binarization), spatial overlay analysis, visualization, and report generation.

[0075] Output tool information: Types: Image QA / Interpretation, Code Interpreter, Geospatial Analysis, Data Visualization, Report Generator; Quantity: 5; Call order: Image QA, Code Interpreter, Geospatial Analysis, Data Visualization, Report Generator.

[0076] Furthermore, when determining the difficulty level of the assessment question based on the tool information of the target tool, the following steps (A) to (B) may be included: (A) Determine the logical dependencies between the target tools; the logical dependencies refer to workflow relationships where the output of one tool is used as the input of another tool.

[0077] As is easy to understand, logical dependency refers to whether there is a workflow association between multiple target tools required to solve the evaluation problem, such that "the output of one tool is used as the input of another tool". If there is such a forward and backward relationship between any two tools, it means that there is a logical dependency between the two tools; otherwise, it means that the two tools are independent or have no dependency relationship.

[0078] (B) Determine the difficulty level of the evaluation problem based on the tool information of the target tool and the logical dependencies.

[0079] In this step, after obtaining the tool information and logical dependencies of the target tool, the difficulty level of the evaluation problem can be determined.

[0080] The following is a detailed explanation of the classification of each level: (1) If the number of target tools is not greater than the first quantity threshold, and the number of target tool pairs with logical dependencies is less than the first threshold, the difficulty level of the evaluation problem is determined to be the first level.

[0081] In this embodiment, the first quantity threshold is 1.

[0082] It can be understood that if the number of target tools is no more than 1 (either 0 or 1), and the number of target tool pairs with logical dependencies is 0, that is, there are no logical dependencies between target tools, and the task logic depth is only 0 or 1, then the difficulty level of this evaluation problem is Level 1.

[0083] (2) If the number of target tools is between the first quantity threshold and the second quantity threshold, and the number of target tool pairs with logical dependencies is less than the first threshold, the difficulty level of the evaluation problem is determined to be the second level.

[0084] In this embodiment, the second quantity threshold is 2.

[0085] In this step, if the number of target tools is between 1 and 2, and the number of target tool pairs with logical dependencies is less than 1, it indicates that there is no dependency between the target tools. However, if the task logic depth is only between 1 and 2, then the difficulty level of this evaluation problem is level 2.

[0086] (3) If the number of target tools is not less than the second quantity threshold, and the number of target tool pairs with at least two different types of target tools and logical dependencies is not less than the first threshold, the difficulty level of the evaluation problem is determined to be the third level.

[0087] In this step, if the number of target tools is not less than 2, and the number of target tool pairs with at least two different types of target tools and logical dependencies is not less than 1, and the task logic depth is not less than the second threshold, then the difficulty level of the evaluation problem is the third level.

[0088] The second threshold is 2.

[0089] In other implementations, the values ​​of the first threshold, the second threshold, the first quantity threshold, and the second quantity threshold can be set according to actual needs. For example, different sub-disciplines have different levels of difficulty, so the values ​​of each threshold can be determined adaptively and are not limited here.

[0090] Table 2 shows the rules for determining the difficulty level: Table 2

[0091] In some implementations, the types of test cases generated vary depending on the difficulty level, including single-turn question-and-answer test cases and multi-turn dialogue test cases.

[0092] Specifically, regarding step S103, when generating test cases based on the difficulty level of the evaluation question and using an evaluation case generation strategy corresponding to the difficulty level, if the difficulty level of the evaluation question is the first level or the second level, the single-round question-and-answer test cases are generated based on the evaluation question, the corresponding reference answer, each scoring item and its corresponding score, the tool information of the target tool, and the preset test case generation rules.

[0093] Here, scoring items refer to quantifiable scoring points extracted from the reference answer, such as whether a search tool was used or whether the filter frequency band is within the range of 0.5-10Hz.

[0094] Each rating item is pre-assigned a corresponding score, and each rating item has an upper limit, such as 100 points.

[0095] The preset test case generation rules refer to predefined, structured generation templates or rules for different difficulty levels and question types. For example, the answer to an L1 level assessment question should be concise and unique, with the scoring criteria primarily based on knowledge accuracy; the answer to an L2 level assessment question needs to include code or steps, with scoring criteria including tool selection, parameter rationality, and result accuracy.

[0096] Here, when the difficulty level of the evaluation question is determined to be L1 (knowledge-based) or L2 (research-based), the system generates test cases using a single-round question-and-answer format. During this process, the system integrates multiple existing pieces of information (the question itself, the reference answer, scoring items and scores, and tool information) and follows predefined rules to ultimately output a structured test case that can be directly used for subsequent evaluations.

[0097] Specifically, the difficulty level is either Level 1 (L1) or Level 2 (L2). 1. Input information integration: Evaluation question: Original user question text.

[0098] Suggested answer: A correct answer generated by the system (concise or including steps).

[0099] Each scoring item and its score: Automatically constructed checkpoints, such as "Is the concept definition correct (30 points)?" and "Was the Code Interpreter called (20 points)?"

[0100] Target tool information: The tools needed for prediction (e.g., L2 problems may require a code interpreter).

[0101] Preset rules: Define how to combine these elements into a standardized test case structure.

[0102] 2. Generate single-turn question-and-answer test cases: The test cases consist of only one round of user questions and the expected agent responses (multiple rounds of dialogue do not need to be simulated).

[0103] The test cases will clearly indicate: the reference answer, the required tools (if any), a list of scoring points and their weights.

[0104] The format is usually structured data (such as JSON), which facilitates parsing by subsequent automated evaluation scripts.

[0105] 3. Output: A complete single-round question-and-answer test case, which can be directly stored in the evaluation dataset.

[0106] The following examples illustrate test cases for L1 and L2 levels: 1. L1 Level Single-Round Use Case Generation (Knowledge and Ability Type) Generated content: Each L1 question's evaluation test case contains four core parts: "original question, standard answer, scoring points, and tool call expectations (if any)".

[0107] Specific examples: Original question: "Define the concept of earthquake P-wave and explain its propagation characteristics." The standard answer is: "Earthquake P-waves (longitudinal waves) are a type of seismic wave and belong to body waves; the direction of particle vibration is parallel to the direction of wave propagation (compression and rarefaction processes exist); they can propagate in solids, liquids, and gases; they have the fastest propagation speed in the same medium and are the earliest seismic waves to reach the observation point after the earthquake." Scoring criteria: Total 100 points, of which "Accurate definition of concept" accounts for 40 points (Boolean check item: whether the core attribute of longitudinal waves, 'the direction of vibration is parallel to the direction of propagation', is clearly stated); "Propagation medium" accounts for 30 points (Boolean check item: whether it is mentioned that propagation can occur in solid, liquid, and gaseous media); "Propagation speed characteristics" accounts for 30 points (Boolean check item: whether the fastest / earliest speed is clearly stated).

[0108] Tool call expectation: None (this issue does not require a tool call).

[0109] Generation rules: Standard answers should be concise and unique, conforming to consensus in the field of geosciences; scoring points are fine-grained Boolean check items, with weight allocation based on the importance of knowledge points; tool calls are expected to be added only when the question requires basic tools (such as Calculator), and the tool names and parameter settings should be clearly specified.

[0110] 2. L2 Level Single-Round Use Case Generation (Research Capability Type) Specific examples: Case Study: "Calculate the amplitude ratio of S-wave to P-wave based on SEED format waveform data (station: YN.YB01, sampling rate 100Hz) from the M6.4 earthquake in Yangbi, Yunnan."

[0111] The generated content consists of five core parts: "original question, reference answer, tool call expectation, fault tolerance instructions, and scoring points" for each L2 question evaluation case.

[0112] Original question: "Based on the SEED format waveform data of the M6.4 earthquake in Yangbi, Yunnan (station: YN.YB01, sampling rate 100Hz, recording duration 120 seconds), calculate the amplitude ratio of S-wave to P-wave. Data file path: . / waveforms / YN_YB01_20210521.seed."

[0113] Suggested answer: "Step 1: Use the obspy library to load SEED format data, code:" ``` import obspy; st = obspy.read('. / waveforms / YN_YB01_20210521.seed') ``` Step 2: Perform instrument response processing on the data. Parameters: response file path . / response / YN_YB01.response, output unit is cm / s.

[0114] Step 3: Determine the P-band (0.5-10Hz) and S-band (1-5Hz) through spectrum analysis, and apply bandpass filtering.

[0115] Step 4: Extract the maximum amplitude of the S wave A_s = 3.2cm / s (time: 2021-05-21 12:55:42.68s) and the maximum amplitude of the P wave A_p = 1.6cm / s (time: 2021-05-21 12:55:42.36s).

[0116] Step 5: Calculate the amplitude ratio A_s / A_p = 2.0.

[0117] Expected tool call: Call the Code Interpreter tool to execute the above Python code. Dependencies include obspy (seismic data processing) and numpy (numerical computation). The code must contain complete logic for data loading, preprocessing, filtering, amplitude extraction, and ratio calculation, with reasonable parameter settings (such as filter band and response file path).

[0118] Fault Tolerance Instructions: If the Code Interpreter fails, the following scenarios will be handled: 1. Incorrect data format (not SEED / MINISEED format): Prompt the user to "Please confirm that the data format is SEED or MINISEED. After re-uploading, the calculation will be recalculated"; 2. Missing dependency library (obspy not installed): Automatically execute pip install obspy to install the dependency and then retry; 3. Missing response file: Prompt the user to "The station instrument response file is missing, and the instrument response processing cannot be completed. The original data will be used for calculation (the result may have deviations)".

[0119] Scoring criteria: Total 100 points, including 20 points for "Correct Tool Selection" (Boolean check item: whether CodeInterpreter is called); 20 points for "Reasonable Parameter Settings" (numerical check item: whether the filter frequency band is within the range of 0.5-10Hz / P-wave and 1-5Hz / S-wave); 30 points for "Completeness of Calculation Steps" (Boolean check item: whether it includes the five steps of data loading, instrument response removal, filtering, amplitude extraction, and ratio calculation); and 30 points for "Accuracy of Results" (numerical check item: whether the deviation of the calculation result from the standard value 2.0 is ≤5%).

[0120] Generation rules: Reference answers should include detailed steps and explanations of multiple approaches (correcting the rationality of physical quantities); the expected code logic, parameter settings, and expected output of tool calls should be clearly stated; the fault tolerance instructions should cover common abnormal scenarios of tool calls; the scoring points should include both Boolean and numerical checks, taking into account both process and result evaluation.

[0121] In some implementations, regarding step S103, when generating test cases using a test case generation strategy corresponding to the difficulty level of the evaluation question, please refer to [link to relevant documentation]. Figure 2 The following is a flowchart illustrating a test case generation process provided in an exemplary embodiment of this application: Figure 2 As shown, the process includes the following steps S1031~S1032: S1031: If the difficulty level of the evaluation question is the third level, generate a standard reference answer corresponding to the evaluation question, and decompose the standard reference answer into multiple sub-logical goals according to the preset decomposition dimensions.

[0122] Here, the standard reference answer is also known as the expert reference answer.

[0123] The pre-defined decomposition dimensions include knowledge source acquisition, data extraction, data calculation and verification, and conclusion integration. Each dimension corresponds to a necessary logical stage when solving complex problems.

[0124] Sub-logical objectives refer to independent sub-tasks that are separated from the standard reference answer according to preset decomposition dimensions. Each sub-logical objective corresponds to a specific reasoning stage. For example, "obtaining the geological structural background" is a sub-logical objective of T1 knowledge source acquisition, and "calculating vegetation cover" is a sub-logical objective of T3 calculation and verification.

[0125] In this step, reverse decomposition refers to first having a complete standard reference answer, and then analyzing in reverse which logical steps construct the answer step by step, thereby extracting an ordered sequence of sub-logical targets.

[0126] S1032: Generate the test cases based on the multiple sub-logical objectives.

[0127] In this way, multiple sub-logical objectives are used to represent the various logical steps in solving the evaluation problem. Therefore, test cases can be generated based on multiple sub-logical objectives.

[0128] Specifically, for each sub-logical objective, interactive simulation can be performed to generate question-and-answer dialogues. According to the decomposition sequence between each sub-logical objective, multiple question-and-answer dialogues can be associated to obtain the multi-turn dialogue test cases.

[0129] In this implementation, for each sub-logical objective, one or more rounds of user-agent interaction are simulated (user asks questions and the agent invokes tools and provides intermediate responses). These interactions are then chained together according to their interaction sequence to form a complete multi-round dialogue test case. This test case includes the tool invocation trajectory, intermediate outputs, and the final reference answer.

[0130] The following example illustrates the generation of L3-level multi-round test cases.

[0131] Case Study: "By combining Landsat 9 remote sensing imagery of the Tarim Basin (path 142 / row 32, 2023) with a 1:250,000 geological map, we inverted the vegetation cover of the region and analyzed its correlation with geological structures to generate a research report."

[0132] The specific process is as follows: Step A: Generate a standard reference answer (or expert-level reference answer).

[0133] The structured prompts guide the LLM to generate a compliant reference report, with core content including: "1. Regional Overview: The Tarim Basin is located on the southern slope of the Tianshan Mountains. The geological structure is dominated by thrust faults, and the strata are mainly Paleozoic sandstone and Cenozoic sedimentary rocks; 2. Vegetation Cover Inversion: Based on the NDVI index calculation from Landsat 9 imagery, a pixel-based inversion method was used. The results show that the vegetation cover ranges from 0.1 to 0.6, with high-coverage areas (0.4-0.6) concentrated on alluvial fans on both sides of the faults; 3. Correlation Analysis: The distribution of high vegetation cover areas and alluvial fans highly overlaps, and since alluvial fans are controlled by faults, this indicates that geological structures indirectly control vegetation growth by influencing topography and groundwater distribution; 4. Conclusion: There is a significant positive correlation between vegetation cover and geological structure (faults) in this region. The alluvial fans formed by fault activity provide favorable topographic and hydrological conditions for vegetation growth." Step B: Reverse thinking chain decomposition.

[0134] The LLM program guides the process of breaking down the reference report into four sub-objectives: T1 (knowledge source acquisition), T2 (data detail extraction), T3 (calculation and verification), and T4 (conclusion integration). Each sub-objective must clearly define its core content, supporting statements in the reference report, and recommended tools, as detailed below: T1 (Knowledge Source Acquisition): The core content is "to retrieve the geological structure background (fault type, stratigraphic lithology) and climate characteristics (precipitation, temperature) of the Tarim Basin region, and to clarify the basic environmental conditions for vegetation growth"; the supporting statement in the reference report is "The Tarim Basin region is located on the southern slope of the Tianshan Mountains, with the geological structure mainly consisting of thrust faults, and the strata mainly consisting of Paleozoic sandstone and Cenozoic sedimentary rocks"; recommended tools: Academic Search (to retrieve regional geological papers), Document QA (to interpret 1:250,000 geological map instructions).

[0135] T2 (Data Detail Extraction): The core content is "reading Landsat 9 remote sensing images and 1:250,000 geological maps, extracting the raw NDVI index data from the images, fault distribution vector data (trend and location) from the geological maps, and the distribution range of alluvial fans"; the supporting statement in the reference report is "calculating the NDVI index based on Landsat 9 images, using the pixel bisection method for inversion, with high-coverage areas concentrated on alluvial fans on both sides of the faults"; recommended tools: Code Interpreter (image NDVI calculation, geological map vector data reading), Geospatial Analysis (geological map coordinate calibration and spatial cropping).

[0136] T3 (Calculation and Verification): The core content is "Calculating vegetation cover using the pixel dichotomy method based on NDVI data, verifying the spatial correlation between high vegetation cover areas and alluvial fans and faults through spatial overlay analysis, and calculating the correlation coefficient (r)"; the supporting statement in the reference report is "Vegetation cover is between 0.1 and 0.6, and the distribution of high cover areas and alluvial fans highly overlaps"; recommended tools: Code Interpreter (vegetation cover calculation, correlation coefficient analysis), Data Visualization (generating spatial distribution maps of vegetation cover and correlation scatter plots).

[0137] T4 (Conclusion Integration): The core content is "to deduce the correlation mechanism between vegetation cover and geological structure by integrating the geological structural background, vegetation cover inversion results, and correlation analysis data, and to generate a research report that includes data processing, analysis process, conclusions and recommendations"; the supporting statement in the reference report is "Geological structure indirectly controls vegetation growth by influencing topography and groundwater distribution, and there is a significant positive correlation between vegetation cover and geological structure (faults)"; recommended tools: Report Generator (integrate and generate research reports), Document QA (verify the consistency between the report and the geological background).

[0138] Step C: Multi-round interactive simulation.

[0139] Based on the sub-target sequence, user query instructions (User Prompt) and agent responses (AgentResponse) are generated alternately, as shown in the following example: Round 1 (corresponding to T1 sub-target): User query command: "I need to analyze the correlation between vegetation cover and geological structure in the Tarim Basin (path 142 / line 32). First, please obtain the geological structure background (fault type, stratigraphic lithology) and climate characteristics of the region to provide a basis for subsequent analysis."

[0140] Agent Response: "To answer this question, I will invoke the Academic Search and Document QA tools. Academic Search search keywords: 'Tarim Basin, southern foothills of Tianshan Mountains, geological structure', year range 2010-2024, target database CNKI; Document QA parses the 1:250,000 geological map specification (path: . / geology_maps / Tarim_Basin_142_32_specification.pdf), search keywords 'fault type, stratigraphic lithology'. Extract core information: This area is located at the southern foothills of Tianshan Mountains, formed thrust faults (striking NE-SW) under plate compression, the strata are mainly Paleozoic quartz sandstone (accounting for 65%) and Cenozoic conglomerate (accounting for 35%); the climate is temperate continental climate, with annual precipitation of 100-200mm, and vegetation growth mainly depends on groundwater. The next step is suggested to provide Landsat 9 remote sensing images and geological map files to extract data details".

[0141] Round 2 (corresponding to T2 sub-goal): User Query Instruction: "Here are the Landsat 9 remote sensing image of this area (attachment: . / images / Landsat9_142_32_2023.tif) and the 1:250,000 geological map (attachment: . / geology_maps / Tarim_Basin_142_32.shp), please extract the NDVI index, fault distribution data and alluvial-proluvial fan range." Agent Response: "I will invoke the Code Interpreter and Geospatial Analysis tools. Code Interpreter reads the remote sensing image, calculates the NDVI index (calculation formula: NDVI=(NIR-R) / (NIR+R)), and obtains pixel-level NDVI data (range -0.2-0.8); reads the geological map SHP file, extracts thrust fault vector data (3 in total, striking NE30°, NE50°, NE70°) and the distribution range of alluvial-proluvial fan (area about 500 ). Geospatial Analysis performs coordinate calibration on the geological map (WGS84 coordinate system), spatially matches it with the remote sensing image, and crops to obtain a data set with a unified range of the study area. Core code snippet for tool invocation: … from osgeo import gdal; import numpy as np; import geopandas as gpd; # Image NDVI calculation ndvi=(nir-r) / (nir+r); # Geological map clipping gdf.clip(...) … The next step will be to calculate vegetation coverage and analyze the correlation based on this data.

[0142] Round 3 (corresponding to T3 sub-target): User query command: "Calculate vegetation cover based on the extracted data, analyze its spatial correlation with faults and alluvial fans, and generate a visualization chart."

[0143] The agent responds: "I will invoke the Code Interpreter and Data Visualization tool. The Code Interpreter performs the following steps: 1. Calculate vegetation cover using the pixel-based binary method, formula:" ,in (bare land) (Dense vegetation); 2. The calculated vegetation cover ranged from 0.1 to 0.6, with an average of 0.32; 3. Spatial overlay analysis showed that the average vegetation cover in the alluvial fan region was 0.45, and the average in the non-alluvial fan region was 0.21, with a correlation coefficient r = 0.78 (P < 0.05), indicating a significant positive correlation. Data Visualization generated the following charts: 1. Spatial distribution map of vegetation cover (gradation coloring: 0.1-0.2 is light green, 0.2-0.4 is medium green, and 0.4-0.6 is dark green); 2. Scatter plot of vegetation cover versus distance from alluvial fan (the slope of the fitted line is negative); 3. Overlay map of fault distribution and high vegetation cover areas. The next step will be to integrate all the information to generate a research report. Round 4 (corresponding to T4 sub-target): User query command: "Please synthesize all analysis results, deduce the correlation mechanism between vegetation cover and geological structure, and generate a complete research report." The agent responded: "I will call the Report Generator and Document QA tools. The core sections of the research report are: 1. Introduction (Research background and objectives); 2. Overview of the study area (Geological structure, climate characteristics); 3. Data and methods (Image / geological map data, NDVI calculation, pixel binarization, spatial overlay analysis); 4. Results analysis (Vegetation cover distribution and its correlation with geological structures); 5. Discussion (Correlation mechanism: Fault activity forms alluvial fans, providing flat terrain and groundwater recharge, promoting vegetation growth); 6. Conclusions and recommendations. The Document QA verification report is consistent with the geological background: the fault type and stratigraphic lithology description in the report are consistent with the geological map description, and the correlation analysis method conforms to geoscientific research standards. The final report includes three visualization charts and a complete data table, which can be directly used for academic reference." Step D: Connecting Dialogue Logic. Connect the user query commands and agent responses from each round in chronological order to ensure coherent dialogue logic, clear tool call trajectories, and progressive sub-goals, forming complete multi-round dialogue test cases.

[0144] In this implementation, by adopting a "first have the standard answer, then break down the sub-goals in reverse" approach, the system can automatically generate logically rigorous and clearly defined multi-round interaction trajectories without the need for manually writing complex dialogue scripts. This significantly reduces the cost of constructing high-quality, complex evaluation data while ensuring the authenticity and verifiability of the tool call trajectory and reasoning chain. The output of this step (the sequence of sub-logical goals) will be directly used for subsequent multi-round interaction simulations and scoring item construction.

[0145] In this embodiment, the test cases generated based on tool information naturally include the set of tools to be used and their calling order at that difficulty level. This allows subsequent scoring points to be automatically anchored to key tool actions, achieving process-oriented, detailed evaluation, rather than simply comparing the final answer.

[0146] In this embodiment of the application, in order to examine the adaptability of the large language model in a dynamic scientific research environment, prompt information is injected during the generation of L3 level test cases.

[0147] Here, the injection rules for the prompt message include the following three aspects: Injection probability: Each L3 test case has a 30% probability of triggering a dynamic environment challenge, of which the "data update" scenario accounts for 20% and the "hypothesis conflict" scenario accounts for 10%.

[0148] Injection timing: Inject after T3 (calculation and verification) and before T4 (conclusion integration) in a multi-round dialogue to ensure that the model has completed basic data processing and calculation and can adjust the analysis logic based on the new scenario.

[0149] Scenario authenticity: The injected scenario must conform to the actual situation of geoscience research, data updates must provide reasonable new data sources, and hypothetical conflicts must be based on real academic disputes or alternative explanations.

[0150] Based on the above injection rules, when injecting prompt information during the generation of L3 level test cases, the specific steps include (I) to (III): (I) For the evaluation question, obtain the prompt information corresponding to the evaluation question and the answer requirement information corresponding to the prompt information.

[0151] The notification information may include data update notification information or hypothesis conflict notification information.

[0152] For example, data update prompts indicate that the agent's existing data or parameters are outdated and require discarding old data, acquiring new data, and recalculating. For instance, "The latest groundwater data shows a depth decrease of 0.5m, and the atmospheric correction parameter AOD has been updated to 0.15." Hypothesis conflict prompts indicate that the agent has alternative hypotheses or academic viewpoints that contradict the current conclusions, requiring verification of the conflict and adjustment of the conclusions. For example, "Some studies suggest that vegetation cover is primarily controlled by precipitation rather than geological structure; please verify this."

[0153] The response requirements corresponding to the prompt information refer to the normative or standard procedures that match the prompt information and describe how the agent should respond correctly. For example, for data updates, the requirements include "discarding old data, acquiring new data, recalculating, and revising conclusions." This information serves as the standard for subsequent scoring.

[0154] (II) Use the prompt information as the prompt assessment question and the answer requirement information as the reference answer to the prompt assessment question to generate a prompt question and answer dialogue.

[0155] In this step, the prompt assessment question is used as a new round of questions from the user in the interactive simulation. It is not an original assessment question in the assessment question set, but rather a user instruction in a dynamic scenario. Similarly, the prompt question-and-answer dialogue refers to one or more rounds of interaction records generated in response to the prompt assessment question, containing the user's "prompt question" and the agent's "expected answer" (i.e., the response executed according to the answer requirements). It is a new dialogue segment that is additionally inserted into the original dialogue sequence.

[0156] (III) According to the decomposition sequence between each sub-logical objective, associate multiple question-and-answer dialogues, and generate the test cases based on the associated multiple question-and-answer dialogues and the prompt question-and-answer dialogues.

[0157] First, as mentioned above, the multiple question-and-answer dialogues generated according to the sub-logic goal sequence (e.g., the first round of dialogue corresponding to T1, the second round of dialogue corresponding to T2, and the third round of dialogue corresponding to T3) are associated in the order of T1→T2→T3.

[0158] In this step, the prompt-and-answer dialogue generated in step (II) is inserted into the sequence after the T3 dialogue and before the T4 dialogue. Finally, the original T4 dialogue (conclusion integration) is appended to the end.

[0159] Through this interconnected process, a complete multi-turn dialogue history is formed, encompassing "normal reasoning flow + intermediate dynamic challenges + final conclusion adjustment." This history serves as the final Level 3 test case, which can be used to evaluate the model's adaptability and corrective capabilities in the face of unexpected changes.

[0160] For example, the dialogue consists of T1 (knowledge acquisition), T2 (data extraction), T3 (computation verification), prompt-and-answer dialogue (data update or hypothesis conflict), and T4 (conclusion integration, requiring correction of the original conclusion).

[0161] The following section provides a detailed explanation of scene injection, building upon the above case study on vegetation cover analysis in the Tarim Basin. The specific process is as follows: (i) Example of a "data update" scenario (based on a case study of the Tarim Basin): Injection prompt (user query command): "Note that the latest 2024 groundwater observation data for the Tarim Basin shows that the groundwater depth of the alluvial fan in the region has decreased by 0.5m compared to 2023; at the same time, the atmospheric correction parameters for the Landsat 9 image have been updated (Aerosol optical thickness AOD=0.15, the original parameter was 0.10). Please recalculate the vegetation cover based on the new data and analyze the correlation."

[0162] The agent response requirements include a complete process: "Old data obsolescence logic (marking the original groundwater data and NDVI calculation results as outdated), new data acquisition and parsing (calling Document QA to extract the 2024 groundwater observation report data), recalculation (calling Code Interpreter to reload the original image, applying the new AOD parameters to execute the atmospheric correction algorithm, and recalculating NDVI and vegetation cover), and conclusion correction (if the average vegetation cover drops to 0.28, the correlation coefficient r=0.75, which is still a significant positive correlation, it should be explained that the decrease in groundwater depth has led to a slight decrease in vegetation cover, but the control of geological structure on hydrological conditions still dominates)".

[0163] (ii) Example of a "hypothetical conflict" scenario (based on a case study of the Tarim Basin): Injection prompt (user query command): "Some studies have proposed an alternative hypothesis, suggesting that the distribution of vegetation cover in this region is mainly controlled by precipitation distribution rather than geological structure, based on the fact that in 2023, the high precipitation area and the high vegetation cover area in this region partially overlapped. Please verify the rationality of this alternative hypothesis, analyze the conflict points between the two hypotheses and the solutions."

[0164] The agent response requirements include: a "conflict hypothesis verification" process (calling Geospatial Analysis to read the 2023 precipitation raster data for the region, or calling Academic Search to retrieve relevant literature and extracting data points using Code Interpreter), an alternative analysis process (calling Code Interpreter to perform spatial correlation analysis, calculating the correlation coefficient r between precipitation and vegetation cover (r=0.42), which is found to be much lower than the r=0.78 for geological structure, and the contribution of precipitation to vegetation in arid areas is usually lagging), and a conclusion correction basis (existing quantitative evidence supports the geological structure-dominant hypothesis, with precipitation as a secondary influencing factor; the conflict lies in the limitations of single-factor attribution, and the solution is to "use Geodetector or multivariate regression analysis," while incorporating variables such as geological structure, precipitation, and topography to quantify the contribution of each factor in a complete process).

[0165] In this embodiment of the application, a "dynamic environment challenge simulation" is added to the test cases, so that the final generated test cases not only include normal and ideal tool call chains, but also include unexpected situations common in real scientific research (data updates, hypothesis conflicts), thereby enabling a comprehensive evaluation of the robustness, dynamic reasoning ability, and error correction ability of large language models and intelligent agents.

[0166] In some implementations, for step S104, after generating the test case, the scoring items of the test case can be synthesized. Specifically, the evaluation reasoning steps can be extracted from the test case, and the scoring items of the evaluation reasoning steps can be constructed.

[0167] For example, following the above example, the evaluation reasoning steps may include "calling Academic Search to retrieve the regional geological background", "extracting three thrust fault vector data", "calculating vegetation cover using the pixel dichotomy method (NDVI_min=0.1, NDVI_max=0.7)", "calculating the correlation coefficient", "correctly handling data update scenarios (calling Code Interpreter to apply the new AOD parameter 0.15 to recalculate NDVI)", and "deriving conclusions based on two types of core evidence: geological structure and correlation analysis".

[0168] For each evaluation reasoning step, a score can be assigned based on the importance of the task completion for each evaluation reasoning step. The total score is 100 points, of which the basic tool selection and parameter setting check items account for 20%-30%, the process completeness and accuracy check items account for 40%-50%, and the conclusion derivation and result presentation check items account for 20%-30%.

[0169] Then, the above evaluation and reasoning steps are converted into scoring items in a preset format. The preset format can be Boolean or numerical, as shown in Table 3: Table 3

[0170] Optionally, in step S105, when generating the evaluation dataset, each test case can be verified. If the verification passes, the evaluation dataset is generated.

[0171] Specifically, for each test case, the logic of the question-and-answer dialogue in the test case can be verified, the tool information of the target tool called can be verified, and each scoring item can be verified to obtain the verification result.

[0172] In this step, logical verification refers to checking the causal consistency of the question-and-answer dialogue in the test case. For multi-round question-and-answer dialogues, it is also necessary to check their contextual coherence, causal consistency, and the rationality of the time sequence. For example, ensure that the data referenced in the later round of dialogue has indeed been generated in the previous round, and ensure that the tool call order does not contradict the workflow dependency.

[0173] Validating the tool information of the target tool invocation refers to verifying whether the list of tools predicted or invoked in the test case matches the problem requirements, whether the tool parameters are within a valid range, and whether the invocation order conforms to the preset workflow logic. For example, checking whether a non-existent tool is used, or whether the tool output type is compatible with the input of the next tool.

[0174] Validating each scoring item involves checking whether it is clearly defined, quantifiable (e.g., Boolean or numerical), and has clear judgment criteria, without any ambiguous or unexecutable checkpoints. This ensures that the scoring items can be accurately judged by the automated evaluation script.

[0175] The verification result refers to the comprehensive conclusion obtained after performing the above three verifications on each test case. It usually includes "pass" or "fail" (with specific reasons). Pass means that the test case meets the predefined quality standards in terms of logic, tool information and scoring items.

[0176] Furthermore, a target number of test cases whose verification results meet preset verification conditions is determined. If the ratio of the target number to the total number of test cases is greater than a preset ratio threshold, the evaluation dataset is generated based on the evaluation data corresponding to each test case.

[0177] Here, preset verification conditions refer to the specific criteria set in advance to determine whether a test case is qualified. For example: no logical contradictions, the tool type exists and its parameters are valid, and all scoring items are executable checkpoints. These conditions can be configured according to actual needs.

[0178] The target number refers to the number of test cases that pass the verification in all test cases.

[0179] The preset ratio threshold is a constant value set according to actual needs. In this embodiment, the preset ratio threshold is 0.9. In other embodiments, the preset ratio threshold can be other values, such as 0.85, 0.8, etc., which are not limited here.

[0180] In this step, if the pass rate (the ratio of the target number to the total number of test cases) is greater than or equal to the preset percentage threshold, the evaluation data corresponding to all the test cases that have passed the verification (each test case includes fields such as questions, reference answers, reasoning steps, scoring items, and tool information) will be encapsulated in a unified format (such as JSONL) to form the final evaluation dataset, which will be used for the subsequent formal evaluation of the Earth Science Large Language Model.

[0181] Conversely, if the pass rate is less than the preset threshold, test cases need to be regenerated or unqualified test cases need to be manually corrected until the requirements are met.

[0182] In this embodiment of the application, multi-dimensional automated verification is used to eliminate unqualified test cases, thereby ensuring the logical correctness, tool rationality, and scoring operability of the final dataset.

[0183] In this application, when generating the evaluation dataset, each test case can be encapsulated into a standardized JSONL format to improve the universality and ease of use of the evaluation dataset. Specifically, this includes the following steps: 1. JSONL format field definition: A single evaluation data point contains seven core fields, and the specific definition and example of each field are as follows: Question: Original question text, for example: "Combining Landsat 9 remote sensing imagery of the Tarim Basin (path 142 / line 32, 2023) with a 1:250,000 geological map, invert the vegetation cover of the region and analyze its correlation with geological structures to generate a research report."

[0184] level: difficulty label, "L3".

[0185] history: multi-turn dialogue list, each element is a dictionary containing user_input (user query instruction), agent_tool_call (agent tool invocation code / instruction), and agent_response (agent response content, with an example as follows: …json { "user_input": "I need to analyze the correlation between vegetation coverage and geological structure in the area of Path 142 / Row 32 in the Tarim Basin. First, please obtain the geological structure background (fault type, stratigraphic lithology) and climatic characteristics of this area to provide a basis for subsequent analysis.", "agent_tool_call": "Academic Search(query='Tarim Basin South Tianshan Piedmont Geological Structure', year_range='2010-2024', database='CNKI'); Document QA(doc_path='. / geology_maps / Tarim Basin_142_32 Specification.pdf', query='fault type, stratigraphic lithology')", "agent_response": "To solve this problem, I will invoke the Academic Search and Document QA tools. The Academic Search uses the retrieval keywords: 'Tarim Basin South Tianshan Piedmont Geological Structure', the year range is 2010-2024, and the target database is CNKI; the Document QA parses the 1:250,000 geological map specification (path: . / geology_maps / Tarim Basin_142_32 Specification.pdf) and uses the retrieval keywords 'fault type, stratigraphic lithology'. The extracted core information is as follows: This area is located at the South Tianshan Piedmont, where thrust faults (striking NE-SW) are formed by plate compression. The strata are mainly Paleozoic quartz sandstone (accounting for 65%) and Cenozoic conglomerate (accounting for 35%); the climate is temperate continental climate with annual precipitation of 100-200mm, and the vegetation growth mainly depends on groundwater. It is recommended to provide Landsat 9 remote sensing images and geological map files in the next step to extract detailed data.", }, … … ground_truth: expert-level reference report, which includes core contents such as "regional overview, vegetation coverage inversion, correlation analysis, conclusion".

[0186] ​​scoring_points: A list of scoring points, where each element is a dictionary containing check_item (description of the check item), weight (weight), type (check item type: boolean / numerical), and standard (judgment criteria). An example is shown below: …json [ { "check_item": "Whether to call Code Interpreter to recalculate based on the new AOD parameters", "weight": 15, "type": "Boolean", "standard": "Yes = 15 points, No = 0 points (No points will be awarded if only textual corrections are made without recalculation)" }, ... ] … tools_required: A list of required tools, where each element is a dictionary containing name (tool name), category (tool category), and parameters (key parameters). An example is shown below: …json [ { "name": "Code Interpreter", "category": "Code execution and calculation class", "parameters": { "libraries": ["gdal", "numpy", "geopandas"], "task": "NDVI calculation, Statistics" } }, ... ] data_modality: Data modality type, "Remote sensing image + geological map (multimodal)".

[0187] 2. Hierarchical storage of datasets: The difficulty levels are stored in three JSONL files: difficulty_L1_qa.jsonl (L1 test cases), difficulty_L2_qa.jsonl (L2 test cases), and difficulty_L3_qa.jsonl (L3 test cases).

[0188] Generate a dataset statistics report, including key indicators such as the number of use cases at each difficulty level, the distribution of sub-disciplines, the modal distribution, the average number of rounds (L3 only), and the validation pass rate.

[0189] Please see Figure 3 This is a schematic diagram illustrating the number of test cases and the distribution of sub-disciplines at the first level, as provided in an embodiment of this application. Figure 3 As shown, for Level L1, test cases in space physics account for 4% of the evaluation dataset, test cases in geochemistry account for 8%, test cases in solid geophysics account for 8%, test cases in hydrology account for 10%, test cases in atmospheric science account for 11%, test cases in oceanography account for 14%, test cases in geology account for 17%, test cases in geography account for 21%, and test cases in other sub-disciplines account for 7%.

[0190] Please see Figure 4 This is a schematic diagram illustrating the number of test cases and the distribution of sub-disciplines at a second level, as provided in an embodiment of this application. Figure 4 As shown, for Level L2, test cases in space physics account for 2% of the evaluation dataset, test cases in geochemistry account for 7%, test cases in solid geophysics account for 9%, test cases in hydrology account for 10%, test cases in atmospheric science account for 11%, test cases in oceanography account for 15%, test cases in geology account for 17%, test cases in geography account for 27%, and test cases in other sub-disciplines account for 2%.

[0191] Please see Figure 5 This is a schematic diagram illustrating the number of test cases and the distribution of sub-disciplines at a third level, as provided in an embodiment of this application. Figure 5 As shown, for Level L3, test cases in space physics account for 4% of the evaluation dataset, test cases in geochemistry account for 6%, test cases in solid geophysics account for 10%, test cases in hydrology account for 8%, test cases in atmospheric science account for 16%, test cases in oceanography account for 18%, test cases in geology account for 22%, test cases in geography account for 10%, and test cases in other sub-disciplines account for 6%.

[0192] 3. Auxiliary document generation: Generate a README document containing a dataset overview, field descriptions, usage instructions, tool dependencies, and precautions. For example, the tool dependencies section specifies that "the Code Interpreter tool requires the gdal library (version 3.6.0+) and the geopandas library (version 0.12.0+) to process raster and vector data, and the Image QA / Interpretation tool requires the Visual Large Model API (such as GPT-4V) to be configured." The precautions section states that "Boolean checks for scoring points must be strictly judged as 'yes / no,' and numerical checks must be scored according to the deviation threshold."

[0193] Generate sample code demonstrating how to load a JSONL dataset, parse field content, and perform automated evaluation of the tested geoscience large language model based on score points.

[0194] This implementation method achieves automated transformation from unstructured Earth science problems to high-quality, full-process evaluation datasets through the above process.

[0195] Corresponding to the embodiments of the aforementioned evaluation dataset generation method, this application also provides embodiments of an evaluation dataset generation apparatus.

[0196] Please see Figure 6 This is a schematic diagram illustrating the structure of an apparatus for generating an evaluation dataset, as shown in an exemplary embodiment of this application. Figure 6 As shown, the apparatus 600 for generating the evaluation dataset includes: The evaluation question acquisition module 610 is used to acquire a set of evaluation questions; the set of evaluation questions includes multiple evaluation questions about earth science knowledge; The difficulty level determination module 620 is used to predict the tool information of the target tool required to solve the evaluation problem using a large language model for each evaluation problem, and to determine the difficulty level of the evaluation problem based on the tool information of the target tool. The test case generation module 630 is used to generate test cases according to the difficulty level of the evaluation question, using an evaluation case generation strategy corresponding to the difficulty level; the test cases include the evaluation question, the evaluation reference answer to the evaluation question, and at least one evaluation reasoning step performed to generate the evaluation reference answer; The evaluation data generation module 640 is used to construct a scoring item for each evaluation reasoning step of each test case, and generate evaluation data corresponding to the test case based on each evaluation reasoning step and its corresponding scoring item; the scoring item is used to verify the consistency between the actual reasoning steps of the Earth Science Big Language Model and the evaluation reasoning steps. The evaluation dataset generation module 650 is used to generate an evaluation dataset based on the evaluation data corresponding to each test case. The evaluation dataset is used to evaluate the Earth Science Large Language Model.

[0197] In some implementations, the difficulty level determination module 620 is specifically used for: For each assessment question, determine the sub-discipline type to which the assessment question belongs; Based on the sub-discipline type of the evaluation problem and the corresponding text semantics, determine the tool information of the target tools that need to be called to solve the evaluation problem; the tool information includes type, quantity and calling order.

[0198] In some implementations, the difficulty level determination module 620 is specifically used for: Determine the logical dependencies between the target tools; the logical dependencies refer to workflow relationships where the output of one tool is used as the input of another tool. Based on the tool information of the target tool and the logical dependencies, the difficulty level of the evaluation problem is determined.

[0199] In some implementations, the difficulty level determination module 620 is specifically used for: If the number of target tools is not greater than a first quantity threshold, and the number of target tool pairs with logical dependencies is less than the first threshold, the difficulty level of the evaluation problem is determined to be the first level. If the number of target tools is between the first and second quantity thresholds, and the number of target tool pairs with logical dependencies is less than the first threshold, the difficulty level of the evaluation problem is determined to be the second level. If the number of target tools is not less than the second quantity threshold, and the number of target tool pairs with at least two different types of target tools and logical dependencies is not less than the first threshold, the difficulty level of the evaluation problem is determined to be the third level; the difficulty of the first level, the second level and the third level increases sequentially.

[0200] In some implementations, the test cases include single-turn question-and-answer test cases; the test case generation module 630 is specifically used for: If the difficulty level of the evaluation question is the first level or the second level, the single-round question-and-answer test cases are generated based on the evaluation question, the corresponding reference answer, each scoring item and its corresponding score, the tool information of the target tool, and the preset test case generation rules.

[0201] In some implementations, the test case generation module 630 is specifically used for: If the difficulty level of the evaluation question is the third level, a standard reference answer corresponding to the evaluation question is generated, and the standard reference answer is decomposed into multiple sub-logical objectives according to a preset decomposition dimension; the preset decomposition dimension includes knowledge source acquisition, data extraction, data calculation and verification, and conclusion integration; The test cases are generated based on the multiple sub-logical objectives.

[0202] In some implementations, the test cases include multi-turn dialogue test cases; the test case generation module 630 is specifically used for: For each sub-logical objective, an interactive simulation is performed on the sub-logical objective to generate a question-and-answer dialogue; Based on the decomposition sequence between each sub-logical objective, multiple question-and-answer dialogues are associated to obtain the multi-turn dialogue test cases.

[0203] In some implementations, the test case generation module 630 is further configured to: For the evaluation question, obtain the prompt information corresponding to the evaluation question and the answer requirement information corresponding to the prompt information; The prompt information is used as the prompt assessment question, and the answer requirement information is used as the reference answer to the prompt assessment question to generate a prompt question and answer dialogue; When the test case generation module 630 associates multiple question-and-answer dialogues according to the decomposition sequence between each sub-logical objective to obtain the test cases, it is specifically used for: According to the decomposition sequence between each sub-logical objective, multiple question-and-answer dialogues are associated, and the test cases are generated based on the associated multiple question-and-answer dialogues and the prompt question-and-answer dialogues.

[0204] In some implementations, the evaluation dataset generation module 650 is specifically used for: For each test case, the question-and-answer dialogue in the test case is logically validated, the tool information of the target tool called is validated, and each scoring item is validated to obtain the validation results. Determine the target number of test cases whose verification results meet the preset verification conditions. If the ratio of the target number to the total number of test cases is greater than the preset ratio threshold, generate the evaluation dataset based on the evaluation data corresponding to each test case.

[0205] The specific implementation process of the functions and roles of each unit in the above device can be found in the implementation process of the corresponding steps in the above method, and will not be repeated here.

[0206] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this application according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0207] Corresponding to the above-described method for generating the evaluation dataset, this disclosure also provides a computer device, such as... Figure 7 The diagram shown is a structural schematic of a computer device provided in an embodiment of this disclosure. Figure 7 As shown, the computer device 700 includes a processor 710, an internal bus 720, memory 730, a network interface 740, and non-volatile memory 750, and may also include other hardware required for its functions. One or more embodiments of this specification can be implemented in software, for example, the processor 710 reads the corresponding computer program from the non-volatile memory 750 into the memory 730 and then runs it. Of course, besides software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution entity of the following processing flow is not limited to individual logic units, but can also be hardware or logic devices.

[0208] The memory 730, also known as internal memory, is used to temporarily store the computational data in the processor 710, as well as the data exchanged with non-volatile memory 750 such as hard disk. The processor 710 exchanges data with the non-volatile memory 750 through the memory 730.

[0209] In this embodiment, memory 730 is specifically used to store application code that executes the solution of this application, and its execution is controlled by processor 710. That is, when the computer device is running, processor 710 communicates with network interface 740, memory 730 and non-volatile memory 750 through internal bus 720, so that processor 710 executes the application code stored in memory 730 and non-volatile memory 750, thereby executing the evaluation dataset generation method described in the above method embodiment.

[0210] Processor 710 may be an integrated circuit chip with signal processing capabilities. The aforementioned processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware microservices. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this invention. The general-purpose processor can be a microprocessor or any conventional processor.

[0211] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the computer device 700. In other embodiments of this application, the computer device 700 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0212] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the evaluation dataset generation method described in the above method embodiments. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0213] This disclosure also provides a computer program product carrying program code. The program code includes instructions that can be used to execute the steps of the evaluation dataset generation method in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0214] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0215] The embodiments of the subject matter and functional operation described in this specification can be implemented in the following ways: digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. Alternatively or additionally, the program instructions may be encoded on artificially generated propagation signals, such as machine-generated electrical, optical, or electromagnetic signals, which are generated to encode information and transmit it to a suitable receiving device for execution by the data processing apparatus. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0216] The processing and logic flow described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform corresponding functions by operating on input data and generating output. The processing and logic flow can also be executed by dedicated logic circuitry—such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the device can also be implemented as dedicated logic circuitry.

[0217] Computers suitable for executing computer programs include, for example, general-purpose and / or special-purpose microprocessors, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory and / or random access memory. Basic computer microservices include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more mass storage devices for storing data, such as disks, magneto-optical disks, or optical disks, or the computer will be operatively coupled to such mass storage devices to receive data from or transfer data to them, or both. However, a computer is not required to have such devices. Furthermore, a computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive, to name a few.

[0218] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented by or incorporated into dedicated logic circuitry.

[0219] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of any invention or the scope of the claims, but rather are primarily intended to describe features of specific embodiments of a particular invention. Certain features described in the various embodiments herein may also be implemented in combination in a single embodiment. Conversely, various features described in a single embodiment may also be implemented separately in various embodiments or in any suitable sub-combination. Furthermore, while features may function in certain combinations as described above and even initially claimed in this way, one or more features from a claimed combination may be removed from that combination in some cases, and a claimed combination may refer to a sub-combination or a variation thereof.

[0220] Similarly, although the operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring these operations to be performed in the specific order shown or sequentially, or requiring all illustrated operations to be performed to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and microservices in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program microservices and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0221] Thus, specific embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims may be performed in a different order and still achieve the desired result. Furthermore, the processes depicted in the drawings are not necessarily shown in a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing may be advantageous.

[0222] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for generating an evaluation dataset, characterized in that, include: Obtain a set of assessment questions; the set of assessment questions includes multiple assessment questions about Earth science knowledge; For each evaluation question, a large language model is used to predict the tool information of the target tool required to solve the evaluation question, and the difficulty level of the evaluation question is determined based on the tool information of the target tool. Based on the difficulty level of the evaluation question, an evaluation case generation strategy corresponding to the difficulty level is adopted to generate test cases; the test cases include the evaluation question, the evaluation reference answer to the evaluation question, and at least one evaluation reasoning step performed to generate the evaluation reference answer; For each evaluation reasoning step of each test case, a scoring item is constructed for the evaluation reasoning step, and evaluation data corresponding to the test case is generated based on each evaluation reasoning step and its corresponding scoring item; the scoring item is used to verify the consistency between the actual reasoning steps of the Earth Science Big Language Model and the evaluation reasoning steps. An evaluation dataset is generated based on the evaluation data corresponding to each test case. This evaluation dataset is used to evaluate the Earth Science Large Language Model. If the evaluation question is at level three, the test cases include multi-turn dialogue test cases; The step of generating test cases based on the difficulty level of the evaluation question, using an evaluation case generation strategy corresponding to the difficulty level, includes: A standard reference answer corresponding to the evaluation question is generated, and the standard reference answer is decomposed into multiple sub-logical objectives according to a preset decomposition dimension. The preset decomposition dimension includes a knowledge source acquisition dimension, a data extraction dimension, a data calculation and verification dimension, and a conclusion integration dimension. Each sub-logical objective corresponds to an independent reasoning stage. For each sub-logical objective, interactive simulation is performed to generate a question-and-answer dialogue including user query commands and agent responses. According to the decomposition sequence between each sub-logical objective, multiple question-and-answer dialogues are associated to obtain the multi-turn dialogue test cases. During the association of multiple question-and-answer dialogues, prompting question-and-answer dialogues are inserted between the question-and-answer dialogues corresponding to the data calculation and verification dimension and the question-and-answer dialogues corresponding to the conclusion integration dimension, according to preset injection probabilities and injection timing. The prompting question-and-answer dialogues include prompting information and corresponding answering requirements. The prompting information includes data update prompts indicating that the agent's original data or parameters are outdated and that old data needs to be discarded, new data needs to be acquired, and recalculated, or hypothesis conflict prompts indicating that the agent has alternative hypotheses or academic viewpoints that contradict the current conclusions and that the conflict needs to be verified and the conclusions adjusted. The answering requirements refer to the specifications that match the prompting information and describe how the agent should respond correctly.

2. The method according to claim 1, characterized in that, The tool information for the target tool required to solve the evaluation problem using large language models includes: For each assessment question, determine the sub-discipline type to which the assessment question belongs; Based on the sub-discipline type of the evaluation problem and the corresponding text semantics, determine the tool information of the target tools that need to be called to solve the evaluation problem; the tool information includes type, quantity and calling order.

3. The method according to claim 2, characterized in that, The determination of the difficulty level of the evaluation question based on the tool information of the target tool includes: Determine the logical dependencies between the target tools; the logical dependencies refer to workflow relationships where the output of one tool is used as the input of another tool. Based on the tool information of the target tool and the logical dependencies, the difficulty level of the evaluation problem is determined.

4. The method according to claim 3, characterized in that, Determining the difficulty level of the evaluation problem based on the tool information of the target tool and the logical dependencies includes: If the number of target tools is not greater than a first quantity threshold, and the number of target tool pairs with logical dependencies is less than the first threshold, the difficulty level of the evaluation problem is determined to be the first level. If the number of target tools is between the first and second quantity thresholds, and the number of target tool pairs with logical dependencies is less than the first threshold, the difficulty level of the evaluation problem is determined to be the second level. If the number of target tools is not less than the second quantity threshold, and the number of target tool pairs with at least two different types of target tools and logical dependencies is not less than the first threshold, the difficulty level of the evaluation problem is determined to be the third level; the difficulty of the first level, the second level and the third level increases sequentially.

5. The method according to claim 4, characterized in that, The test cases include single-round question-and-answer test cases; the step of generating test cases according to the difficulty level of the evaluation questions, using an evaluation case generation strategy corresponding to the difficulty level, includes: If the difficulty level of the evaluation question is the first level or the second level, the single-round question-and-answer test cases are generated based on the evaluation question, the corresponding reference answer, each scoring item and its corresponding score, the tool information of the target tool, and the preset test case generation rules.

6. The method according to claim 1, characterized in that, The evaluation dataset is generated based on the evaluation data corresponding to each test case, including: For each test case, the question-and-answer dialogue in the test case is logically validated, the tool information of the target tool called is validated, and each scoring item is validated to obtain the validation results. Determine the target number of test cases whose verification results meet the preset verification conditions. If the ratio of the target number to the total number of test cases is greater than the preset ratio threshold, generate the evaluation dataset based on the evaluation data corresponding to each test case.

7. An apparatus for generating an evaluation dataset, characterized in that, include: The evaluation question acquisition module is used to acquire a set of evaluation questions; the set of evaluation questions includes multiple evaluation questions about earth science knowledge; The difficulty level determination module is used to predict the tool information of the target tool required to solve each evaluation problem using a large language model, and to determine the difficulty level of the evaluation problem based on the tool information of the target tool. The test case generation module is used to generate test cases based on the difficulty level of the evaluation question, using an evaluation case generation strategy corresponding to the difficulty level; the test cases include the evaluation question, the evaluation reference answer to the evaluation question, and at least one evaluation reasoning step performed to generate the evaluation reference answer; The evaluation data generation module is used to construct a scoring item for each evaluation reasoning step of each test case, and generate evaluation data corresponding to the test case based on each evaluation reasoning step and its corresponding scoring item; the scoring item is used to verify the consistency between the actual reasoning steps of the Earth Science Big Language Model and the evaluation reasoning steps. The evaluation dataset generation module is used to generate an evaluation dataset based on the evaluation data corresponding to each test case. The evaluation dataset is used to evaluate the Earth Science Big Language Model. If the evaluation question is at level three, the test cases include multi-turn dialogue test cases; the test case generation module is specifically used for: A standard reference answer corresponding to the evaluation question is generated, and the standard reference answer is decomposed into multiple sub-logical goals according to a preset decomposition dimension. The preset decomposition dimension includes knowledge source acquisition dimension, data extraction dimension, data calculation and verification dimension, and conclusion integration dimension. Each sub-logical goal corresponds to an independent reasoning stage. For each sub-logical goal, interactive simulation is performed to generate a question-and-answer dialogue including user query commands and agent responses. Based on the decomposition sequence between each sub-logical objective, multiple question-and-answer dialogues are associated to obtain the multi-turn dialogue test cases. During the association of multiple question-and-answer dialogues, prompting question-and-answer dialogues are inserted between the question-and-answer dialogues corresponding to the data calculation and verification dimension and the question-and-answer dialogues corresponding to the conclusion integration dimension, according to preset injection probabilities and injection timing. The prompting question-and-answer dialogues include prompting information and corresponding answering requirements. The prompting information includes data update prompts indicating that the agent's original data or parameters are outdated and that old data needs to be discarded, new data needs to be acquired, and recalculated, or hypothesis conflict prompts indicating that the agent has alternative hypotheses or academic viewpoints that contradict the current conclusions and that the conflict needs to be verified and the conclusions adjusted. The answering requirements refer to the specifications that match the prompting information and describe how the agent should respond correctly.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the method for generating the evaluation dataset according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method for generating the evaluation dataset according to any one of claims 1-6.

Images