Geoscience big data oriented agent evaluation method, device and product

Abstract

This application discloses a method, device, and product for evaluating intelligent agents based on geoscience big data, relating to the geoscience field. The method includes constructing a first evaluation test case based on configuration information from multiple geoscience data sources; constructing a second evaluation test case based on a geoscience knowledge graph and predefined multiple meta-path templates; constructing a third evaluation test case based on geoscience business specifications and process documents; and evaluating the target intelligent agent based on the first, second, and third evaluation test cases. This application can cover the "retrieval-analysis-decision" closed-loop process commonly found in geoscience work, thereby effectively reflecting the end-to-end capabilities of intelligent agents in real business processes.

Description

Technical Field

[0001] This application relates to the field of geosciences, and in particular to a method, device and product for evaluating intelligent agents for geoscience big data. Background Technology

[0002] With the rapid evolution of large language model technology, it has demonstrated significant advantages in tasks such as text understanding, information extraction, and reasoning generation. Furthermore, with the maturation of agentic capabilities such as tool invocation, task planning, and multi-step execution, the geosciences field has begun to explore its application in complex scenarios such as disaster monitoring and emergency response, remote sensing interpretation, geological exploration and evaluation, climate and environmental analysis, and research assistance. However, unlike general natural language tasks, typical geoscience tasks often inherently possess characteristics of "multi-source heterogeneous evidence input, strong spatiotemporal constraints, long reasoning chains, and strict process specifications": on the one hand, evidence retrieval, alignment, and fusion across data sources are required; on the other hand, multi-step reasoning along the geosciences knowledge chain is necessary, along with procedural decisions made under the constraints of business norms. Simultaneously, geosciences big data generally carries clearly defined meta-information such as time range, spatial range, coordinate reference system, and unit system. Relying solely on the model's linguistic priors or "pure text-based responses" makes it difficult to guarantee the verifiability and reproducibility of conclusions, thus limiting the reliable implementation of large models in geosciences scenarios.

[0003] To evaluate and constrain the behavior of agents in the aforementioned real-world geoscientific task chains, current methods typically employ question-and-answer or small-scale question bank-based evaluation frameworks. However, when considering the new paradigm of geoscientific agents—characterized by "executable, tool-callable, and long-chain reasoning trajectories"—systematic shortcomings remain. First, current evaluation tasks are mostly single-round question-and-answer or knowledge-based tests, failing to cover the "retrieval-analysis-decision" closed-loop process prevalent in geoscientific work, thus failing to reflect the end-to-end capabilities of agents in real-world business chains. Second, many evaluations focus only on the final textual answer, lacking process recording and quantification of key stages such as retrieval, spatial analysis, and map querying under a unified tool interface. This leads to a disconnect between evaluation results and actual usage scenarios, making it difficult to determine whether errors stem from tool selection, parameter configuration, or improper evidence use. Third, test case construction often heavily relies on experts manually creating and annotating questions, which is not only costly and slow to update but also difficult to quickly cover different regions, different data products, and constantly changing business process specifications. Meanwhile, if the evaluation questions highly overlap with public datasets or online corpora, the model may obtain inflated scores through memory or data leakage, weakening the credibility of the evaluation. Finally, current evaluations typically only output scores or simple conclusions, lacking explainable attributions for failures (e.g., incorrect query parameters, incorrect coordinate systems / units, missing key intermediate inference nodes, incorrect sequence of process steps, etc.), making it difficult to provide actionable diagnostic criteria for model iteration and engineering repair. Summary of the Invention

[0004] The purpose of this application is to provide an intelligent agent evaluation method for geoscience big data, which can cover the "retrieval-analysis-decision" closed-loop process that is common in geoscience work, thereby effectively reflecting the end-to-end capabilities of intelligent agents in real business links.

[0005] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a method for evaluating intelligent agents based on geoscientific big data, including: The configuration information of multiple geoscientific data sources is obtained, and the output of each geoscientific data source is standardized into a preset field structure. Specific queries are generated according to the configuration information of each geoscientific data source. Geoscientific retrieval tools are called to execute each specific query to obtain return results from the corresponding geoscientific data source and extract evidence from the return results. Each piece of evidence and question type constraint are input into a large model generator, and the first natural language question and first standard answer output by the large model generator are obtained. The corresponding first natural language question, first standard answer, evidence, and specific query are bound to form a first evaluation case. Obtain a geoscience knowledge graph and multiple predefined meta-path templates. Sample the geoscience knowledge graph according to each meta-path template to obtain the corresponding knowledge graph reasoning path. Input each knowledge graph reasoning path into a large model generator and obtain the second natural language question, the second standard answer, and the equivalent SPARQL query output by the large model generator. Bind the corresponding second natural language question, the second standard answer, and the SPARQL query to form a second evaluation case. Obtain geoscience business specifications and process documents, merge the geoscience business specifications and process documents into a unified decision tree, adopt a root-to-leaf geoscience reasoning path from the unified decision tree and generate corresponding case descriptions, input each geoscience reasoning path and corresponding case description into a large model generator and obtain the third natural language question and third standard answer output by the large model generator, and bind the corresponding third natural language question, the third standard answer and the geoscience reasoning path to form a third evaluation case; The target agent is evaluated based on the first evaluation case, the second evaluation case, and the third evaluation case.

[0006] Secondly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described intelligent agent evaluation method for geoscience big data.

[0007] Thirdly, this application provides a computer program product, including a computer program that, when executed by a processor, implements the aforementioned intelligent agent evaluation method for geoscience big data.

[0008] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a method, device, and product for evaluating intelligent agents for geoscientific big data. First, it generates first, second, and third evaluation cases with different dimensions. Then, it evaluates the target intelligent agent based on the first, second, and third evaluation cases. Therefore, the evaluation dimensions are comprehensive and consistent with real geoscientific tasks. Through the three-dimensional combination of "retrieval-knowledge graph multi-hop reasoning-business process path reasoning", it can respectively measure the intelligent agent's evidence retrieval ability, structured knowledge reasoning ability, and standardized process decision-making ability, covering the core capability stack in geoscientific big data applications.

[0009] Meanwhile, in the process of generating test cases, a large model generator is used to output natural language questions and standard answers, which has the advantages of low generation cost and scalability: the large model automatically generates questions and answers and standard answers under evidence constraints, and uses structured queries / paths as evidence, which significantly reduces the cost of manual annotation; at the same time, tens of thousands of test cases can be generated through parallel data source retrieval and path sampling. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a flowchart illustrating an intelligent agent evaluation method for geoscientific big data in one embodiment of this application. Figure 2 This is a schematic diagram illustrating the generation logic of an evaluation test case in one embodiment of this application. Detailed Implementation

[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0013] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0014] The intelligent agent evaluation method for geoscience big data provided in this application embodiment can be applied to computer equipment, which can be a terminal device or a server.

[0015] like Figure 1 As shown, the intelligent agent evaluation method for geoscience big data provided in this application embodiment specifically includes steps S110 to S140.

[0016] Step S110: Obtain configuration information of multiple geoscientific data sources, standardize the output of each geoscientific data source into a preset field structure, generate corresponding specific queries according to the configuration information of each geoscientific data source, call the geoscientific retrieval tool to execute each specific query, obtain the return results from the corresponding geoscientific data source, extract evidence from the return results, input each piece of evidence and question type constraints into the large model generator, obtain the first natural language question and the first standard answer output by the large model generator, and bind the corresponding first natural language question, first standard answer, evidence and specific query to form a first evaluation case.

[0017] The first evaluation case is a retrieval-type evaluation case. Configuration information includes the data source identifier, access method, authentication information, queryable fields and their types, evidence extraction rules, and spatiotemporal metadata field mapping. Evidence is extracted from the returned results according to the evidence extraction rules. The preset field structure includes content, spatiotemporal range, coordinate reference system, unit, source, and traceable link. The generation steps for each specific query include: generating a query template and candidate parameter space based on the access method, queryable fields and their types, and spatiotemporal metadata field mapping of the corresponding geoscientific data source. The query template includes spatiotemporal constraint parameters, topic parameters, and filtering parameters; randomly sampling spatiotemporal constraint parameters, topic parameters, and filtering parameters from the candidate parameter space to instantiate the query template and form a specific query. Question type constraints include: questions must rely on evidence returned by the tool, contain a clear spatiotemporal range or dataset identifier, and the answer must be uniquely determined from the evidence.

[0018] Step S110 is the construction step for retrieval-type evaluation test cases. The following example illustrates step S110 in detail. In this example, step S110 includes steps S111 and S112.

[0019] In step S111, the system receives the configuration of the geoscientific data sources to be accessed and constructs a data source registry. Each data source includes at least a data source identifier, access method, authentication information, queryable fields and their types, evidence extraction rules for the returned results, and spatiotemporal metadata field mapping. The system standardizes the output of different data sources to form a unified evidence object, whose fields include at least content, time, spatial range, coordinate reference system, unit, source, and traceable link, thereby eliminating the differences between multi-source heterogeneous interfaces and providing a consistent data carrier format for subsequent "evidence constraint generation" and "recalculation verification".

[0020] In step S112, the system automatically generates retrieval-based test cases for each data source. First, the system generates a query template based on the data source fields (including access methods, queryable fields and their types, and spatiotemporal metadata field mappings). The query template must at least include spatiotemporal constraint parameters, topic parameters, and filtering parameters. Then, it samples a parameter vector (including elements such as spatiotemporal constraint parameters, topic parameters, and filtering parameters) from the candidate parameter space and instantiates it to form a specific query. Subsequently, it calls a geoscientific retrieval tool to execute the specific query and obtain the returned results, extracting evidence from the returned results according to evidence extraction rules. After obtaining the evidence, the system inputs the evidence and question type constraints into a large model generator to generate standard answers to natural language questions, and binds the specific query and evidence as verifiable evidence to the retrieval-based test cases. Question type constraints must at least include: the question must rely on evidence returned by the tool, contain a clear spatiotemporal range or dataset identifier, and the answer must be uniquely determined from the evidence. This achieves question-answer generation driven by verifiable evidence, reducing manual annotation costs and ensuring verifiability.

[0021] Step S120: Obtain the geoscience knowledge graph and multiple predefined meta-path templates. Sample the geoscience knowledge graph according to each meta-path template to obtain the corresponding knowledge graph reasoning path. Input each knowledge graph reasoning path into the large model generator and obtain the second natural language question, the second standard answer, and the equivalent SPARQL query output by the large model generator. Bind the corresponding second natural language question, the second standard answer, and the SPARQL query to form a second evaluation case.

[0022] The second evaluation case is a knowledge graph multi-hop reasoning evaluation case. Preferably, the knowledge graph reasoning path is subject to deduplication constraints, acyclic constraints, and minimum support constraints.

[0023] In one example, the system constructs or accesses a geoscientific knowledge graph, which contains a set of entity nodes and a set of relation edges (which may include attributes). The system predefines a set of meta-path templates, each used to describe a sequence of entity types and a sequence of relations, such as "geological unit-origin-mineral-indicator-metallogenic environment" or "watershed-containment-hydrological station-observation-runoff process-impact-flood risk". For example... Figure 2 The system uses the "Geological Anomaly - Mineral Resources - Specific Metallogenic Environment - Characteristic Elements - Exploration Quality Elements - Exploration Project" framework. For each meta-path template, the system performs path sampling: randomly selecting nodes that meet the starting point type. The knowledge graph reasoning path is obtained by sequentially expanding according to relational constraints. Furthermore, it imposes constraints such as deduplication, acyclicity, and minimum support on the knowledge graph reasoning path. For knowledge graph reasoning paths, For relation edges, and All nodes are entity nodes. The system then inputs the text descriptions of the knowledge graph reasoning path and its nodes / relationships into the large model generator, automatically generating corresponding natural language questions and standard answers, and simultaneously generating equivalent SPARQL queries as evidence extraction and scoring criteria. This explicitly transforms the multi-hop reasoning chain into computable evidence, supporting interpretable evaluation of the completeness of the reasoning chain and the coverage of intermediate links.

[0024] Step S130: Obtain the geoscience business specifications and process documents, merge the geoscience business specifications and process documents into a unified decision tree, adopt the root-to-leaf geoscience reasoning path from the unified decision tree and generate the corresponding case description, input each geoscience reasoning path and the corresponding case description into the large model generator and obtain the third natural language question and the third standard answer output by the large model generator, and bind the corresponding third natural language question, the third standard answer and the geoscience reasoning path to form a third evaluation case.

[0025] The third evaluation test case is the business process / specification path reasoning evaluation test case. It merges geoscientific business specifications and process documents into a unified decision tree, specifically including: parsing process documents into conditional directed graphs using a process parser, and parsing geoscientific business specifications into hierarchical decision trees; merging the directed graphs and hierarchical decision trees into a unified decision tree through node semantic alignment and condition normalization.

[0026] In one example, the system receives geoscientific business specifications or process documents, including flowcharts and plain text specifications. The system uses a process parser to parse the flowchart into a conditional directed graph and the text specifications into a hierarchical decision tree; the two are then merged into a unified decision tree through node semantic alignment and condition normalization. The system samples the root-to-leaf geoscientific reasoning paths from the unified decision tree and automatically generates case descriptions with geoscientific contexts (including observation data summaries, regional background, thresholds / gradings, etc.) based on the condition variables in the geoscientific reasoning paths. Then, a large model generator generates natural language questions (e.g., "What should be the next action / conclusion in a given scenario?") and standard answers (leaf node actions and key evidence). The corresponding geoscientific reasoning paths are written as interpretable evidence into the business process / specification path reasoning evaluation test cases to cover the "decision-making under specification constraints" capabilities commonly found in geoscientific tasks, avoiding evaluation that only stays at the knowledge-based question-and-answer level.

[0027] Step S140 involves evaluating the target agent based on the first evaluation case, the second evaluation case, and the third evaluation case. This specifically includes steps S141 and S142.

[0028] Step S141: Perform anti-pollution detection, consistency verification, and metadata verification on the first, second, and third test cases.

[0029] The pollution prevention detection includes: dividing the natural language questions in the first, second, and third evaluation cases into original prefixes and original suffixes respectively; for each natural language question, calling the large model generator that generates natural language questions to continue writing the original prefix of the natural language question to obtain the comparison suffix; calculating the similarity between the comparison suffix and the original suffix of the natural language question; if the similarity is greater than the threshold, the evaluation case containing the natural language question is deleted.

[0030] Consistency checks include recalculating the evidence by performing specific queries and SPARQL queries to verify that the corresponding standard answer can be uniquely derived from the evidence.

[0031] Metadata validation includes verifying whether the units, spatiotemporal range, and coordinate reference value of the data in the standard answer meet the preset range.

[0032] Specifically, to reduce the risk of distorted evaluation due to pre-training data contamination or question memorization, the system performs anti-contamination detection and quality verification. For each generated natural language question, the system segments it by length to obtain the original prefix and suffix. It then calls the same large-scale generator to generate a continuation under deterministic settings, obtaining a comparison suffix. The system calculates the similarity index between the comparison suffix and the original suffix (e.g., ROUGE-L and edit distance similarity). When the similarity exceeds a threshold, the system determines that the natural language question may have a memorization / leakage risk and removes it. Simultaneously, the system performs consistency verification: automatically executing specific queries and SPARQL queries to recalculate evidence, verifying that the standard answer can be uniquely derived from the evidence; and performing unit / coordinate verification: checking the consistency of numerical units, latitude and longitude ranges, CRS (Coordinate Reference System), and the legality of the time range.

[0033] In one embodiment, the system can randomly sample a portion of natural language questions and insert a certain proportion of intentionally erroneous natural language questions as a control. Multiple geoscientists independently label the correctness and security of these questions, and statistically analyze the consistency coefficient (e.g., ...). This allows for the filtering out of unqualified natural language problems, thereby improving the reliability, authenticity, and verifiability of the evaluation set.

[0034] Step S142: Evaluate the target agent based on the first, second, and third evaluation cases that have passed the anti-contamination detection, consistency verification, and metadata verification. This step specifically includes: injecting a unified toolset into the target agent and defining a strict JSON schema for each tool. The toolset includes GeoSearchTool for structured retrieval of authoritative geoscientific APIs, GeoSpatialTool for spatial analysis capabilities such as coordinate transformation and spatial statistics, GeoKGTool for performing SPARQL / graph traversal queries on GeoKG, DocRetrieveTool for paragraph-level retrieval of standard documents and guidelines, and MathStatsTool for unit conversion and statistical calculations; inputting the first, second, and third evaluation cases that have passed the anti-contamination detection, consistency verification, and metadata verification into the target agent one by one; recording the final answer output by the target agent, the tool call sequence, and the parameters and returns for each call. The process involves summarizing and synchronously recording token consumption and latency; performing structured parsing on the final answer and tool call sequence; matching the structured parsed final answer with the standard answer in terms of key entities, numerical values, spatiotemporal range, and conclusion actions, and scoring the matching results according to a pre-set scoring scale; comparing the structured parsed tool call sequence with the expected evidence to evaluate the necessity of tool calls, the correctness of parameters, and the rationality of the returned results; for the second and third test cases, evaluating whether the structured parsed tool call sequence covers key intermediate reasoning nodes or decision steps, and finally obtaining the evaluation results; and performing error pattern clustering on test cases with scores below the scoring threshold and abnormal evaluation results to form interpretable failure type labels.

[0035] In one example, step S142 includes the following steps: (1) The system injects a unified set of tools into the tested agent and defines a strict JSON schema for each tool. The tools include at least GeoSearchTool for structured retrieval of authoritative geoscience APIs, GeoSpatialTool for spatial analysis capabilities such as coordinate transformation and spatial statistics, GeoKGTool for performing SPARQL / graph traversal queries on GeoKG, DocRetrieveTool for paragraph-level retrieval of standard documents and guidelines, and MathStatsTool for unit conversion and statistical calculation. During the evaluation, the system sends the evaluation test cases to the agent one by one. The agent can decide whether, when, and how to call the tools. The system records the final answer output by the agent, the tool call sequence, the parameters of each call, and the return summary, and records the trajectory data such as token consumption and latency at the same time. This expands the evaluation from a plain text dialogue to an executable capability evaluation and provides an objective trajectory basis for the evaluation of the correctness of tool use. Figure 2 This is a schematic diagram illustrating the logic for generating test cases.

[0036] (2) The system performs structured parsing of the tool call sequence, extracting at least the tool name, call order, parameter key value, key return field and spatiotemporal object; at the same time, it parses the numerical value, unit, place name, coordinate and time expression in the agent's answer and normalizes it into a comparable structured representation to support subsequent evidence-based fact matching, spatiotemporal error calculation and error attribution.

[0037] (3) The system performs automatic scoring and multi-indicator statistics for each use case. The system matches the final answer with the standard answer in terms of key entities, numerical values, spatiotemporal range, and conclusion actions. Based on a pre-set scoring scale, the rule engine and / or model judges provide a score of 0–5 and calculate the pass rate (Score≥4). At the same time, the system compares the tool call sequence with the expected evidence to evaluate whether the tool call is necessary, whether the parameters are correct, and whether the return is used reasonably. It also judges whether the knowledge graph multi-hop reasoning evaluation use cases and business process / normative path reasoning evaluation use cases cover key intermediate nodes / decision steps to measure the completeness of the reasoning chain. Finally, the system outputs statistical results such as average score, pass rate, error type distribution, and token / latency-quality relationship by dimension, thereby forming a reproducible and comparable quantitative evaluation conclusion.

[0038] Table 1 Preset rating scale The scoring considers not only factual completeness but also spatiotemporal accuracy, unit / coordinate correctness, and the safety of recommendations for disaster scenarios. The scale thresholds and detailed rules can be expanded and configured according to task type and risk level, and are written into the Item along with the scoring rules to ensure cross-version consistency.

[0039] (4) The system performs error pattern clustering based on low-scoring use cases (e.g., Score≤2) and abnormal tool trajectory use cases to form interpretable failure type labels, such as query parameter errors, coordinate system / unit errors, missing key intermediate inference nodes, incorrect process step order, and answer drift. The system automatically generates improvement suggestions and new blinding / adversarial use cases for each type of error, and performs anti-pollution and quality verification of the new use cases in step S141 before incorporating them into the next version evaluation set, thus forming a closed-loop evaluation-analysis-expansion mechanism, and supporting versioned storage and traceable auditing of evaluation data, evidence, and scoring results.

[0040] In summary, the intelligent agent evaluation method for geoscience big data in this application has the following technical effects: 1. Comprehensive evaluation dimensions and consistent with real-world geoscience tasks: Through a three-dimensional combination of "retrieval - knowledge graph multi-hop reasoning - business process path reasoning", it can respectively measure the agent's evidence retrieval ability, structured knowledge reasoning ability, and standardized process decision-making ability, covering the core capability stack in geoscience big data applications.

[0041] 2. Low generation cost and scalable: It automatically generates question-and-answer and standard answers under evidence constraints using large models, and uses structured queries / paths as evidence, which significantly reduces the cost of manual annotation; at the same time, it can generate tens of thousands of test cases through parallel data source retrieval and path sampling.

[0042] 3. Reliable evaluation with anti-pollution and traceability: By using prefix continuation similarity detection and a repeatable evidence recalculation mechanism, the artificially high scores caused by pre-training memory and data leakage are reduced; each use case carries verifiable evidence and generation trajectory, which is convenient for auditing.

[0043] 4. Quantifiable tool usage ability: The unified function call tool schema enables different agents to be compared in the same tool environment. Combined with trajectory analysis, indicators such as "whether the tool is called correctly, whether the parameters are reasonable, and whether the return evidence is used" can be directly quantified, thereby expanding the evaluation from pure text dialogue to the evaluation of executable capabilities.

[0044] 5. Automatic report parsing and closed-loop incremental generation: Through error pattern clustering and targeted use case generation, it can continuously fill the evaluation blind spots, form an evaluation-parsing-expansion closed loop, and support long-term iteration and model capability tracking.

[0045] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0046] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0047] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0048] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0049] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0050] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0051] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0052] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for evaluating intelligent agents based on geoscience big data, characterized in that, include: The configuration information of multiple geoscientific data sources is obtained, and the output of each geoscientific data source is standardized into a preset field structure. Specific queries are generated according to the configuration information of each geoscientific data source. Geoscientific retrieval tools are called to execute each specific query to obtain return results from the corresponding geoscientific data source and extract evidence from the return results. Each piece of evidence and question type constraint are input into a large model generator, and the first natural language question and first standard answer output by the large model generator are obtained. The corresponding first natural language question, first standard answer, evidence, and specific query are bound to form a first evaluation case. Obtain a geoscience knowledge graph and multiple predefined meta-path templates. Sample the geoscience knowledge graph according to each meta-path template to obtain the corresponding knowledge graph reasoning path. Input each knowledge graph reasoning path into a large model generator and obtain the second natural language question, the second standard answer, and the equivalent SPARQL query output by the large model generator. Bind the corresponding second natural language question, the second standard answer, and the SPARQL query to form a second evaluation case. Obtain geoscience business specifications and process documents, merge the geoscience business specifications and process documents into a unified decision tree, adopt a root-to-leaf geoscience reasoning path from the unified decision tree and generate corresponding case descriptions, input each geoscience reasoning path and corresponding case description into a large model generator and obtain the third natural language question and third standard answer output by the large model generator, and bind the corresponding third natural language question, the third standard answer and the geoscience reasoning path to form a third evaluation case; The target agent is evaluated based on the first evaluation case, the second evaluation case, and the third evaluation case; The first evaluation case, the second evaluation case, and the third evaluation case, which have passed the pollution prevention detection, consistency verification, and metadata verification, are input into the target intelligent agent one by one; Record the final answer output by the target agent, the tool call sequence, and the parameters and return summary for each call, and simultaneously record the token consumption and latency; Perform structured parsing on the final answer and the tool call sequence; The final answer obtained after structured analysis is matched with the standard answer in terms of key entities, numerical values, spatiotemporal range, and conclusion actions, and the matching results are scored according to a preset scoring scale. By comparing the structured parsed tool call sequence with the expected evidence, the necessity of the tool calls, the correctness of the parameters, and the rationality of the return results are evaluated. For the second and third test cases, the structured parsed tool call sequence is evaluated to see if it covers key intermediate inference nodes or decision steps, and the final evaluation result is obtained. For evaluation cases with scores below the scoring threshold or abnormal evaluation results, error pattern clustering is performed to form interpretable failure type labels.

2. The intelligent agent evaluation method for geoscientific big data according to claim 1, characterized in that, The configuration information includes the data source identifier, access method, authentication information, queryable fields and their types, evidence extraction rules, and spatiotemporal metadata field mapping. The evidence is extracted from the returned results according to the evidence extraction rules. The preset field structure includes content, spatiotemporal range, coordinate reference system, unit, source, and traceable link.

3. The intelligent agent evaluation method for geoscientific big data according to claim 2, characterized in that, The steps for generating each specific query include: Based on the access method, queryable fields and their types, and spatiotemporal metadata field mapping of the corresponding geoscientific data source, a query template and a candidate parameter space are generated. The query template includes spatiotemporal constraint parameters, topic parameters, and filtering parameters. The spatiotemporal constraint parameters, topic parameters, and filter parameters are randomly sampled from the candidate parameter space to instantiate the query template and form the specific query.

4. The intelligent agent evaluation method for geoscientific big data according to claim 1, characterized in that, The constraints on the question type include: the question must rely on evidence returned by the tool, contain a clear spatiotemporal range or dataset identifier, and the answer must be uniquely determined from the evidence.

5. The intelligent agent evaluation method for geoscientific big data according to claim 1, characterized in that, The knowledge graph reasoning path is subject to deduplication constraints, acyclic constraints, and minimum support constraints.

6. The intelligent agent evaluation method for geoscientific big data according to claim 1, characterized in that, The aforementioned geoscience business specifications and process documents are merged into a unified decision tree, specifically including: The process parser parses the process document into a conditional directed graph and the geoscience business specification into a hierarchical decision tree. The directed graph and the hierarchical decision tree are merged into the unified decision tree by node semantic alignment and conditional normalization.

7. The intelligent agent evaluation method for geoscientific big data according to claim 1, characterized in that, The target agent is evaluated based on the first evaluation case, the second evaluation case, and the third evaluation case, specifically including: The first evaluation case, the second evaluation case, and the third evaluation case are subjected to anti-pollution detection, consistency verification, and metadata verification. The pollution prevention detection includes: dividing the natural language questions in the first evaluation case, the second evaluation case, and the third evaluation case into original prefixes and original suffixes respectively; for each natural language question, calling the large model generator that generates the natural language question to continue writing the original prefix of the natural language question to obtain a comparison suffix; calculating the similarity between the comparison suffix and the original suffix of the natural language question; if the similarity is greater than a threshold, deleting the evaluation case containing the natural language question. The consistency verification includes: recalculating the evidence by executing the specific query and the SPARQL query to verify that the corresponding standard answer can be uniquely derived from the evidence; The metadata verification includes: verifying whether the units, spatiotemporal range, and coordinate reference of the data in the standard answer meet the preset range; The target agent is evaluated based on the first evaluation case, the second evaluation case, and the third evaluation case, which are obtained through the pollution prevention detection, the consistency verification, and the metadata verification.

8. The intelligent agent evaluation method for geoscientific big data according to claim 7, characterized in that, The target intelligent agent is evaluated based on the first evaluation case, the second evaluation case, and the third evaluation case, which pass the pollution prevention detection, the consistency verification, and the metadata verification. Specifically, this includes: A unified toolset is injected into the target intelligent agent, and a strict JSON schema is defined for each tool. The toolset includes GeoSearchTool for structured retrieval of authoritative geoscience APIs, GeoSpatialTool for spatial analysis capabilities including coordinate transformation and spatial statistics, GeoKGTool for performing SPARQL / graph traversal queries on GeoKG, DocRetrieveTool for paragraph-level retrieval of standard documents and guidelines, and MathStatsTool for unit conversion and statistical calculations.

9. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the intelligent agent evaluation method for geoscience big data as described in any one of claims 1-8.

10. A computer program product, comprising a computer program, characterized in that, When executed by a processor, the computer program implements the intelligent agent evaluation method for geoscience big data as described in any one of claims 1-8.

Images