A multi-modal intelligent agent-oriented multi-round scientific research workflow evaluation system and method

Abstract

The application relates to a multi-round scientific research workflow evaluation system and method for a multi-modal intelligent agent, and belongs to the technical field of large language models and multi-modal large models. The application comprises a multi-round task generation module, a multi-modal information processing module, and the like. The multi-round task generation module constructs a multi-round task scene closer to a real scientific research process. The multi-modal information processing module enables the measured intelligent agent to complete paper active retrieval, multi-modal evidence extraction alignment, cross-paper evidence integration, and round reuse of evidence. The protection points of the application are not limited to a single task sample or a single scoring index, but are focused on a multi-round evaluation benchmark constructed around a real scientific research workflow, a process-level evaluation protocol matched with the benchmark, and an overall implementation scheme supporting landing execution and fine-grained diagnosis of the benchmark.

Description

Technical Field

[0001] This invention relates to a multi-round scientific research workflow evaluation system and method for multimodal intelligent agents, belonging to the technical fields of large language models and multimodal large models. Background Technology

[0002] With the development of large language models and multimodal large models, research-aiding agents are gradually expanding from single-round question answering to more complex task scenarios such as multi-step planning, external tool invocation, and paper reading and analysis. In actual scientific research, researchers typically need to actively search for relevant literature around a research question, read the figures, tables, and text in papers, align experimental settings across papers, and continuously accumulate, reuse, and revise existing evidence in multiple rounds of analysis to ultimately form verifiable and reproducible research conclusions. Although existing agents already have the ability to search web pages, read PDFs, and invoke tools, systematic evaluation of the entire process of "multi-round scientific research workflow" is still insufficient, especially in terms of proactive literature discovery, integration of multi-figure evidence across papers, and evidence memory reuse in subsequent rounds. There is a lack of evaluation methods and benchmark systems that can truly reflect the characteristics of scientific research tasks.

[0003] The most similar technical solutions to this invention mainly include benchmarks for academic paper comprehension, scientific graph question answering, and multimodal multi-turn reasoning. For example, SciVQA, CharXiv, SPIQA, and LiveXiv primarily evaluate questions and answers for graphs or local content within a single paper, assessing the model's ability to understand scientific graphs. However, they are typically limited to single-document scenarios and rarely involve cross-paper evidence integration and multi-turn evidence reuse. SIN-Bench has begun to focus on the evidence chain problem in long-context scientific literature, but its emphasis remains on multimodal understanding within a single long document. PaperArena is closer to this invention, evaluating cross-paper navigation and tool-enhanced reasoning, but its tasks typically do not require active searching from indirect clues and are limited to single-turn question answering questions. It also rarely requires the integration of multiple graphs and the continuous reuse of preceding graph evidence to arrive at the final conclusion. Existing technologies also have general intelligent agent evaluation for transaction processing scenarios. For example, the invention published in CN120144417B is entitled "An Intelligent Agent Evaluation Processing Method, Device, Storage Medium and Electronic Device". Although its technical solution also focuses on process-level evaluation, its process-level indicators mainly revolve around task coverage and logical consistency. It focuses more on evaluating the completion status of the task graph and the execution logic, and lacks an error diagnosis mechanism for cross-paper and multimodal scientific research evidence chains.

[0004] While existing technologies can evaluate individual capabilities of large models or multimodal agents, such as document understanding, graph-based question answering, and tool usage, they still have significant shortcomings when applied to real-world research tasks. First, evaluation tasks are often too static, typically providing target papers, explicit search objects, or direct evidence sources by default, failing to effectively assess the agent's ability to proactively discover and progressively retrieve literature based on indirect clues. Second, most evaluations focus only on single documents or single rounds of question answering, lacking systematic coverage of evidence integration processes across papers, multiple graphs, and multiple rounds, making it difficult to accurately reflect the continuous workflow in research analysis. Third, most existing solutions only focus on the correctness of the final answer, lacking fine-grained evaluation of process behaviors such as the accuracy of evidence extraction, the effective reuse of prior evidence in subsequent rounds, and the redundancy of tool usage, thus making it difficult to accurately define the agent's capability boundaries in key areas such as retrieval, perception, reasoning, and memory reuse. These shortcomings limit the current technologies in terms of evaluation accuracy, diagnostic effectiveness, and application guidance, making it difficult to provide a reliable basis for the optimization and iteration of research-aided agents.

[0005] Therefore, there is an urgent need to propose a multi-round scientific research workflow evaluation system and method for multimodal intelligent agents to solve the above-mentioned technical problems. Summary of the Invention

[0006] To address the aforementioned problems, a multi-round scientific research workflow evaluation system and method for multimodal intelligent agents is provided. A brief overview of the invention is given below to provide a basic understanding of certain aspects of the invention. It should be understood that this overview is not an exhaustive summary of the invention. It is not intended to identify key or essential parts of the invention, nor is it intended to limit the scope of the invention.

[0007] The technical solution of the present invention: A multi-round scientific research workflow evaluation system for multimodal intelligent agents includes: Multi-round task generation module: Constructs multi-round task scenarios that are closer to the real scientific research process; Multimodal information processing module: Enables the tested intelligent agent to perform active paper retrieval, multimodal evidence extraction and alignment, cross-paper evidence integration, and round-reuse of evidence.

[0008] Preferred: The multimodal information processing module includes: Active search module: Enables the tested intelligent agent to actively search for and obtain relevant papers related to the research question; Evidence extraction and alignment module: Extracts and aligns chart evidence from the aforementioned paper to achieve the association of multimodal information; Cross-paper multimodal information integration module: integrates multimodal information from several papers; Multimodal information recording and round reuse module: Reuse the acquired evidence in subsequent rounds to complete the final answer.

[0009] Preferably, the system further includes a process-level evaluation mechanism module: evaluating whether the final result of the agent's task processing is correct, whether the evidence is correct, whether the necessary steps are completed, and locating which capability link has an error.

[0010] A multi-round scientific research workflow evaluation method for multimodal intelligent agents, based on the aforementioned multi-round scientific research workflow evaluation system for multimodal intelligent agents, includes the following steps: S1. Constructing an evaluation benchmark framework: This involves abstracting a set of stages from a real scientific research workflow, including research question formulation, relevant paper search, graph and chart inspection, cross-paper evidence alignment, intermediate evidence archiving, and final conclusion verification.

[0011] S2. Multimodal task execution process: In the first round, images, charts, and problem descriptions are input into the agent to retrieve relevant papers, open the target paper, and extract image, table, or text evidence. The extracted evidence is used to answer the questions in this round. In the intermediate rounds, the agent continues to search for more papers based on the current problem and previous evidence results, supplements new chart evidence, and gradually forms intermediate conclusions across papers; In the final round, the agent directly provides a comprehensive answer based on the image evidence, tabular evidence, and text evidence previously archived in the memory pool.

[0012] Preferably, the method further includes S3, practical testing and verification using multiple representative multimodal intelligent agents: A minimum necessary steps list is set for the evaluation samples to represent the key tool usage actions that must be included in any valid solution path. The completion and accuracy of the process are evaluated by comparing the agent's actual tool calling behavior with the minimum necessary steps list to determine whether its final answer is correct, whether there are redundant calls, and whether the error occurred in the search, perception, reasoning or memory reuse stage.

[0013] Preferred: The test objects include open source models, closed source models, and human expert baselines. Through unified task input, tool environment and evaluation criteria, their overall performance in multiple rounds of scientific research workflow is compared and analyzed.

[0014] Preferred approach: Classify failures into retrieval errors, perception errors, reasoning errors, memory retrieval errors, and a small number of other execution-related errors, thereby pinpointing the bottleneck in model capabilities.

[0015] The present invention has the following beneficial effects: This invention proposes a multi-round research workflow evaluation benchmark for multimodal intelligent agents. This benchmark, with characteristics of a real research process, is fundamentally structured by creating a multi-round, continuous, and interdependent research workflow for research assistance tasks, rather than a single-round static question-and-answer session. The benchmark has the following key features: First, multi-round coherence, where subsequent questions explicitly depend on intermediate conclusions obtained in previous rounds; second, proactive search, where the task description does not directly provide target papers or explicit identifiers, but instead requires the agent to search for relevant papers independently through citation clues, research intentions, experimental phenomena, or indirect hints; third, cross-paper multimodal evidence integration, requiring the agent to jointly extract image, table, and textual evidence from multiple papers and complete alignment, comparison, filtering, or aggregation; and fourth, evidence memory reuse, requiring the agent to save evidence obtained in previous rounds and continue using it in subsequent rounds. This workflow-oriented benchmark construction method is the first core protection point of this invention.

[0016] The present invention proposes a process-level evaluation protocol and overall implementation scheme to complement the evaluation benchmark. Based on the aforementioned evaluation benchmark, the present invention further proposes an executable and diagnosable evaluation protocol and implementation system. This system not only focuses on whether the final answer is correct, but also further evaluates whether the agent completes the answer based on correct evidence, whether it completes the key steps necessary for solving the problem, and whether it correctly reuses previous evidence in subsequent rounds. Specifically, the present invention preferably sets up a memory constraint mechanism to restrict or disable external tool calls in the final round to explicitly test the agent's evidence memory and reuse capabilities. At the same time, a minimum necessary step list is defined for each evaluation example to describe the key tool usage actions that any valid solution path must include, and the actual execution process of the agent is compared with the list to evaluate its process completeness and redundancy. Furthermore, this invention also forms a complete implementation scheme from task instantiation, evidence unit organization, tool calling process, memory pool management to error attribution analysis. It can subdivide errors into retrieval errors, perception errors, reasoning errors, memory reading errors and other few execution-related errors, thereby clarifying the bottleneck of model capabilities. This process-level evaluation protocol and overall implementation system built around evidence correctness, memory reuse, minimum necessary steps and fine-grained error analysis is the second core protection point of this invention.

[0017] The protection points of this invention are not limited to a single task sample or a single scoring index, but rather to the multi-round evaluation benchmarks constructed around real scientific research workflows, the process-level evaluation protocols that accompany them, and the overall implementation scheme that supports the implementation and fine-grained diagnosis of the benchmarks; the multi-round scientific research workflow evaluation benchmark system and method for multimodal intelligent agents together constitute the core technical content of this invention. Attached Figure Description

[0018] Figure 1 It is a set of representative key stage flowcharts abstracted from real scientific research workflow; Figure 2 It refers to the specific methods for instantiating multi-round tasks and the execution process of the tools. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the invention is described below with reference to specific embodiments shown in the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concept of the invention.

[0020] Specific implementation method one: Combining Figure 1-2 This embodiment describes a multi-round scientific research workflow evaluation system for multimodal intelligent agents, comprising: Multi-round task generation module: Constructs multi-round task scenarios that are closer to the real scientific research process; Multimodal information processing module: Enables the tested intelligent agent to perform active paper retrieval, multimodal evidence extraction and alignment, cross-paper evidence integration, and round-reuse of evidence; The purpose of this invention is to provide a method for constructing and evaluating multi-round scientific research workflow evaluation benchmarks for multimodal intelligent agents. By constructing multi-round task scenarios that are closer to the real scientific research process, the tested intelligent agent needs to actively search for relevant papers around the research question, extract and align chart evidence, integrate multimodal information across papers, and reuse the acquired evidence in subsequent rounds to complete the final answer.

[0021] Specific Implementation Method Two: Combining Figure 1-2 This embodiment describes a multi-round scientific research workflow evaluation system for multimodal intelligent agents. The multimodal information processing module includes: Active search module: Enables the tested intelligent agent to actively search for and obtain relevant papers related to the research question; Evidence extraction and alignment module: Extracts and aligns chart evidence from the aforementioned paper to achieve the association of multimodal information (evidence); Cross-paper multimodal information integration module: integrates multimodal information from several papers; Multimodal information recording and round reuse module: Reuse the acquired evidence in subsequent rounds to complete the final answer.

[0022] Specific implementation method three: Combining Figure 1-2This embodiment describes a multi-round scientific research workflow evaluation system for multimodal intelligent agents. The system further includes a process-level evaluation mechanism module: evaluating whether the final result of the agent's task processing is correct, whether the evidence is correct, whether the necessary steps are completed, and locating which capability link has an error; thereby improving the authenticity, interpretability, and diagnostic value of the evaluation, and providing more effective technical support for the training optimization, capability analysis, and system design of multimodal scientific research intelligent agents. This invention further provides a supporting process-level evaluation mechanism, which not only evaluates whether the final result is correct, but also evaluates whether the evidence is correct, whether the necessary steps have been completed, and in which capability aspect the error mainly occurred, thereby improving the authenticity, interpretability, and diagnostic value of the evaluation, and providing more effective technical support for the training optimization, capability analysis, and system design of multimodal scientific research agents.

[0023] Specific implementation method four: Combination Figure 1-2 This embodiment describes a multi-round research workflow evaluation method for multimodal intelligent agents, based on the multi-round research workflow evaluation system for multimodal intelligent agents described in embodiments one through three, and includes the following steps: S1. Based on the multi-round task generation module, construct an evaluation benchmark framework corresponding to the human scientific research process: like Figure 1 As shown, a set of representative key stages are abstracted from the real scientific research workflow as examples. These stages include research question formulation, relevant paper search, chart inspection, cross-paper evidence alignment, intermediate evidence archiving, and final conclusion verification. like Figure 2 As shown, the present invention further provides a specific method for instantiating multi-round tasks and the tool execution process; a complete evaluation sample is organized into an episode, each episode contains several interrelated turns; each turn contains at least a problem description, an input image or related context, a standard answer, and an optional set of related papers; The core features of the framework are: multi-turn coherence, proactive paper search, cross-paper evidence alignment, and evidence memory mechanism; constraints are applied using these core features. This framework comprises at least four core features: First, multi-round coherence, meaning that an evaluation example is not a single, independent question-and-answer session, but rather consists of multiple interdependent rounds of questions, with subsequent questions requiring explicit use of findings from earlier rounds; second, proactive paper search, meaning that the task description does not directly provide precise identifiers for target papers, but only offers research intent, citation clues, experimental phenomena, or indirect hints, requiring the agent to actively understand the question and search for relevant papers independently; third, cross-paper evidence alignment, meaning that the agent needs to simultaneously locate figure, table, or textual evidence from multiple papers and perform alignment analysis on the model, dataset, task settings, and evaluation metrics; and fourth, evidence memory mechanism, meaning that textual, image, and tabular evidence acquired by the agent in early rounds needs to be cached and managed, and then retrieved and reused in subsequent rounds, especially the final round. Through these designs, this invention enables evaluation tasks to more closely resemble real scientific research analysis, moving beyond a single-round static question-and-answer level.

[0024] S2. Multimodal task execution flow based on the multimodal information processing module: In the first round, the agent is input with images, charts, and question descriptions through the active search module to retrieve relevant papers. The target paper is opened, and the evidence extraction and alignment module extracts image, table, or text evidence. The extracted evidence is stored / cached in memory to answer the questions in this round. Specifically, in the first round, the agent usually needs to search for potentially relevant papers using web search tools based on the input image, local chart content, or indirect description in the question. After locating the candidate papers, it opens the document (candidate paper) using a PDF parsing tool and further locates the specific target image or target table using a content extraction tool, extracting image, table, or text evidence to answer the current question. In the intermediate rounds, the agent continues to retrieve more papers based on the current problem and previous evidence results, supplementing new chart evidence. The new evidence is aligned with the evidence in the first round for analysis. Through the cross-paper multimodal information integration module, intermediate conclusions are gradually formed across papers, generating new memory pool evidence. The paper parsing and extraction refer to the tools and methods used in the first round, which will not be elaborated here. In the final round, for the final round question, the present invention preferably prohibits the re-calling of external tools, and requires the agent to complete a comprehensive answer directly based on the image evidence, tabular evidence and text evidence previously archived in the memory pool. The evidence reuse and multimodal memory integration capabilities are explicitly tested based on the multimodal information recording and round reuse module.

[0025] Specific Implementation Method Five: Combining Figure 1-2This embodiment describes a multi-round scientific research workflow evaluation method for multimodal intelligent agents. The method further includes S3, a process-level evaluation mechanism module, for practical testing and verification of various representative multimodal intelligent agents. This invention also sets a minimum necessary steps list for each evaluation example to characterize the key tool usage actions that must be included in any valid solution path. For example, if a problem requires extracting and comparing results from specified charts in two papers, at least the necessary steps of searching for relevant papers, parsing papers, locating target charts, and extracting corresponding content must be completed. By comparing the actual tool call behavior of the agent with the minimum necessary steps list, this invention can not only determine whether its final answer is correct, whether there are redundant calls, and whether its error occurs in the search, perception, reasoning, or memory reuse stages, but also determine whether its process is efficient through the number of tool calls and the time spent on evidence extraction. Thus, this invention not only achieves the evaluation of the success rate of the result, but also conducts a detailed evaluation and analysis of the process completion and accuracy. This invention proposes a method for constructing and evaluating multi-round scientific research workflow evaluation benchmarks for multimodal intelligent agents. This method is used to systematically evaluate the agent's proactive search capabilities, multi-round reasoning capabilities, cross-paper multimodal evidence integration capabilities, and evidence memory reuse capabilities in real-world scientific research assistance scenarios. The technical solution includes both a method for constructing evaluation data and a design for the execution process and a result evaluation mechanism for this type of task. Its core idea is to transform the iterative workflow in real-world scientific research processes into a multi-round evaluation task that can be standardized, automatically executed, and diagnosed with fine granularity.

[0026] Specific Implementation Method Six: Combination Figure 1-2 This embodiment describes a multi-round scientific research workflow evaluation method for multimodal intelligent agents. The invention not only provides a method for constructing and executing an evaluation benchmark, but also conducts practical tests and verifications on various representative multimodal intelligent agents based on this benchmark. The test objects can include open-source models, closed-source models, and human expert baselines. Through unified task input, tool environment, and evaluation standards, their overall performance in multi-round scientific research workflows is compared and analyzed. Test results show that although existing models possess certain capabilities in paper retrieval, document reading, and graph understanding, they still have significant shortcomings in complex tasks that more closely resemble real scientific research processes. In particular, the overall success rate remains low (only 29.23%) in areas such as cross-paper evidence alignment, joint analysis of multiple graphs, evidence reuse in subsequent rounds, and the generation of final comprehensive conclusions, showing a significant gap compared to human experts. This indicates that current multimodal intelligent agents do not yet possess stable and reliable scientific research assistance capabilities, further verifying the necessity and practical value of the evaluation scheme proposed in this invention.

[0027] Specific implementation method seven: Combining Figure 1-2 This embodiment describes a multi-round scientific research workflow evaluation method for multimodal intelligent agents. This invention not only focuses on the correctness of the final answer but also provides fine-grained analysis of the reasons for model failures. Specifically, failures can be categorized into retrieval errors, perception errors, reasoning errors, memory retrieval errors, and a small number of execution-related errors. The proportion of each type of error helps pinpoint the bottleneck in the model's capabilities. Retrieval errors primarily manifest as failure to find the correct paper based on clues; perception errors primarily manifest as failure to correctly read the content of the target chart despite locating it; reasoning errors primarily manifest as drawing incorrect conclusions during comparison, screening, and aggregation of evidence; and memory retrieval errors primarily manifest as failure to correctly retrieve or integrate valid evidence obtained in previous rounds, especially in the final round. Through this process-level error analysis, this invention provides more targeted guidance for subsequent model optimization, tool design, and memory mechanism improvement, going beyond a coarse-grained result evaluation. This invention focuses on key aspects of real-world research processes, including problem formulation, proactive search, graph reading, cross-paper alignment, evidence archiving, and multi-round reuse. It constructs a multi-round research workflow evaluation benchmark and its supporting evaluation method for multimodal intelligent agents. This technical solution not only more realistically reflects the actual capabilities of intelligent agents in research assistance scenarios but also reveals their weaknesses through fine-grained process analysis. It provides reliable technical support for the capability assessment, system optimization, and iterative upgrading of multimodal research intelligent agents, demonstrating significant application value and potential for widespread adoption.

[0028] Specific implementation method eight: Combination Figure 1-2 This embodiment describes a multi-round scientific research workflow evaluation method for multimodal intelligent agents. Regarding the construction of evaluation benchmarks, the specific task organization form of the multi-round scientific research workflow can be adjusted according to application needs. The number of questions in each round can be set to different lengths, and the task types can be expanded to other forms of scientific research analysis besides comparison, screening, constraint satisfaction, and evidence aggregation. Task clues can come from citation information, research descriptions, and experimental phenomena, or from figure / table fragments, keyword combinations, or other indirect hints. Evidence carriers can be further expanded to supplementary materials or code repository information, in addition to the main text of the paper, appendix content, figures, and tables. As long as it still satisfies the technical idea of ​​"actively searching around the research question, accumulating evidence round by round, and completing the final comprehensive judgment," it can be considered an equivalent alternative to this invention. Regarding evaluation protocols and implementation methods, different implementations can be adopted for tool types, memory organization forms, and process analysis mechanisms; functions such as web page retrieval, document parsing, chart extraction, and content location can be completed by different types of tool modules; the memory mechanism can adopt an explicit memory pool approach, or it can adopt structured caching, retrieval-based memory, or other reusable intermediate state management methods; the granularity of error analysis can also be divided into coarser or finer categories according to needs; as long as the alternative can still achieve active search, cross-paper evidence integration, memory reuse, and process-level evaluation of multimodal agents in multi-round scientific research workflows.

[0029] Example 1: Combination Figure 1-2 The multi-round scientific research workflow evaluation system and method for multimodal intelligent agents is shown in this embodiment 1. The organization of evaluation samples, definition of evidence units, tool calling interface, memory management mechanism and process-level scoring method are further explained as follows.

[0030] In this invention, a complete evaluation task is organized into an episode. Each episode contains multiple interconnected turns, which together constitute a short-range scientific research workflow. Each turn includes at least the following information: the text of the current round's question, optional input images or clues, the standard answer for the current round, the set of evidence units required for the current round, and candidate paper information associated with the question. Preferably, the input images or clues include one or more of the following: partial screenshots of charts, citation clues from papers, descriptions of research questions, descriptions of experimental phenomena, bibliographic information such as authors or conferences. Unlike existing single-round static question answering, in this invention, subsequent rounds of questions rely on evidence retrieved, extracted, and archived in previous rounds, giving the evaluation task explicit multi-round coherence and workflow attributes. This organization method is consistent with the definition of an episodeic multi-turn task in the paper, that is, each episode consists of multiple turns, and the agent continuously accumulates and reuses evidence in different turns to complete the final scientific research question answering task.

[0031] In this invention, to make the evaluation process verifiable and automatically scoreable, evidence units are further defined. An evidence unit is preferably represented as: a paper identifier, an evidence type, and a figure / table number. The paper identifier is used to uniquely identify the source paper; the evidence type includes image evidence and tabular evidence; and the figure / table number is used to label specific figures and tables. Preferably, the evidence units are stored in a structured form in a memory pool, allowing subsequent rounds to directly call and verify them.

[0032] In this invention, the execution process of the intelligent agent relies on a unified tool environment. This tool environment includes at least a web search tool, a PDF parsing tool, a document-based text retrieval tool, and a figure / table content extraction tool. The web search tool takes natural language search terms, citation clues, or keyword combinations as input and outputs a list of candidate papers, titles, abstract fragments, and link addresses, used for actively discovering target papers. The PDF parsing tool takes paper titles, URLs, or arXiv identifiers as input and outputs normalized document representations, metadata, and cache handles, used for subsequent content access. The document-based text retrieval tool takes paper identifiers and text queries as input and outputs relevant text fragments and their lightweight source information, used for locating evidence within the main text. The content extraction tool is used to extract target figures or target tables; its input is paper identifiers and figure / table indexes, and its output is image content, structured table content, titles, adjacent text, and source information. The tool environment section of the paper also clearly provides the unified tool interfaces and functional divisions of WebSearch, PDFExtractor, PDFExtractorRAG, and ContentExtractor, demonstrating that the toolchain of this invention has clear input-output relationships and feasibility.

[0033] In this invention, before each round of answering, the agent receives the current question and prior memory state, and then decides whether to invoke tools and which tools to invoke based on the content of the current question. For the first few rounds of questions, the agent preferably performs the following steps: First, it constructs a search query based on the research intent, citation clues, or image clues in the question, and locates candidate papers using a web search tool; Second, after locating a paper, it opens the corresponding document using a PDF parsing tool, and further locates target figures, target tables, or target text fragments using a text retrieval tool or content extraction tool within the document; Third, it stores the extracted image evidence, table evidence, and text evidence into a memory pool, and generates an answer in the current round; Fourth, in subsequent rounds, based on the current question and prior memory, it continues to search for new papers or supplement new evidence, gradually forming intermediate and final conclusions across papers. (From the patent disclosure) Figure 2 The multi-round tool-use workflow is clearly demonstrated: in each round, the agent uses tools to search for papers, open PDFs, locate and extract the content of figures and tables, and then writes the evidence into the memory pool for later reuse.

[0034] In this invention, the memory pool is an episode-level evidence caching structure used to store, in chronological order, the documents, images, tables, text fragments, and intermediate analysis results accessed by the agent in each round. Preferably, each record in the memory pool includes at least the evidence content itself, the source paper identifier, the round of access, the evidence type, and its location information. This allows for tracking in subsequent rounds whether a conclusion originates from previously accessed evidence, rather than from prior model knowledge or guesswork. This design supports postthoc reuse and efficiency calculations. To explicitly evaluate the agent's ability to remember and reuse evidence, this invention preferably imposes tool access constraints in the final round. Specifically, for the final round question that pre-determines the need for reuse of prior evidence, the system disables external tool call interfaces, prohibiting the agent from accessing web search, PDF parsing, document retrieval, and content extraction tools again. The agent is only allowed to read evidence already archived in the memory pool from previous rounds and provide the final answer. If the agent fails to provide a correct answer in the final round using previously acquired evidence, it can be further determined that it has failed in memory reuse or multimodal evidence integration.

[0035] To enable fine-grained diagnostics in the evaluation process of this invention, a minimum necessary step list is defined for each episode. This minimum necessary step list describes the essential tool usage actions indispensable in an effective solution path, preferably including: target paper search, target chart or text location, target content extraction, etc. The agent's actual tool call sequence can be compared with the minimum necessary step list to determine whether it has completed the essential steps necessary for problem solving and whether there are any obvious redundant operations.

[0036] The scoring mechanism of this invention not only focuses on whether the final answer is correct, but also further includes evaluation of evidence correctness, evaluation of process redundancy, and error attribution analysis. Preferably, the correctness of the final result includes at least the episode success rate, the accuracy rate of the final round, and the average accuracy rate of the non-final rounds; evidence correctness is used to measure whether the evidence accessed by the agent covers the standard set of evidence units, assuming a correct answer; process redundancy is used to measure the degree of deviation of the actual number of tools invoked by the agent when successfully completing the task from the minimum necessary steps list. This is used for process-level evaluation and diagnosis of research agents.

[0037] Furthermore, to perform process-level attribution of agent failures, this invention preferably categorizes failure examples into the following types: retrieval errors, perception errors, reasoning errors, memory retrieval errors, and other execution-related errors. Retrieval errors refer to the agent's failure to locate the correct paper based on research clues or citation hints; perception errors refer to the agent's failure to correctly read image values, legends, table fields, or corresponding text information even after accessing and correctly reading the required evidence; reasoning errors refer to the agent's failure to complete cross-paper comparisons, constraint satisfaction, or multi-evidence aggregation despite having accessed and correctly read the required evidence; and memory retrieval errors refer to the agent's failure to correctly call, combine, or align the evidence in subsequent rounds, especially the final round, even after acquiring valid evidence in previous rounds.

[0038] In a specific embodiment, the present invention can be constructed and operated as follows: First, a seed paper set is selected from publicly available paper sources, and the candidate paper pool is expanded using a citation expansion tool; then, a multi-round question chain is generated around the real scientific research process, with the first few rounds focusing on paper location and graph evidence extraction, and the later rounds focusing on evidence reuse and cross-paper integration; next, human annotators review and correct the questions, answers, evidence units, and minimum necessary steps list; then, the evaluation sample is input into the tested multimodal agent, and its complete tool call trajectory and answer results are recorded in a unified tool environment; finally, based on the standard answer, evidence unit set, minimum necessary steps list, and error attribution rules, the evaluation results are output to the tested agent. In one implementation of EpiBench of the present invention, 68 seed papers are first collected, then expanded to obtain 485 candidate papers, ultimately forming 102 high-quality episodes, which are divided into Easy and Hard difficulty subsets for unified evaluation of different types of models.

[0039] In summary, the core of this invention lies in constructing a multi-round scientific research workflow evaluation benchmark, process-level evaluation protocol, and its implementation system for multimodal intelligent agents. Its purpose is to evaluate the capabilities of the tested intelligent agent in areas such as proactive search, cross-paper multimodal evidence integration, evidence memory reuse, and final comprehensive judgment. This invention does not rely on training, fine-tuning, or parameter updating of the tested intelligent agent model itself as a necessary technical means, nor does it consider improving the internal network structure of the tested model as its inventive point. The tested intelligent agent can be an open-source model, a closed-source model, or a human expert agent; as long as it can complete episode-level multi-round scientific research tasks within a unified tool environment, it can be used as the evaluation object of this invention.

[0040] In the embodiments or examples of the present invention, in order to construct evaluation data, an existing large model can be used to generate a draft of the questions, and then human annotators can review, revise and confirm the question chain, target evidence, standard answer and minimum necessary steps list.

[0041] The advantages of this invention are: This invention targets research agents in multimodal research workflows, focusing on evaluating their ability to proactively search for literature, locate evidence across papers, perform joint alignment analysis of multiple charts and text, archive evidence, and reuse memory in subsequent rounds within research assistance scenarios. EpiBench, on the other hand, targets multi-round research workflows, requiring agents to navigate across multiple papers, integrate information from multiple charts, and use accumulated evidence in memory to answer questions requiring cross-paper comparison and multi-chart integration. Therefore, this invention differs from general transactional agent evaluation in terms of evaluation objects, task content, and target capabilities.

[0042] This invention abstracts the real scientific research process into a multi-stage, interdependent episode-turn structure. It involves discovering papers and extracting evidence in earlier stages, and then explicitly reusing and integrating that evidence in subsequent stages to complete the final answer. This is a workflow-driven rather than subtask graph-driven evaluation paradigm. This characteristic is explicitly highlighted as core: a single example is not a single, independent question-and-answer session, but rather a multi-stage, continuous, interdependent scientific workflow where subsequent questions explicitly utilize findings from earlier stages. Therefore, this invention more closely reflects the real scientific research analysis process.

[0043] This invention further proposes a process-level evaluation protocol for scientific research workflows. In addition to assessing the correctness of the final result, it also evaluates the correctness of the evidence, whether the minimum necessary steps have been completed, whether there is redundant tool usage, and whether preceding evidence has been correctly reused in subsequent rounds. Specifically, by defining evidenceunits, minimally viable checklists, and final no-toolturns, this invention enables automatic verification of key issues such as "whether the source of evidence is correct," "whether the necessary chart access actions have been completed," and "whether the final round of responses truly relied on memory."

[0044] This invention preferably disables external tool access in the final round, allowing the agent to answer questions only using evidence archived in the memory pool from previous rounds. This technical arrangement can explicitly distinguish between "true evidence memory and reuse capability" and "the ability to obtain answers through repeated searches," thereby enabling a more realistic and rigorous evaluation of the agent's research workflow capabilities.

[0045] Furthermore, this invention can perform fine-grained attribution of retrieval errors, perceptual errors, reasoning errors, and memory retrieval errors for failed examples based on access trajectories, evidence comparison relationships, and response content. This type of error classification can effectively reveal the bottlenecks in the capabilities of different models in active search, graph reading, cross-evidence integration, and memory retrieval, thus possessing stronger diagnostic value and optimization guidance.

[0046] This invention proposes a multi-round scientific research workflow evaluation benchmark and process-level evaluation system for multimodal intelligent agents in scientific research support scenarios. This scheme can more realistically examine the agent's ability to proactively discover papers based on indirect cues, integrate charts and textual evidence across papers, and reuse prior memory to form verifiable research conclusions under limited tool conditions. Therefore, it has better technical effects in terms of the authenticity of scientific research workflows, interpretability of evidence, diagnosticability of processes, and relevance to practical applications.

[0047] It should be noted that in the above embodiments, as long as the technical solutions are not contradictory, they can be permuted and combined. Those skilled in the art can exhaust all possibilities based on the mathematical knowledge of permutation and combination. Therefore, the present invention will not describe the technical solutions after permutation and combination one by one, but it should be understood that the technical solutions after permutation and combination have been disclosed by the present invention.

[0048] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multi-round scientific research workflow evaluation system for multimodal intelligent agents, characterized in that: include: Multi-round task generation module: Constructs multi-round task scenarios that are closer to the real scientific research process; Multimodal information processing module: Enables the tested intelligent agent to perform active paper retrieval, multimodal evidence extraction and alignment, cross-paper evidence integration, and round-reuse of evidence.

2. The multi-round scientific research workflow evaluation system for multimodal intelligent agents according to claim 1, characterized in that: The multimodal information processing module includes: Active search module: Enables the tested intelligent agent to actively search for and obtain relevant papers related to the research question; Evidence extraction and alignment module: Extracts and aligns chart evidence from the aforementioned paper to achieve the association of multimodal information; Cross-paper multimodal information integration module: integrates multimodal information from several papers; Multimodal information recording and round reuse module: Reuse the acquired evidence in subsequent rounds to complete the final answer.

3. A multi-round scientific research workflow evaluation system for multimodal intelligent agents according to claim 1 or 2, characterized in that: The system also includes a process-level evaluation mechanism module: evaluating whether the final result of the agent's task processing is correct, whether the evidence is correct, whether the necessary steps are completed, and locating which capability link has an error.

4. A multi-round scientific research workflow evaluation method for multimodal intelligent agents, characterized in that: A multi-round scientific research workflow evaluation system based on any one of claims 1-3 includes the following steps: S1. Constructing an evaluation benchmark framework: This involves abstracting a set of stages from a real scientific research workflow, including research question formulation, relevant paper search, graph and chart inspection, cross-paper evidence alignment, intermediate evidence archiving, and final conclusion verification. S2. Multimodal task execution process: In the first round, images, charts, and problem descriptions are input into the agent to retrieve relevant papers, open the target paper, and extract image, table, or text evidence. The extracted evidence is used to answer the questions in this round. In the intermediate rounds, the agent continues to search for more papers based on the current problem and previous evidence results, supplements new chart evidence, and gradually forms intermediate conclusions across papers; In the final round, the agent directly provides a comprehensive answer based on the image evidence, tabular evidence, and text evidence previously archived in the memory pool.

5. The multi-round scientific research workflow evaluation method for multimodal intelligent agents according to claim 4, characterized in that: The method also includes S3, and practical testing and verification with various representative multimodal intelligent agents: A minimum necessary steps list is set for the evaluation samples to represent the key tool usage actions that must be included in any valid solution path. The completion and accuracy of the process are evaluated by comparing the agent's actual tool calling behavior with the minimum necessary steps list to determine whether its final answer is correct, whether there are redundant calls, and whether the error occurred in the search, perception, reasoning or memory reuse stage.

6. The multi-round scientific research workflow evaluation method for multimodal intelligent agents according to claim 5, characterized in that: The test subjects include open-source models, closed-source models, and human expert baselines. Through unified task input, tool environment, and evaluation criteria, their overall performance in multiple rounds of scientific research workflows is compared and analyzed.

7. The multi-round scientific research workflow evaluation system for multimodal intelligent agents according to claim 6, characterized in that: Failures are categorized into retrieval errors, perception errors, inference errors, memory retrieval errors, and a small number of other execution-related errors, thereby pinpointing the bottleneck in the model's capabilities.

Images