A context management method and system for multi-modal intelligent agent retrieval-oriented image adaptive loading

Abstract

The application discloses a context management method and system for image adaptive loading for multi-modal intelligent agent retrieval, which allocates identifiers for each query image and establishes an image registry; a dual-state context decoupling is adopted to enable the intelligent agent to alternate between a pure text main state and a temporary multi-modal branch state, wherein the main state only maintains a text plan, and the branch state loads original images on demand; the image guide is used to initialize the main state to inject a text priori; an adaptive branch-return folding is used to assign the loaded image subset according to a sub-task, and the image subset can be an empty set; after the branch is completed, the track is folded into a text evidence and written back to the main state, while the branch details are discarded. The process provided by the application is free of training and can be adapted to various multi-modal large models, and is used for open retrieval question answering of query images.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence, computer vision and multimodal information processing technology, and specifically relates to a context management method and system for adaptive image loading for multimodal intelligent agent retrieval. Background Technology

[0002] As multimodal large models continue to improve their capabilities in areas such as image and text understanding, visual question answering, and cross-modal reasoning, users' demand for open-ended question answering based on one or more query images is growing, and their retrieval intent is becoming increasingly complex. Such question answering often cannot be completed solely based on the model's own knowledge or single-step retrieval. Against this backdrop, the Multimodal Agentic Retrieval (MAR) task has emerged: given a query image and a natural language question, the agent autonomously plans the retrieval path, iteratively invokes external tools such as visual search, text retrieval, and web page access, collects multimodal evidence from the network, and completes the final answer.

[0003] Currently, to accomplish the aforementioned multimodal agent retrieval tasks, various agents generally adopt an "append-only" paradigm, represented by the reasoning-action alternation (ReAct). This paradigm continuously appends the original multimodal input, intermediate model inference, tool calls, and their return results to the same ever-growing context, with each round of model calls using the accumulated context as input. While this paradigm is simple to implement and highly versatile, it suffers from two problems in long-range, multi-step retrieval: First, the original image and historical interactions are indiscriminately preserved, leading to a continuous accumulation of noise information, which dilutes the model's attention to key information and consequently reduces its inference ability. Second, the context expands rapidly with each iteration, causing a corresponding rapid increase in input length and inference overhead. In multimodal scenarios, these costs are particularly pronounced because the original image itself occupies a large amount of context.

[0004] To alleviate the above problems, existing research has explored pure text retrieval scenarios. The methods involved can be roughly classified into two categories: one is the text summarization and compression baseline, which compresses the historical interactions in the context into a text summary and replaces the original history with the summary, which is then fed into the model along with the new input information; the other is the branching and folding baseline, which derives temporary processing procedures outside the main context to take over sub-tasks, and merges the processing results into the main context in text form after the sub-tasks are completed. However, its processing does not distinguish between modalities and does not perform special management on the original visual content.

[0005] The methods described above have achieved some success in pure text scenarios, but context management methods for multimodal retrieval scenarios are still relatively lacking. Due to the fundamental differences between text and images in terms of information density and context occupancy, applying context management methods designed for pure text to multimodal retrieval scenarios suffers from the following drawbacks:

[0006] (1) Text compression loses fine-grained visual evidence: The text summarization baseline summarizes visual evidence and text information together into a text summary. Fine-grained information in the image is difficult to be fully expressed by the text, and once it is not preserved in the summary, it cannot be recovered, resulting in a decrease in the accuracy of subsequent reasoning by the agent due to the lack of corresponding visual evidence.

[0007] (2) Indiscriminate retention of original images exacerbates context growth: The branch-folding baseline does not distinguish between modalities and incorporates the original images into each processing process indiscriminately. The massive original images continuously occupy the context, causing the context to grow continuously. At the same time, in sub-tasks that are mainly text reasoning, the original images that are not specially managed become redundant information, the proportion of key text evidence in the context decreases, and the model’s attention to them is diverted.

[0008] I. A search revealed that invention CN121351869A discloses a method for constructing an intelligent agent for long-range visual question answering and multimodal reasoning tasks. This method targets long-range visual question answering and multimodal reasoning tasks, generating answers through the ordered invocation of an external toolkit for input images and natural language questions. The intelligent agent construction process includes global planning, autonomous tool execution, and response synthesis: a global planning navigator analyzes user needs and selects a subset of tools; an autonomous tool executor performs multi-step invocations of tools for visual detail supplementation, background knowledge retrieval, and precise calculation; and a response synthesizer further refines the reasoning trajectory and outputs the answer.

[0009] Similar to this application, both focus on input images and natural language questions, and involve multi-step reasoning, external tool invocation, visual detail supplementation, background knowledge retrieval, and final answer generation. This document can serve as the closest application scenario document, illustrating that the long-range visual question-answering agent and tool invocation process have already been the subject of relevant research.

[0010] The main difference lies in the fact that CN121351869A focuses on "how to construct and schedule a multimodal reasoning agent," while this application focuses on "how to manage the original image and context state during the agent's operation." Corresponding to the first feature mentioned above, this document does not disclose an image registry and stable image identifiers, nor does it allow the main state to reference the original image solely by its identifier. Corresponding to the second feature mentioned above, this document does not disclose the modal asymmetry between the plain text main state and the temporary multimodal branch state. Corresponding to the third and fourth features mentioned above, although this document provides tool results to assist in the next step of decision-making, it does not disclose the initialization of branch-generated image concise descriptions, the assignment of image subsets by subtask, and the folding of branch evidence into text and discarding the branch's local trajectory and the loaded original image by the return action. Corresponding to the fifth feature mentioned above, this document also does not establish an image on-demand loading and release mechanism under branch budget and global budget constraints. Therefore, this document cannot cover the context management control flow.

[0011] II. According to the search, the invention patent with publication number CN118885627A discloses a multi-turn dialogue processing method and system based on RAG and knowledge graph. This solution belongs to the multi-turn dialogue processing method based on RAG and knowledge graph. Its process includes: retrieving and identifying entities related to the user question based on the proprietary knowledge graph to obtain associated entities, constructing inference links and converting them into inference text sequences; retrieving text, image, and audio data related to the user question through a multimodal large model to obtain relevant multimodal data; integrating the relevant multimodal data, the current inference text sequence, the user question, and the generated answer text as model input to obtain the model answer; in subsequent questions to be processed, relevant multimodal data is filtered from the historical dialogue list, and the filtered multimodal data is used as context and input into the multimodal large model along with the questions to be processed.

[0012] Similar to this application, both involve multi-turn question processing, external knowledge enhancement, and the retrieval and contextual use of multimodal data such as text and images. This document can be used to illustrate that, in multi-turn dialogues, filtering relevant multimodal data based on historical dialogues and using it as context for subsequent models is a processing path in existing technologies.

[0013] The main difference is that CN118885627A focuses on improving the quality of multi-turn dialogue responses through entity recognition, proprietary knowledge graph reasoning links, and multimodal data retrieval, and its "context" is relevant multimodal data selected from a list of historical dialogues; this application focuses on managing the loading, retention, and release boundaries of query images in long-range inference of intelligent agents. Regarding the first characteristic mentioned above, this document does not disclose the establishment of an image registry and stable identifier for each query image, nor does it ensure that the main state only references the original image using the identifier. Regarding the second characteristic mentioned above, this document uses the selected text, images, audio, and other related multimodal data as model input or context, and does not disclose that the persistent main state must remain plain text and that the original image must not reside in the main state for an extended period. Regarding the third characteristic mentioned above, its inference text sequence comes from the knowledge graph inference link, which is not equivalent to generating a concise description for all query images and returning to the main state in the initialization branch of this application. Regarding the fourth and fifth characteristics mentioned above, this document does not have the mechanism of creating temporary multimodal branches by subtask, retrieving the original image by image identifier, and using the return action to only write the evidence text back to the main state and release the local trajectory of the branch. Therefore, what this document discloses is "knowledge graph + RAG + multi-turn dialogue context filtering", rather than the image adaptive loading and branch-return folding control flow of this application.

[0014] III. A search revealed that invention CN121858763A discloses a multimodal dynamic retrieval enhanced question answering method, apparatus, device, and medium. This solution belongs to the multimodal dynamic retrieval enhanced question answering method and is applicable to fields such as medicine and finance. Its core process is as follows: based on domain category and image data, pre-inference is performed to generate preliminary answers and confidence indices for natural language questions; the answer generation path is dynamically selected based on the confidence level; when the enhanced retrieval path is activated, the fusion retrieval chain is invoked to search for natural language questions, image data, and domain categories, obtaining enhanced retrieval results and reordering them to form a confidence evidence set; finally, the evidence set, question, and image data are fused and inferred in context to generate the answer.

[0015] Similar to this application, both focus on supplementing external knowledge in multimodal question answering and incorporate the idea of ​​"dynamically deciding whether to conduct further retrieval or reasoning based on the task status." This document can be used to illustrate that selecting the retrieval path based on confidence level is an existing solution in multimodal RAG scenarios.

[0016] The main difference lies in the dynamics. CN121858763A's dynamism stems from confidence-driven answer generation path selection, addressing the question of "whether to initiate enhanced retrieval and how to fuse retrieval evidence." This application's dynamism arises from the main state selecting image subsets at the subtask granularity and switching to temporary branches, addressing the question of "when the original image enters the context and when it exits the context." Corresponding to the first and second features mentioned above, this document does not disclose an image registry, stable image identifiers, or a plain text main state; instead, it continuously uses image data in pre-inference, fusion retrieval, and context-fusion inference. Corresponding to the third feature mentioned above, this document does not use the initialization of branches to generate concise descriptions of each image as a priori for main state planning before regular inference. Corresponding to the fourth and fifth features mentioned above, this document also lacks a mechanism where the main state assigns image subsets, branches retrieve the original image by identifier, and then the return action folds textual evidence and releases the branch trajectory. Therefore, this document discloses a dynamic retrieval enhanced question-answering path, while this application discloses a context state switching mechanism for adaptive image loading.

[0017] IV. A search revealed that Patent CN117010500A discloses a method for visual knowledge reasoning and question answering that leverages multi-source heterogeneous knowledge. The basic process is as follows: First, a multimodal model is used to generate text descriptions of images and candidate answers to questions, serving as auxiliary information for multi-source heterogeneous knowledge retrieval. Then, four types of heterogeneous knowledge are retrieved: corresponding encyclopedic knowledge, relevant webpage text knowledge, relevant webpage image knowledge, and large-scale language model knowledge. Subsequently, a heterogeneous knowledge fusion reasoning model is trained based on the multi-source heterogeneous knowledge to improve the quality and accuracy of the visual question answering model's responses to questions requiring external knowledge.

[0018] Similar to this application, both involve images, questions, external knowledge retrieval, and visual question answering reasoning, and both use image-text descriptions to assist subsequent processing. This document can be used to illustrate that using image descriptions and candidate answers to improve the quality of external knowledge retrieval and train a knowledge fusion reasoning model is an existing approach in the field of visual knowledge question answering.

[0019] The main difference lies in the role of "image text description" in CN117010500A. In CN117010500A, the "image text description" serves as auxiliary information for multi-source heterogeneous knowledge retrieval, improving the quality of retrieved encyclopedia entries, webpage text, webpage images, and large-scale model knowledge, and further training the knowledge fusion reasoning model. In this application, the "image-guided initialization" function returns a concise description of all queried images to the pure text main state as a priori for subsequent subtask planning, while retaining the original images for subsequent review by identifiers in temporary branches. Corresponding to the first feature mentioned above, this document does not disclose an image registry and stable image identifiers. Corresponding to the second feature mentioned above, this document does not form an asymmetric structure between the pure text main state and the temporary multimodal branch state. Corresponding to the fourth feature mentioned above, this document retrieves multi-source heterogeneous knowledge and trains the fusion reasoning model, but does not disclose the process of selecting image subsets by subtask in the main state, loading original images on demand for branches, and only writing back text evidence after the branch ends. Corresponding to the fifth feature mentioned above, this document emphasizes training the heterogeneous knowledge fusion reasoning model, while this application emphasizes runtime context management under training-free conditions. Therefore, this document cannot extend "generating image descriptions for retrieval assistance" to the combined mechanism of "initialization description + original image registration for review + temporary branch loading and release" of this application.

[0020] V. A search revealed that invention publication number CN120218237A discloses a multimodal explicit memory system, device, storage medium, and program product. This solution discloses a multimodal explicit memory system, including a multimodal encoding module, a unified memory representation module, an adaptive sparsity module, a hierarchical retrieval and fusion module, and a context-enhanced reasoning module. Its basic idea is to convert multimodal data such as text, visual, and auditory data into vectorized features, perform cross-modal alignment, construct an explicit memory bank, and then provide context-enhanced reasoning results to a large language model through pruning, quantization, hierarchical retrieval, and fusion.

[0021] Similar to this application, both recognize that multimodal information presents challenges in context organization and compression during retrieval and reasoning, and both attempt to enable large language models to obtain more effective multimodal relevant evidence. This document can serve as a comparative document in the direction of "multimodal memory and context compression".

[0022] The main difference lies in the fact that CN120218237A's core is to establish a persistent cross-modal explicit memory and reduce memory and retrieval costs through vectorization, sparsity, pruning, or quantization; this application does not uniformly convert query images into persistent vector memory, nor does it rely on pruning or quantization of original image features to achieve context compression. Corresponding to the first feature mentioned above, this application retains original images that can be retrieved by identifier through an image registry; corresponding to the second feature mentioned above, this application keeps the main state as plain text, rather than allowing multimodal memory to participate in the main state for a long time; corresponding to the third and fourth features mentioned above, this application only generates concise image descriptions as planning priors during the initialization phase, and then temporarily loads the original images according to subtasks and folds text evidence with the return action; corresponding to the fifth feature mentioned above, this application constrains the inference trajectory through branch budget and global budget, rather than using memory sparsity as the main control method. Therefore, this document cannot provide the technical inspiration for this application to "preserve the reproducibility of the original image while avoiding the original image residing in the context for a long time."

[0023] Therefore, there is an urgent need for a context management method and system for multimodal agent retrieval that can manage multi-source context information such as query images, text questions, tool call results, and intermediate inferences. At the same time, it can control the compression, loading, and retention range of different modal information in multi-round inference, so as to avoid the irrecoverable key visual evidence caused by uniform compression without distinguishing modal characteristics, or the context expansion and long-range inference interference caused by the indiscriminate retention of multimodal information. Summary of the Invention

[0024] The technical problem to be solved by this invention stems from two contradictions that arise when directly reusing context management methods in plain text scenarios to multimodal agent retrieval. These contradictions correspond one by one to the two defects pointed out in the background art:

[0025] Contradiction 1 (corresponding to defect 1, text compression leads to the loss of fine-grained visual evidence): When simplifying multi-turn contexts and controlling inference overhead, it is necessary to minimize the long-term residence of original images in the context. However, while uniformly compressing original images into text can reduce context occupancy, it will lose fine-grained visual evidence that text cannot fully express, resulting in decreased accuracy of subsequent inferences due to the lack of corresponding visual evidence. Therefore, it is necessary to solve the problem of how to reduce the long-term residence of original images to simplify the context without losing the fine-grained visual evidence necessary for inference.

[0026] Contradiction Two (corresponding to Defect Two: Indiscriminate retention of original images exacerbates context growth): While preserving necessary visual information, it is necessary to control the context occupancy of original images and historical observations in multi-round reasoning. However, while indiscriminately inheriting original images to each processing stage allows for the readily available use of visual information, it causes the massive original images to continuously occupy context, exacerbating the expansion and becoming redundant interference in text-based reasoning subtasks. Therefore, the solution lies in how to control the context occupancy and interference of original images in multi-round reasoning while preserving necessary visual information.

[0027] To address the aforementioned technical problems, this invention proposes a context management method and system for adaptive image loading in multimodal agent retrieval, wherein the method includes:

[0028] S1. Register and define input tools: Assign stable identifiers to each query image and build an image registry. At the same time, define the toolset that the agent can call and the capabilities and input / output specifications of each tool, so as to provide a basis for the agent to assign images by identifiers and call tools to collect evidence.

[0029] S2. Construct a dual-state context decoupling mechanism: The same agent alternates between a main state and a branch state, with each state corresponding to a different context. The two states are modally asymmetric. The main state is always a plain text state, corresponding to a continuously retained plain text context that does not carry the original image information. The branch state is a temporary multimodal state, corresponding to a temporary multimodal context. The branch is invoked by the main state and runs based on the inherited main state context. After the branch ends, its branch context is folded into a plain text summary, the result is written back to the main state, and then the original details of the branch are discarded and the system switches back to the main state.

[0030] S3. Image-guided initialization injects planning priors into the main state: Before the regular branch-return loop, a fixed initialization loop generates a text description for each query image and writes it into the main state, thereby performing high-level planning related to images without holding the original images.

[0031] S4. Drive retrieval and response with an adaptive branch-back folding loop: The main state plans subtasks in a plain text context and explicitly assigns the image subsets to be loaded. The image subsets can be empty sets, which correspond to plain text subtasks that do not require visual information. The main state invokes a branch and switches states. The branch completes visual perception and tool retrieval within budget, then folds the obtained evidence into a plain text summary and writes it back to the main state, and then discards the branch context. This process is iterated until sufficient evidence is available, and the main state outputs the final answer.

[0032] The above process does not update any parameters of the backbone model, i.e., it is training-free throughout. Through multi-step inference control of dual-state context decoupling, image-guided initialization and adaptive branch-return folding, the original image is loaded on demand and released in a timely manner. By folding the multimodal interaction trajectory in the temporary branch into a plain text summary and writing it back to the main context, the multimodal agent retrieval and response are completed.

[0033] The specific steps described above are explained below:

[0034] S1. Input Image Registration and Tool Definition:

[0035] Given the original input, a natural language problem With a set of query images The present invention first registers the input image and defines the available tools.

[0036] This invention provides each query image Assign a stable image identifier And construct an image registry, which can preferably be formalized as follows:

[0037] ;

[0038] in, Represents the image registry. Indicates assignment to the first Zhang queries the image's stability identifier. Indicates the first Zhang query image, This indicates the total number of images being queried. Using this registry, the agent can retrieve the images solely based on the set of identifiers. Reference and assign images.

[0039] Preferably, the present invention addresses the natural language problem. and As input to the intelligent agent.

[0040] This invention defines the toolset that an intelligent agent can invoke. And clearly define the capabilities and input / output specifications of each tool.

[0041] Preferably, the toolset The system includes three types of tools: visual search, text retrieval, and web page access. Visual search takes an image as input, retrieves visually relevant information from the web, and returns images that are visually similar to the original image, along with the titles and links to the source web pages. More preferably, visual search can accept the original image or its image identifier as input; if the identifier is used as input, the system retrieves the original image from the image registry before searching. Text retrieval takes a text query generated autonomously by the agent as input, performs an open-domain web search, and returns several relevant results along with their titles, abstracts, and web page links. Web page access takes a specified URL as input, accesses the web page, and returns its parsed text content.

[0042] S2, Dual-state context decoupling:

[0043] To address the problem that existing append-only paradigms involve the original visual token residing entirely in a single context and being coupled with textual planning, this invention enables the agent to alternate between two states during reasoning, thereby decoupling visual perception from high-level textual planning. Specifically, the pure text main state only maintains the question, textual evidence, and high-level task planning, without carrying original image information; the temporary multimodal branch state, based on the assignment of the main state, follows the image registry described in step S1. The corresponding original image is loaded on demand for local visual perception. Both states share the same multimodal large model backbone. However, the action space is different: the main state only emits... or When higher-level control actions are initiated, the branch state invokes the toolset. or Control actions; when the main state is in progress After the action, the branch is invoked by the main state, runs on the basis of the main state context, and after it is finished, its local context is collapsed into a plain text summary, the result is written back to the main state, and the state then switches back to the main state.

[0044] The transition between the main state and the branch state is determined by and Two actions to manage: Actions are initiated from the main state and carry sub-tasks. With image identifier subset This triggers a switch from the main state to the branch state; The action is initiated by the branch state, folds the multimodal evidence collected in the branch into a plain text summary, returns to the main state, and terminates the branch. Preferably, it can be defined in detail as follows:

[0045] ;

[0046] in, Indicates the first The branch action initiated in the main state, This indicates the subtask that this branch needs to complete. This indicates the subset of image identifiers to be injected in this branch.

[0047] ;

[0048] This indicates a return action initiated within a branch. This represents a plain text evidence summary formed by branching and folding.

[0049] After each main-branch loop is completed (i.e., terminated with RETURN and the local context of that branch is immediately discarded), the main state context is updated accordingly, preferably as follows:

[0050] ;

[0051] in, Indicates the first The plain text context maintained by the main state. This represents the updated main state context. Indicates the first The inference text generated by the main state. This indicates the branch action of this step. This represents a plain text evidence summary indicating that the branch has returned to the main state.

[0052] The dual states are modally asymmetric: when the main state is running, the model context is always text information, the original visual evidence only exists in the temporary branch state context and is only written back to the main state in the form of folded text, and the branch context (including the original image and redundant interaction trajectory) is discarded after the branch ends, thereby eliminating the continuous residence and accumulation of the original visual token in the main state.

[0053] S3, Image Guidance Initialization:

[0054] Because the main state context is plain text and only holds a set of image identifiers. Without knowing the specific content of each image, it is difficult to make effective high-level image-related planning. Therefore, this invention introduces an initialization loop before the agent enters the regular branch-return loop, initiated by the main state and directed towards all visual inputs. The initialization branch loads all the original images and generates a concise text description for each query image. These descriptions are then returned to the main state as reliable priors for subsequent image-related planning.

[0055] The initiation and return of the initialization branch are as follows: The main state first initiates the initialization branch action, which can preferably be expressed as follows:

[0056] ;

[0057] in, The inference text representing the initialization step. This indicates the initialization branch action. The backbone model representing the intelligent agent, To indicate a problem, The set of identifiers representing all images. This represents a pre-defined, problem-independent initialization subtask that requests a high-level overview of all visual inputs (without involving fine-grained visual verification); subsequently, within this initialization branch, all raw images are loaded and a return action is performed. Preferably, it can be formalized as follows:

[0058] ;

[0059] in, This indicates the reasoning text within the initial branch. This indicates the return action within the initialization branch. Indicates the local context for initializing the branch. This represents the complete original query image that was injected into this branch.

[0060] The return action An initialization result consisting of concise text descriptions of each image is generated, and an initial main context is constructed based on this. Preferably, it can be formalized as follows:

[0061] ;

[0062] ;

[0063] in, This represents the set of descriptions that indicate the initialization branch reverts to the main state. Represented as the first Zhang query image The generated concise text description, This represents the main state context after initialization is complete.

[0064] Preferably, the description Deliberately keeping it brief and capturing only high-confidence visual content as a reliable planning prior rather than replacing fine-grained details of the original image; thus, the initial main context... It contains no original image information, but has been obtained through... Obtain semantic priors for the content of each image, thereby supporting high-level planning related to images in the plain text master state.

[0065] S4, Adaptive Branching-Return Folded Reasoning:

[0066] Since the necessity of the original image changes dynamically with the task and reasoning process, and different subtasks have different requirements for visual information, this invention obtains the initial main context. Then, the process proceeds to regular iterations: the main state adaptively plans subtasks within a plain text context, and explicitly assigns the subset of images to be injected into branches according to the granularity of the subtasks; the branches load the corresponding original images as needed, complete visual perception and tool retrieval within their branch budget, and then... The obtained subtask evidence is folded into text and written back to the main state, then the entire branch context is discarded; this process is iterated until sufficient evidence is available, at which point the final answer is output from the main state or the budget is invoked globally. When the maximum number of calls allowed in the entire retrieval process is exhausted, a response must be forced based on existing evidence.

[0067] In each regular iteration step, the master state makes a decision based on the current master context, which can preferably be formulated as follows:

[0068] ;

[0069] Specifically, the main state is first based on the current main context. Assess whether the evidence collected is sufficient to answer the question. If the collected evidence is sufficient, then choose... Output the final answer; otherwise, generate a new subtask for the currently unmet information gap. And assign a subset of image identifiers to be injected into the branch based on the visual requirements of the subtask. .in Indicates the first The reasoning text of the master state, Indicates the first The control action selected by the main state. Indicates the first The main state context of the step, This indicates that the collected evidence is sufficient to answer the question. Output the final answer in time. And end the search. Indicates planning new sub-tasks And assign a subset of image identifiers to be injected into the branch. In particular, Can be an empty set This indicates that the subtask does not require visual information and is a plain text subtask, thereby enabling adaptive image allocation at the subtask granularity.

[0070] When the main state is initiated At this time, the branch inherits the main context and loads the assigned original image as needed. Preferably, the branch context initialization is formulated as follows:

[0071] ;

[0072] in, Indicates the first state in the main state. The initial local context of the branch generated by the step. This indicates the action that generated the branch. Indicates based on a subset of identifiers The original image set is retrieved and injected into the branch on demand from the image registry P; thus, the branch simultaneously possesses the global task background, previously collapsed text evidence, and the original images loaded on demand.

[0073] Within a branch, the model revolves around the subtask it has undertaken. Based on the loaded original images and the information previously collected within this branch, the next steps are planned step by step, including calling in tools to continue collecting evidence, or, when the obtained information is sufficient to complete the subtask... Convergence. Preferably, it is formulated as follows:

[0074] ;

[0075] in, , They represent the first Within the first branch The reasoning text and actions of each step Indicates the first branch within that branch Step into the previous local context, For toolsets; if Then execute the tool call to obtain observations. and the triplet Add to the branch's local context; if The branch terminates and the text summary is generated. Return to the main state. Preferably, the complete local trajectory of this branch is formulated as follows:

[0076] ;

[0077] in, Indicates the first The complete local trajectory of each branch This indicates the actual number of steps the branch scrolled. This represents the branch budget (i.e., the maximum number of steps / rounds allowed within a single branch); if the branch budget is exhausted before it is explicitly issued... Then a forced call Summarize and return the collected information, while notifying the main state that the subtask is incomplete, to ensure that the branch will eventually converge.

[0078] This module sets the fold boundary: only Plain text summary of the action (Including the current subtask result, supporting evidence, and remaining uncertainty) is written to the persistent main state context, and the complete multimodal trajectory of the branch is then discarded. The main state context is then processed in accordance with the method described in step S2 (i.e. Update; furthermore, tool usage within branches follows a "visual perception first, text retrieval later" pattern: when a branch receives a non-empty subset of images... In practice, visual search is typically invoked first to perform visual perception and identify candidate entities or clues, before proceeding to text retrieval and web page access. Optionally, each tool call carries a target field describing the search intent to reduce redundant calls.

[0079] The present invention also provides a context management system for adaptive image loading for multimodal agent retrieval, corresponding to the above method. The system includes a dual-state context decoupling module, an image guidance initialization module, and an adaptive branch-return folding inference module.

[0080] The dual-state context decoupling module is used to construct an image registry and enable the agent to switch between a plain text main state and a temporary multimodal branch state, thereby separating high-level text planning from local visual perception. The image guidance initialization module is used to generate concise text descriptions for query images, providing initial text priors related to the image for the main state. The adaptive branch-return folding inference module is used to assign the subset of images to be loaded (allowing for an empty set) according to the current subtask requirements, complete the corresponding visual perception, retrieval, and evidence collection within the branch state, and fold the branch results into text evidence to write back to the main state. The correspondence between the above three modules and the various parts of the technical solution is as follows: the dual-state context decoupling module implements input registration, tool definition, and dual-state decoupling.

[0081] The image guidance initialization module implements image guidance initialization.

[0082] The adaptive branch-return folding inference module implements adaptive branch-return folding.

[0083] All of the above modules operate collaboratively without updating the parameters of the backbone model, meaning that training is not required throughout the entire process.

[0084] The technical solution of the present invention has the following beneficial technical effects:

[0085] 1. In the method provided by this invention, the agent assigns a subset of images to be loaded according to the subtask granularity. After completing visual perception and tool retrieval within the branch according to the budget, the conclusive evidence is folded into a text write-back master state, and then the original image is released. After the original image is registered in the image registry, it can be retrieved again as needed by the identifier. Its fine-grained content is discarded if it is not summarized into text all at once. This overcomes the defects of irreversible text compression in the background technology, and the inability to recover visual evidence once it is summarized into text and not preserved by the summary. It allows subsequent reasoning to reload the original image to obtain fine-grained visual evidence when needed, thereby improving the accuracy of answers in complex and multi-image scenarios.

[0086] 2. The original image is loaded only on demand in the temporary multimodal branch and exits with the branch after use. The plain text main state does not carry the original image, and the main state context is accumulated only with the collapsed text evidence. Specifically, the image guidance initialization module generates concise text descriptions for each query image and writes them into the main state before entering the regular branch-return loop. This allows the main state to replace the persistent retention of the original image with a concise text prior, so that it can still carry out high-level planning related to the image based on the text description without holding the original image. This addresses the shortcomings of the background technology, such as the indiscriminate inheritance of images and the continuous occupation of the context by the huge amount of visual content (corresponding to defect 2). It prevents the main state context from accumulating the original visual content with each round, thereby reducing the occupation of working context and the overhead of a single inference. Furthermore, since the proportion of key text evidence in the main state is no longer diluted by the original image, the model's attention to key text evidence is maintained in long-range multi-step retrieval, alleviating the interference of context expansion on inference.

[0087] 3. In the system provided by this invention, each module is implemented through stateful context organization and control flow scheduling during the inference phase, without updating any parameters of the backbone model used; the training-free approach is the implementation method of the above-mentioned effects of this invention rather than an independent effect. Its function is to enable the above-mentioned context management to access a multimodal large model with the ability to understand graphics and text, follow instructions and call tools without relying on parameter fine-tuning, which is convenient for migration and expansion on different backbones. Attached Figure Description

[0088] Figure 1 This is a schematic diagram of the overall process of the image adaptive loading context management method for multimodal intelligent agent retrieval according to the present invention;

[0089] Figure 2 This is a schematic diagram of the module structure of a context management system for adaptive image loading for multimodal intelligent agent retrieval according to the present invention;

[0090] Figure 3 This is a schematic diagram of the dual-state structure and state switching of the plain text main state and the temporary multimodal branch state in this invention.

[0091] Figure 4 This is a schematic diagram of the image registry and the data flow of images being loaded on demand and released after use in this invention;

[0092] Figure 5 This is a schematic diagram of the image-guided initialization process in this invention;

[0093] Figure 6 This is a schematic diagram of the adaptive branch-return folding reasoning process in this invention under branch budget and global budget constraints. Detailed Implementation

[0094] The specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0095] I. Implementation Examples:

[0096] Example 1:

[0097] This embodiment describes the overall reasoning process, implemented according to the input registration, dual-state context decoupling, image-guided initialization, and adaptive branch-return folding parts of the aforementioned scheme. The overall process is as follows: Figure 1 As shown, the module structure of the corresponding system is as follows: Figure 2 As shown.

[0098] This embodiment achieves end-to-end retrieval and response without updating the backbone model parameters (i.e., without training). During the inference phase, it implements multi-step control through dual-state context decoupling, image-guided initialization, and adaptive branch-return folding. The overall process includes the following steps:

[0099] S1, for each query image Assign stable identifiers Build an image registry and define the available toolset. ;

[0100] S2. Run the initialization loop once at a fixed time to generate concise text descriptions for each query image. And accordingly initialize the main context to ,in ;

[0101] S3, when no answer has been received and the global budget is invoked. If not exhausted, repeat steps S4 to S6.

[0102] S4. Assess whether the collected evidence is sufficient to answer the question using the plain text master state. If sufficient, then issue. Output the final answer And end; otherwise, plan subtasks to address the current information gaps. Assigning a subset of image identifiers ( It can be an empty set (∅), and emits ;

[0103] S5. Create a branch and switch to branch state: The branch inherits the main context, by... Load the corresponding original image from the registry, within the branch budget. Within the scope of the task, visual perception and tool retrieval are performed around the sub-tasks, and tool calls are issued at each step. ;

[0104] S6, branch After concluding, summarize the text. Return to the main context, discard the local branch trajectory, switch back to the main state, and return to step three to continue the loop.

[0105] The backbone model is not updated throughout the entire process. Any parameters are implemented only through stateful context organization and control flow scheduling during inference, therefore this invention is a training-free framework.

[0106] at the same time, Figure 3 It shows the asymmetry between the main state and the branch state in the action space, as well as the relationship between state switching and text evidence rewriting driven by branch actions and return actions; Figure 4 The process is illustrated, including the registration of the image and image identifier, the main state referencing the image only by the identifier, the retrieval of the original image by the identifier when a branch is assigned, and the release of the image after the branch ends. Figure 5 The process is illustrated by loading all query images by the initialization branch before the regular branch-back loop, generating concise text descriptions for each image, and then looping back to the main state to build the initial main context. Figure 6 The process is illustrated as follows: the main state adaptively assigns the subset of images to be loaded at the subtask granularity; the branch completes visual perception and tool retrieval within the branch budget; the obtained evidence is folded into text and written back to the main state and the branch's local trajectory is discarded; and the process iterates under the global budget constraint until a response is given.

[0107] Example 2:

[0108] This embodiment is a preferred configuration of the interface, model, and hyperparameters. In one preferred configuration, the toolset... The three types of tools are implemented using corresponding external service interfaces: visual search performs image search via an interface with reverse image retrieval capabilities, text search performs web page search via an interface with open domain web page retrieval capabilities, and web page access retrieves and parses the web page text of a specified URL via an interface with web page text extraction and parsing capabilities. Each tool call carries a goal field that describes the search intent to reduce redundant calls. Preferably, the visual search and text search can use the reverse image search interface and web page search interface provided by the same search service provider (e.g., a type of aggregated search interface), and web page access can use a type of web page text extraction interface; alternatively, other interfaces with corresponding retrieval and parsing capabilities can also be used, and the implementation is not limited to the above.

[0109] Preferably, a backbone model This invention employs various multimodal large models, such as open-source or closed-source multimodal large models of different sizes. The specific model type and parameter scale of the model used are not limited; any multimodal large model with the capabilities of text / image comprehension, instruction following, and tool invocation can be used as an inference base model. More preferably, the sampling temperature during model inference is 0.2, and top_p is 0.95. More preferably, branch budget... Take 5, global call budget The value is set to 20. Optionally, the above sampling parameters and budget upper limit can be adjusted according to the specific task difficulty, backbone model capabilities, and latency requirements. These are just examples and are not limited to the values ​​mentioned above.

[0110] Example 3:

[0111] This embodiment serves as a prompt word template for various model-driven steps. In this invention, several steps driven by a multimodal large model (image-guided initialization description generation, branch-folded summary generation, and forced return when branch budget is exhausted) all run on the same backbone model. The above uses the common sampling configurations such as sampling temperature and top_p described in Embodiment 2; the specific prompt templates, input and output fields, and their respective constraints for each step are given below. The following templates are one of the representative preferred embodiments, and the wording can be adjusted without changing their functions and output constraints during actual deployment. In the following prompt templates, the curly braces {} are placeholders, and the corresponding content should be filled in when calling them; the square brackets [ ] are fixed output field names, which must be output as is.

[0112] (a) Image-guided initialization description Generation:

[0113] This step loads all original images via a branch in the initialization loop. The model then generates a concise high-level overview description for each query image, capturing only high-confidence visual content without fine-grained verification. The output length is limited to ensure it serves as a reliable planning prior rather than replacing fine-grained details of the original images. In one embodiment, the prompt template is as follows (where {} are placeholders):

[0114] You are an image overview assistant responsible for generating concise, high-level overview descriptions of the images in the following query, enabling the plain text master state to perform high-level image-related planning without holding the original images.

[0115] enter:

[0116] Original question: {question} }:

[0117] Image identifier: {identifier }:

[0118] Image content: {the Zhang Original Image }:

[0119] rule:

[0120] 1. Describe only the overall content that is highly confident and easily identifiable (such as the main subject of the image, category, scene, prominent entity, or text), providing an overview;

[0121] 2. No fine-grained fact-checking, no speculation about invisible information, and no use of any tools;

[0122] 3. This description is for planning priors only and is not intended to replace details of the original image;

[0123] 4. Each image corresponds to a description, and is identified by its identifier. Annotation.

[0124] Please output in the following format, with the description of each image not exceeding { } tokens:

[0125] Image { identifier }: {A one- or two-sentence high-level overview description}.

[0126] (b) Branch Folding Summary Generation:

[0127] This step is in the branch During convergence, the model converges the multimodal interaction trajectories within the branches into a plain text evidence summary and writes it back to the main state. During generation, the original images and redundant observations are discarded, retaining only conclusive evidence relevant to the subtask, while limiting the output length. The summary is organized according to three fixed fields: [Current Hypothesis], [Supporting Evidence], and [Remaining Uncertainty]. In one embodiment, the prompt template is as follows (where {} are placeholders and [ ] are fixed output field names):

[0128] You are an evidence folding assistant. Please gather the multimodal information collected in the current branch to complete the subtask, summarize it into a plain text evidence, and write it back to the main context for the main state to use for subsequent high-level planning.

[0129] enter:

[0130] Subtask: {Subtask} }:

[0131] Observations and intermediate conclusions collected in this branch: {Tool calls, visual perception, and reasoning records in the branch's local trajectory}:

[0132] rule:

[0133] 1. Output only plain text conclusions, without including original images or image data, and without attaching images or screenshots;

[0134] 2. Only retain evidence relevant to the subtask and supporting subsequent responses, and discard redundant or irrelevant observations;

[0135] 3. If the evidence is insufficient to complete the subtask, it should be stated truthfully in the remaining uncertainty section.

[0136] Please strictly adhere to the following three-field format when outputting, with a total length not exceeding { } tokens:

[0137] [Current Assumption] {Current conclusion or most likely answer for the subtask}:

[0138] [Supporting Evidence] {Key Facts and Sources Supporting the Above Hypothesis}:

[0139] [Remaining Uncertainties] {Aspects that have not yet been confirmed or require further investigation}.

[0140] (c) Forced return when branch budget is exhausted:

[0141] When the number of tool calls within a branch reaches the branch budget The model has not yet been explicitly released. At this time, the forced final action is a summary return of the collected information, which folds the existing findings according to the three-field format described in (b) and writes them back to the main state. At the same time, it notifies the main state that the subtask has not yet been completed, so as to ensure that the branch will inevitably converge. In one embodiment, the prompt template is as follows (where {} are placeholders and 【 】 are fixed output field names):

[0142] The tool call budget for this branch has reached its limit ({number of tool calls used} / {branch budget}). Now, force the termination of this branch and return. Do not request any further calls to tools.

[0143] enter:

[0144] Subtask: {Subtask} }:

[0145] Observations and intermediate conclusions collected in this branch: {Tool calls, visual perception, and reasoning records in the branch's local trajectory}:

[0146] rule:

[0147] 1. Summarize only the information currently collected, without acquiring new information or initiating any tool calls;

[0148] 2. Discard the original images and redundant observations, retaining only the usable conclusions;

[0149] 3. Mark the remaining uncertainties for subtasks that were not completed due to budget exhaustion, as well as any missing information.

[0150] Please strictly adhere to the following three-field format when outputting, with a total length not exceeding { } tokens:

[0151] [Current Assumption] {Tentative Conclusion Based on Current Information}:

[0152] [Supporting Evidence] {Key Facts and Source Clues Already Obtained}:

[0153] [Remaining Uncertainties] {Parts that could not be completed due to budget exhaustion and missing information}.

[0154] Example 4:

[0155] This embodiment provides a computer device for implementing the image adaptive loading context management method for multimodal intelligent agent retrieval described in Embodiments 1 and 2 above. The computer device includes a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the steps of the aforementioned method, namely, the input image registration and tool definition, image guidance initialization, and adaptive branch-back folding inference processes described in steps S1 to S6. Optionally, the computer device may further include a network interface for communicating with external tool services to call tools such as visual search, text retrieval, and web page access described in Embodiment 2 via the corresponding interface; the backbone model... It can be deployed on local computing resources or invoked via a remote service interface; this invention does not limit this. The computer device can be a server, desktop computer, portable computer, or computing node with corresponding computing power, or it can be composed of multiple computing devices working together. The method steps executed by the processor, the interfaces used, the models and hyperparameters, and the prompt word templates in this embodiment are as described in Embodiments 1 to 3, and will not be repeated here; the above specific forms are only examples and do not constitute a limitation on the scope of protection.

[0156] II. Experimental Comparison Results:

[0157] Baseline method comparison:

[0158] This study selected the following representative baseline methods for comparison in the multimodal agent retrieval task. All baseline methods share the same backbone, the same tool API, the same answer format, and the same global call budget with this invention. = 20, to ensure the fairness of the comparison:

[0159] Direct inference: Answers using only the parameterized knowledge of the backbone model, without calling any external tools, serving as a lower bound reference without using retrieval.

[0160] ReAct, presented at the ICLR conference in 2023, is a reasoning-action interleaved agent paradigm that employs a single, continuously growing context that is only appended and is widely used as the skeleton in multimodal deep research systems. This invention uses WebWatcher, an open-source high-performance multimodal deep research agent released in 2025, as its implementation.

[0161] AgentFold, presented at the 2025 ICLR conference, is an active context management method for long-range web intelligent agents that performs multi-scale text compression on historical interactions. This invention retains its original compression logic and configures it with multimodal query and Visual Search interfaces.

[0162] ContextFold, presented at the 2026 ICML conference, is a method for extending long-range LLM agents through context folding. It employs a branch-return controller and context inheritance rules, and folded memory maintains plain text (text-only), but its state inheritance is modally symmetric, and the original image in the main context is propagated to each subsequent branch.

[0163] For the evaluation indicators in the table:

[0164] This invention uses accuracy (%) as the primary evaluation metric, which is the percentage of samples correctly determined by LLM-as-Judge. Considering the varying sizes of the benchmark samples, a weighted average accuracy (Avg.) weighted by the number of benchmark samples is used as the comprehensive metric.

[0165] Results and Tables:

[0166] To fully illustrate the technical effects of the present invention, experimental comparison results are provided in specific embodiments, wherein:

[0167] Table 1. Comparison of the main result accuracy (%) of this invention on five backbone models and seven benchmarks (rows of this invention are bolded, best values ​​in each column are bolded, and second-best values ​​are underlined):

[0168] Table 2. Comparison of weighted average accuracy (Avg.) (%) of the five methods on each backbone model (rows are bolded, optimal values ​​of each backbone are bolded, and suboptimal values ​​are underlined):

[0169] Table 3 shows a comparative ablation experiment of each core module of this invention (accuracy %, with the percentage in parentheses representing the decrease in accuracy compared to the complete MM-ContextFold), conducted on two backbones: Gemini-3-Flash and Qwen3.5-35B. The Image Caption variant (texturing the query image using initialization, with subtasks using only Text Retrieval / WebVisit and no longer viewing the original image in branches) showed the largest decrease, indicating that texturing cannot replace fine-grained visual details; the w / o Init variant (removing the initialization loop) showed the largest decrease on BC-VL, indicating that the initialization description is crucial for master state planning; the w / o Image in Branch variant (disabling the original image in branches and using only the initialization text result) degraded on fine-grained matching tasks; the w / Image in Main variant (injecting the original image into the master state and breaking the visual range constraint) decreased to near the ContextFold level, confirming the diminishing marginal benefit of retaining the original image after early visual perception.

[0170] Table 3:

[0171] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention in any other way. Any modifications or equivalent changes made based on the technical essence of the present invention shall still fall within the scope of protection claimed by the present invention.

Claims

1. A context management method for adaptive image loading in multimodal agent retrieval, which receives a natural language question and a set of query images, and the agent iteratively calls external tools to complete evidence collection and response, characterized in that... Specifically, the steps include the following: Step S1: Register the input image: Assign a stable identifier to each query image and build an image registry, while defining the toolset that the agent can call and the capabilities and input / output specifications of each tool; Step S2: Construct a two-state asymmetric context decoupling: The same agent can alternate between a main state and a branch state, and the main state and the branch state correspond to different contexts; the main state and the branch state are asymmetric in modality, and the main state is always a plain text state, corresponding to a continuously retained plain text context, without carrying the original image information. The branch state is a temporary multimodal state, corresponding to a temporary multimodal context; The branch state is invoked by the main state, runs on the basis of inheriting the main state context, and after it ends, its branch context is collapsed into a plain text summary, and the result is written back to the main state. At the same time, the original details of the branch are discarded and the main state is switched back. Step S3: Initialize the master state by injecting planning priors through image guidance: Before the regular branch-return loop, execute an image-guided initialization loop once to generate a concise text description for each query image and write it to the main state, so that the main state has image-related text priors without holding the original image. Step S4: Execute adaptive branching - return to the folded loop: The main state assesses whether the collected evidence is sufficient; if it is, the final answer is output and the process ends. Otherwise, the main state plans subtasks in the plain text context and explicitly assigns the subset of images to be loaded; the main state invokes a branch and switches states, the branch completes visual perception and tool retrieval within budget, then folds the obtained evidence into a plain text summary and writes it back to the main state, and then discards the branch context. This process is repeated until sufficient evidence is available, at which point the final answer is output from the main state.

2. The context management method for adaptive image loading for multimodal agent retrieval according to claim 1, characterized in that, The image registry constructed in step S1 is represented as follows. ； in, Represents the image registry. Indicates assignment to the first Zhang queries the image's stability identifier. Indicates the first Zhang query image, Indicates the total number of images queried; The agent relies on a stable set of identifiers Reference and assign images.

3. The context management method for adaptive image loading for multimodal agent retrieval according to claim 2, characterized in that, The external tools include a visual search tool, a text retrieval tool, and a webpage access tool; the visual search tool takes a query image or stable identifier as input and returns visually similar image results, along with the titles and source webpage links of each result; the text retrieval tool takes a text query as input and performs an open-domain network search, returning several relevant results, along with the titles, abstract fragments, and webpage links of each result; the webpage access tool takes a specified webpage link as input and returns the parsed text content of that webpage.

4. The image adaptive loading context management method for multimodal agent retrieval according to claim 3, characterized in that, The context of the main state is updated as follows: ； in, Indicates the first The plain text context maintained by the main state. This represents the updated main state context. Indicates the first The inference text generated by the main state. This indicates the branch action of this step. This represents a plain text evidence summary indicating that the branch has returned to the main state.

5. The context management method for adaptive image loading for multimodal agent retrieval according to claim 4, characterized in that, The image-guided initialization loop in step S3 specifically includes: Use initialization subtask Within the initialization branch, all original images are loaded and a concise text description is generated for each query image, resulting in a collection of descriptions. Write back to the main state; And based on the obtained description set Construct the initial main state context ; in, This relates to natural language issues. A set of stable identifiers representing all images. The inference text representing the initialization step. This indicates the initialization branch action. Represented as the first Zhang's query image generates a concise text description; The initial master state context does not contain any original image information.

6. The context management method for adaptive image loading for multimodal agent retrieval according to claim 5, characterized in that, In step S4, during each regular iteration of the adaptive branch-return fold loop, the main state is based on the plain text context maintained by the current main state. Select control action : ； Choose when sufficient evidence has been collected to answer the question. Output the final answer And end; Otherwise, plan subtasks to address the current information gaps. And assign a subset of image identifiers to be injected into the branch. ,and Allows for empty sets ; When the main state initiates a branch action, the branch state, inheriting the main state context, retrieves the corresponding original image from the image registry according to the assigned subset of image identifiers, and initializes its local context, that is: ， ； in, This indicates the initial local context of the branch. Indicates a subset based on image identifiers Retrieve and inject the original set of images into the branch as needed from the image registry. This represents the complete original query image that was injected into this branch.

7. The context management method for adaptive image loading for multimodal agent retrieval according to claim 6, characterized in that, This method also includes branch budgeting. With overall budget ; The branch budget Used to limit the maximum number of tool calls allowed within a single branch; the global budget Used to limit the maximum number of calls allowed throughout the entire retrieval process; Check the global budget before each master state decision. If the overall budget If all resources have been exhausted, then the main state must respond based on the existing evidence. Check the branch budget after each step of reasoning within the branch. If branch budget If the branch is exhausted and does not initiate a return action, then the branch is forced to summarize the collected findings and return with a return action.

8. The context management method for adaptive image loading for multimodal agent retrieval according to claim 7, characterized in that, The branch budget The global budget is 5; It is 20.

9. A system using the context management method for adaptive image loading for multimodal agent retrieval as described in any one of claims 1-8, wherein the system receives a natural language question and a set of query images, and the agent iteratively invokes external tools to complete evidence collection and response, characterized in that, The system includes a dual-state context decoupling module, an image-guided initialization module, and an adaptive branch-return folding inference module; The dual-state context decoupling module is used to assign a stable identifier to each query image and construct an image registry, and to enable the agent to switch between a plain text main state and a temporary multimodal branch state. The main state is always plain text, does not carry the original image, and only maintains the question, text evidence and high-level planning. The branch state loads the corresponding original image from the image registry for visual perception according to the assignment. The branch action initiated by the main state triggers the switch from the main state to the branch state. The return action initiated by the branch state folds the multimodal evidence collected by the branch into a plain text summary and writes it back to the main state, and then discards the local context of the branch. The original image is only retrieved as needed according to the identifier when it is loaded. The image guidance initialization module is used to execute an initialization loop before the regular branch-return loop, generate a concise text description for each query image and write it back to the main state, so that the plain text main state has image-related text priors without holding the original image. The adaptive branch-return folding reasoning module is used to evaluate whether the collected evidence is sufficient from the main state. If it is sufficient, the final answer is output; otherwise, a subset of image identifiers to be injected into the branch is explicitly assigned at the subtask granularity. The subset of image identifiers can be an empty set corresponding to a plain text subtask, so that the branch iterates through visual perception and tool retrieval within the branch budget and the whole within the global budget, and folds the obtained evidence into a plain text summary to write back to the main state, discarding the local context of the branch, until the evidence is sufficient to answer the question.

Images