Video understanding method, system, device, storage medium and program product

Abstract

The specification provides a video understanding method, system, device, storage medium and program product. The video understanding method comprises: acquiring execution data of a video reasoning task, the execution data comprising: a video, a user question for the video, a predicted reply, a preset true reply and a global tool calling track; the tools in the global tool calling track are selected from a tool set, the tool set comprising micro tools and pre-constructed macro tools; in a case where it is determined that the predicted reply is consistent with the true reply, identifying and removing, by a visual language model, a tool calling step that does not contribute to generating the predicted reply from the global tool calling track, with reference to the video and the user question, extracting a target tool calling sequence, and generating a new macro tool based on a calling sequence and a parameter dependency relationship of each micro tool in the target tool calling sequence; and performing availability verification on the new macro tool, and adding the new macro tool to the tool set in a case where the availability verification is passed.

Description

Technical Field

[0001] This specification relates to the field of artificial intelligence technology, and more particularly to a video understanding method, system, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the explosive growth of multimedia data, long-form video understanding has become an important research direction at the intersection of computer vision and natural language processing. It aims to perform deep semantic analysis on video content ranging from a few minutes to several hours in length to answer complex questions or extract key events.

[0003] Long-form video understanding is a fundamental challenge in the field of visual intelligence. Its challenges stem primarily from the temporal redundancy of video content, the sparse distribution of key information, and the instability of long-range inference. Early methods often adopted an end-to-end paradigm, directly inputting frame sequences into a visual language model for unified inference. However, limited by the aforementioned difficulties, these methods struggle to effectively capture long-range relational information, often resulting in poor inference performance.

[0004] To alleviate the aforementioned problems, agent-based tool enhancement methods are emerging. These methods utilize visual language models to decompose complex tasks into subtasks and invoke atomic-level visual tools, such as object detection, OCR, and scene segmentation, to acquire fine-grained visual evidence. Through iterative planning and execution, the agent gradually accumulates context to generate an answer.

[0005] However, this tool enhancement method still has significant shortcomings. First, its toolset is usually predefined and statically fixed, containing only basic operations. When faced with complex tasks, the agent has to repeatedly construct lengthy call chains consisting of multiple tools. This long sequence of operations not only increases computational overhead and response latency but also leads to a significant cumulative effect of errors in the multi-step inference process. Any small deviation in any single step can be amplified in subsequent steps, causing inference drift or hallucinations, severely affecting the accuracy and robustness of long video understanding. Summary of the Invention

[0006] In view of this, one or more embodiments of this specification provide video understanding methods, systems, electronic devices, computer-readable storage media, and computer program products that can automatically abstract macro tools from the global tool call trajectory of successful video inference tasks and encapsulate the call sequences of multiple micro tools into reusable compound operations. Optionally, constraints related to tools can also be extracted from failed video inference tasks to guide or limit the selection or parameter configuration of tools in subsequent tasks.

[0007] According to a first aspect of one or more embodiments of this specification, a video understanding method is proposed, the method comprising: The execution data of the video inference task is obtained, including: the video, the user question for the video, the predicted answer, the preset real answer, and the global tool call trajectory generated during the generation of the predicted answer; wherein, the tools recorded in the global tool call trajectory are selected from the currently maintained toolset, which includes: micro-tools for performing single visual perception operations, and macro-tools that have been built and are composed of multiple micro-tools to achieve specific visual functions; If the predicted response is consistent with the actual response, a visual language model is used to identify and eliminate tool call steps that do not contribute to the generation of the predicted response from the global tool call trajectory, based on the video and the user question. The target tool call sequence is extracted, and a new macro tool is generated based on the call order and parameter dependencies of each micro tool in the target tool call sequence. The new macro tool includes a function description, input and output interface specifications, and the execution logic of the internal micro tools. The new macro tool is validated for usability. If the validation is passed, the new macro tool is added to the toolset.

[0008] According to a second aspect of one or more embodiments of this specification, a video understanding method is proposed, the method comprising: The system receives a video reasoning task that includes video and user questions, obtains the currently accumulated context information and the functional descriptions of the tools in the currently available toolset, which includes: microtools for performing single visual perception operations, and macrotools that have been built and are composed of multiple microtools to achieve specific visual functions. Using a visual language model, joint reasoning is performed on the video, the context information, the user question, and the functional descriptions of the tools in the toolset to generate a subtask sequence for the current iteration round; wherein each subtask in the subtask sequence includes an execution instruction for the video and at least one tool called to execute the subtask; The subtask sequence is distributed to the execution agent for execution, and the execution results of each subtask are returned by the execution agent. The context information is updated based on each subtask and its execution result, and the tool call records of each subtask in this round are added to the global tool call trajectory. If the updated context information is insufficient to answer the user's question, return to the step of performing joint inference based on the updated context information; If the updated context information can answer the user question, the predicted response generated based on the updated context information, the user question, the video, and the global tool call trajectory are transmitted to the reflective agent so that the reflective agent can execute the method described in the first aspect.

[0009] According to a third aspect of the embodiments of this specification, a video understanding system is provided, including an orchestration agent, an execution agent, and a reflection agent; The orchestration agent is used to perform the method described in the second aspect; The execution agent is used to receive subtasks, call at least one tool specified in the subtask to process the video according to the execution instructions in the subtask, obtain the execution result and return it to the orchestration agent; The reflective agent is used to perform the method described in the first aspect of claim.

[0010] According to a fourth aspect of the embodiments of this specification, an electronic device is provided, comprising: processor; Memory used to store processor-executable instructions; Wherein, when the processor executes the executable instructions, it is used to implement the method described in the first aspect.

[0011] According to a fifth aspect of the embodiments of this specification, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps of the method described in the first aspect.

[0012] According to a sixth aspect of the embodiments of this specification, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect.

[0013] As demonstrated by the above embodiments, the reflexive agent can extract useful tool call sequences from successful global tool call trajectories and abstract them into macro tools. This allows it to encapsulate combinations of micro tools that would otherwise require multiple calls into a single, high-order macro tool. In subsequent tasks, the orchestration agent can directly call this macro tool to complete complex perception tasks, thereby significantly shortening the length of the tool call chain, reducing the number of inference steps per task, and mitigating the risk of accumulated errors caused by long sequence operations.

[0014] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this specification. Attached Figure Description

[0015] Figure 1 This is an exemplary embodiment of the architecture diagram of a video understanding system.

[0016] Figure 2 This is a flowchart of an exemplary embodiment of a video understanding method applied to an orchestration agent.

[0017] Figure 3 This is a flowchart of an exemplary embodiment of a video understanding method applied to a reflective agent.

[0018] Figure 4 This is a flowchart of another video understanding method applied to a reflective agent, provided as an exemplary embodiment.

[0019] Figure 5 This is a schematic diagram of the structure of a device provided in an exemplary embodiment. Detailed Implementation

[0020] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0021] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with relevant laws, regulations and standards, and corresponding operation entry points shall be provided for users to choose to authorize or refuse.

[0022] With the explosive growth of multimedia data, long-form video understanding has become an important research direction at the intersection of computer vision and natural language processing. Long-form video understanding aims to enable machines to perform deep semantic analysis on video content ranging from a few minutes to several hours in length, just like humans, to answer complex questions posed by users, generate content summaries, or retrieve key events. Unlike short-form video or single-frame image understanding, long-form video understanding is a fundamental challenge in the field of visual intelligence, facing the following core challenges: ①Time redundancy: Much of the content in a long video is redundant for a specific task, and key evidence may only appear for a few seconds in an hour-long video.

[0023] ②Sparse but critical events: Key information is scattered over a long period of time, requiring the model to have the ability to locate key information from massive amounts of data.

[0024] ③Long-term inference instability: Vision-Language Models (VLMs) are prone to error accumulation and inference drift during long-term inference, leading to performance degradation.

[0025] To address the aforementioned issues, an agent-based tool enhancement approach is proposed. This type of approach typically employs a "plan-execute" paradigm, utilizing a visual language model as a controller to decompose complex video understanding tasks into multiple sub-tasks and invoking external atomic-level visual tools to acquire fine-grained visual evidence. Through iterative planning and execution, the agent can gradually accumulate contextual information and ultimately generate an answer. This approach alleviates the problem of information overload in long videos to some extent and improves the interpretability of reasoning.

[0026] In the process of implementing the embodiments of this specification, the inventors discovered that the tool-enhanced intelligent agent method of the related technology still has the following major drawbacks in practical applications: First, the capabilities of tools are statically fixed. Toolsets are typically predefined and fixed, containing only basic atomic operations. When faced with different tasks, agents can only repeatedly invoke these basic tools and cannot automatically discover and encapsulate higher-order composite capabilities based on experience in task execution. This means that even if certain combinations of basic tools frequently appear and have stable effects in various scenarios, the system cannot solidify them into reusable new capabilities, resulting in the agent's capability boundaries always being limited and unable to grow naturally with increased usage.

[0027] Second, the inference path is lengthy, and long-range inference is unstable. Due to the lack of abstraction of higher-order capabilities, agents often need to construct extremely long chains of atomic tool calls when processing complex long video tasks. For example, to locate and identify the behavior of a specific person in a video, it may be necessary to sequentially call multiple independent tools such as shot segmentation, person detection, re-identification, and trajectory tracking. This lengthy serial or parallel calling not only increases computational overhead and latency, but more seriously, in the multi-step inference process, any small error in a single step may be amplified and accumulated in subsequent steps, leading to inference drift or the illusion of the final answer, seriously affecting the accuracy and robustness of long video understanding.

[0028] Third, experience cannot be effectively accumulated and reused. Agent frameworks are typically stateless or retain only simple textual memories. Each time a new video task arises, the agent needs to plan the tool call sequence from scratch, unable to utilize successful execution paths or lessons learned from past failures. Even if similar problems have been successfully solved before, the structured knowledge cannot be extracted to optimize the current decision-making process. This "starting from scratch" approach leads to significant redundant computation and trial-and-error costs, resulting in inefficiency in large-scale, long-video processing scenarios and hindering continuous performance optimization.

[0029] In summary, how to overcome the limitations of static toolsets, achieve dynamic growth of agent capabilities, and shorten inference paths to improve the stability and efficiency of long-range inference are the technical challenges that urgently need to be addressed in the field of long-video understanding.

[0030] Based on this, please refer to Figure 1 This specification provides a video understanding system, including an orchestration agent, an execution agent, and a reflective agent. Furthermore, the video understanding system also provides a toolset and a tool invocation experience base. The toolset includes microtools for performing single visual perception operations and pre-built macrotools composed of multiple microtools for implementing specific visual functions. The tool invocation experience base stores constraints extracted from failed video inference tasks to guide or limit tool selection or parameter configuration in subsequent tasks.

[0031] Through the collaborative work of the orchestration agent, the execution agent, and the reflection agent, macro tools can be automatically abstracted from the global tool call trajectory of successful video reasoning tasks, multiple micro tool call sequences can be encapsulated into reusable composite operations, and constraints on tools can be extracted from failed video reasoning tasks to avoid known errors. This overcomes the shortcomings of traditional solutions, such as static and fixed toolsets, inability to grow capabilities with experience, and inability to effectively reuse lessons learned from failures.

[0032] The orchestration agent is responsible for the planning process. It receives video and user questions, analyzes the information gap between the accumulated contextual information and the original question in each iteration, and dynamically decomposes complex tasks into multiple parallel subtasks. The orchestration agent also handles scheduling decisions, allocating appropriate tools to each subtask from the available toolset, and iteratively evaluates the sufficiency of contextual information. Finally, it synthesizes all contextual information to generate an answer, effectively shortening the path length of a single inference and suppressing error accumulation.

[0033] The executing agent performs subtasks issued by the orchestration agent. It translates the natural language instructions of the subtasks into actual calls to the underlying vision tools. The executing agent can operate in parallel, simultaneously calling from atomic-level micro-tools to complex macro-tools for different dimensions or segments of the same video, to obtain fine-grained visual evidence, including target coordinates, text sequences, and spatiotemporal trajectories.

[0034] Reflective agents are a key component for achieving training-free capability growth. They do not participate in real-time video reasoning task solving, but are triggered after a single task solving process to analyze the complete tool call trajectory. By comparing predicted answers with actual answers, reflective agents can abstract macro tools composed of sub-task sequences from successful cases, realizing the combination and encapsulation of skills; at the same time, they can also diagnose key error steps from failed cases and extract structured failure prior knowledge, achieving continuous self-improvement and dynamic capability growth.

[0035] The following sections, in conjunction with the accompanying diagrams, will provide a detailed explanation of the specific implementation processes and internal workflows of the orchestration agent, the execution agent, and the reflection agent.

[0036] 1. About orchestrating intelligent agents.

[0037] Please see Figure 2 This specification provides a video understanding system that can be executed by an orchestration agent. The orchestration agent is built based on a visual language model with frozen parameters. It should be noted that, in this embodiment, "parameters frozen" means that during the inference and planning process after deployment, the internal weight parameters of the visual language model are not fine-tuned or updated in any way; the adaptive evolution of the model only occurs at the toolset and experience base levels. The method includes: In S200, a video inference task including video and user questions is received, and the currently accumulated context information and the functional descriptions of the tools in the currently available toolset are obtained. The toolset includes: micro-tools for performing single visual perception operations, and macro-tools that have been built and are composed of multiple micro-tools to achieve specific visual functions.

[0038] Video reasoning tasks involve analyzing videos and user questions described in natural language. Videos can be visual data streams of any length and format. User questions can be inquiries about specific events in the video, requests for content summaries, or instructions to retrieve key plot points.

[0039] The currently accumulated context information refers to the collection of all visual evidence and information fragments related to answering the user's question that have been acquired and stored before or during this video inference task. Initially, this context information can be empty or it can include basic metadata pre-extracted from the video, such as total duration, resolution, and frame rate. As the inference iterations progress, the context information will be continuously expanded by new execution results.

[0040] The tool descriptions in the currently available toolset are textual descriptions of the capabilities of all tools that can be invoked at the current moment. This toolset is a dynamic collection that contains at least two levels of tools.

[0041] One type is microtools, which refer to atomic-level functional modules used to perform single visual perception operations, such as object detection tools, optical character recognition tools, lens boundary detection tools, and object tracking tools.

[0042] Secondly, there are macro tools, which are pre-built functional modules composed of multiple micro tools combined according to specific logic to achieve specific composite visual functions. For example, a macro tool that can "locate and recognize the text on an object held by a specified entity" may integrate the calling logic of multiple micro tools such as object detection, object tracking, keyframe extraction, and optical character recognition. At any point during system operation, the orchestration agent can obtain the natural language function descriptions, input parameter definitions, and return value formats of these tools to make reasonable tool selections during the planning phase.

[0043] In S202, a visual language model is used to perform joint reasoning on the video, contextual information, user questions, and functional descriptions of tools in the toolset to generate a sequence of subtasks for the current iteration round. Each subtask in the subtask sequence includes an execution instruction for the video and at least one tool called to execute the subtask.

[0044] In one possible implementation, the input to the visual language model includes: a sequence of keyframes obtained by sparse sampling of the video, currently accumulated contextual information, the user question, and functional description text of tools in the toolset. Based on its internal multimodal alignment and reasoning capabilities, the visual language model performs joint semantic analysis on this information, identifies the gap between the visual evidence contained in the current context and the information needed to answer the user question, and generates a sequence of subtasks to bridge this gap.

[0045] Each subtask in the generated subtask sequence comprises two components. The first is the execution instruction for the video, which can be a natural language description explicitly indicating what information needs to be extracted from the video, such as detecting a person wearing a blue shirt appearing between frames 120 and 180. The second is a specification of at least one tool to be invoked to execute the subtask. When specified as a micro-tool, it means the subtask will be completed by invoking a single atomic vision operation. When specified as a macro-tool, it means the subtask will be completed by invoking a pre-encapsulated composite operation, which internally schedules multiple micro-tools but appears externally as a single tool invocation. A subtask can specify the invocation of one tool, or it can specify the sequential or parallel invocation of multiple tools as needed.

[0046] By breaking down complex problems into multiple independently executable subtasks, this step transforms the inference process, which originally required building long-range serial call chains, into a sequence of subtasks executed in one or more rounds in parallel. This helps to shorten the inference path and reduce error accumulation.

[0047] In one alternative implementation, considering that the currently available toolset may contain a large number of tools as the reflective agent's capabilities continue to evolve, directly inputting all the functional descriptions of the tools in the toolset into the visual language model would not only consume a significant amount of context window resources but also potentially introduce tool information irrelevant to the current subtask, interfering with the model's subtask planning and tool selection. To improve planning efficiency and accuracy, a subset of candidate tools that is more relevant to the current inference requirements can be identified from the toolset before performing joint inference.

[0048] The orchestration agent determines the combined semantic vector of the user question and contextual information, as well as the semantic vectors of the functional descriptions of each tool in the toolset. For example, the extraction of these semantic vectors can be performed using a pre-trained text encoder. This text encoder can be a text encoding module that shares underlying parameters with the visual language model, or it can be a separately deployed lightweight semantic representation model. For generating the combined semantic vector, the natural language text of the user question can be concatenated with the text descriptions of each acquired information entry in the currently accumulated contextual information. The concatenated text sequence is then input into the text encoder to obtain a combined semantic vector that integrates the task objective and existing evidence. For the functional descriptions of each tool in the toolset, their corresponding semantic vectors can be extracted in advance or online using the same text encoder, and a tool semantic index can be constructed.

[0049] Next, the orchestration agent determines the similarity between the combined semantic vector and the semantic vectors describing the functions of each tool. Similarity calculation can employ cosine similarity, Euclidean distance, or other vector space distance metrics. By comparing the combined semantic vector with the semantic vector of each tool in the toolset one by one, a similarity score list can be obtained. This similarity score reflects the degree of semantic matching between the visual perception capabilities provided by each tool and the current information gap.

[0050] The orchestration agent sorts the tools in descending order of similarity, selecting the N tools with the highest similarity to form a subset of candidate tools, where N is a positive integer. The value of N can be preset based on the context window capacity of the visual language model, the total size of the toolset, and the system's trade-off between recall and precision. Alternatively, it can be dynamically determined based on similarity distribution characteristics, such as selecting all tools with similarity exceeding a preset threshold. In one optional implementation, when N is 1, it is equivalent to matching only the most relevant tool for the current subtask; when N is greater than 1, it retains some flexibility in tool selection for subsequent subtask planning.

[0051] After determining the candidate tool subset using the above method, the information input to the visual language model is adjusted accordingly to include: the keyframe sequence of the video, contextual information, the user question, and functional descriptions of each tool in the candidate tool subset. Since the candidate tool subset has filtered out tools with low relevance to the current inference needs, semantic noise is significantly reduced. The visual language model can then focus more on evaluating a limited number of highly relevant tools when planning subtasks, thereby improving the accuracy of tool selection and the rationality of subtask generation. Simultaneously, the reduction in input information also lowers the computational cost and latency of model inference.

[0052] In S204, the subtask sequence is distributed to the executing agent for execution, and the execution results of each subtask are received from the executing agent.

[0053] After generating the sequence of subtasks for this round, the orchestration agent distributes each subtask in the sequence to one or more execution agents through a communication interface. In one optional implementation, an execution agent can receive and execute one subtask, and multiple execution agents can form an execution cluster to process different segments or different visual dimensions of the same video in parallel.

[0054] Upon receiving a subtask, the executing agent, according to the execution instructions within the subtask, invokes at least one tool specified in the subtask to perform actual computational operations on the video data. This includes invoking microtools to perform atomic sensing tasks or invoking macrotools to execute their internally defined microtool call chains. After execution, the executing agent returns the execution results of each subtask to the orchestration agent in a structured data format. The execution results may include a type identifier, specific data content (e.g., target bounding box coordinates, recognized text strings, tracking trajectory coordinate sequences, timestamp intervals, etc., but not limited to these), and a confidence score for the result. These structured execution results provide a standardized data foundation for subsequent context updates and trajectory recording.

[0055] In S206, the context information is updated based on each subtask and its execution result, and the tool call records of each subtask in this round are added to the global tool call track.

[0056] After receiving the execution results of all subtasks in the current round, the orchestration agent performs two information maintenance operations.

[0057] The first operation is updating the context information. The orchestration agent associates the execution instructions of each subtask in the current round with their corresponding execution results, forming structured information entries, and appends them to the currently accumulated context information to create the updated context information. For example, the context information can be maintained as a list of records, where each list element contains a pair of "subtask instruction-execution result" and the timestamp or iteration round identifier that generated the pair. Through this incremental context update mechanism, the visual evidence needed to answer user questions can be accumulated in an orderly manner, avoiding information omissions or duplicate acquisitions.

[0058] The second operation is to add the tool call records of each subtask in this round to the global tool call trajectory. The tool call trajectory is a complete record that runs through the entire video inference task solving process, used to record which tools the system called in each iteration, with what parameters, and what results were obtained.

[0059] For example, the tool call record for this round may include information such as the round identifier, the execution instructions for each subtask, the specified tool identifier, the tool execution status (success or failure), and a summary of the returned results. This global tool call trajectory will serve as the basis for capability evolution analysis of the reflexive agent after the inference task is completed.

[0060] In S208, if the updated context information is insufficient to answer the user's question, the step of performing joint inference is returned based on the updated context information.

[0061] After updating the context, the orchestration agent invokes the visual language model to re-evaluate the relationship between the updated context information and the user's original question. For example, the evaluation criteria may include: whether the updated context covers all information categories required to answer the question, and whether the certainty of the existing information meets the confidence criteria required for the answer.

[0062] When the evaluation result indicates that the updated context information is still insufficient to support an accurate answer to the user's question, the orchestration agent does not generate an answer but returns to step S202 to begin a new round of joint reasoning and subtask planning based on the updated context information. This allows the orchestration agent to continuously supplement missing key information from the video through iterative exploration.

[0063] In one alternative implementation, the video inference system maintains a tool invocation experience base. This base stores structured failure prior knowledge extracted by the reflexive agent during the evolutionary phase for historically failed tasks. Each piece of failure prior knowledge may include the following information: the identifier of the tool involved, the conditions or scenarios under which the tool failed, the specific consequences of the failure, and suggested constraints or usage precautions to avoid similar errors.

[0064] Considering the availability of the aforementioned experience base, the orchestration agent can proactively utilize historical failure experiences to avoid known tool usage pitfalls and improve the robustness of reasoning when performing tool selection and subtask planning.

[0065] For a tool in the toolset, the orchestration agent can retrieve constraints matching that tool from the tool invocation experience base. This matching can be precise based on the tool identifier or fuzzy based on the semantic similarity between the tool's functional description and the scenario descriptions in the failure priors. In one implementation, a text encoder similar to that used in the aforementioned candidate tool subset determination step can be employed to extract the functional description vector of the target tool and the semantic vectors of the applicable scenario descriptions in each failure prior in the experience base. The cosine similarity is then calculated to determine whether there are historical invocation constraints with a matching degree exceeding a preset threshold.

[0066] If a matching constraint is found, the orchestration agent appends the obtained constraint text to the original functional description of the corresponding tool to form an enhanced functional description. For example, suppose the original functional description of an optical character recognition tool is "to recognize visible text regions in a specified video frame and return the text content," and the constraint matching this tool in the experience base is "when the resolution of the text region in the video frame is lower than a preset threshold or there is motion blur, the recognition result of this tool may be an empty string or a low-confidence random string, and its recognition result should not be used as the sole factual basis," then the enhanced functional description will append this constraint information to the original functional description. In subsequent joint inference steps, the functional description input to the visual language model is this enhanced functional description.

[0067] If no matching historical call constraints are found, it indicates that there are no known failure modes for this tool in the experience base. In this case, the functional description of the input visual language model will still be the original functional description of the tool.

[0068] Through the aforementioned mechanism, when planning sub-tasks, the orchestration agent can not only understand the functionality of a tool but also anticipate under what circumstances the tool might fail and how to mitigate its limitations. This enables the visual language model to more rationally assess the capability boundaries of tools during tool selection and sub-task instruction generation, avoiding the planning of tool invocation schemes that are infeasible under known conditions. This reduces inference failures caused by repeating known errors, improving the stability of long-range inference and the accuracy of the final answer. Simultaneously, prior knowledge of failures is passed to the model by concatenating it into the tool description, resulting in low implementation costs, requiring no changes to the model structure or additional training, and maintaining the system's training-free evolutionary characteristics.

[0069] In one alternative implementation, to prevent the system from getting stuck in an infinite loop due to insufficient information, a maximum iteration threshold can be preset. When the number of iterations reaches this threshold, even if the context information is still insufficient, the system will be forced to terminate, providing the currently available answer based on the limited context information, and may include a low-confidence flag to indicate that the answer is uncertain.

[0070] In S210, if the updated context information can answer the user's question, the predicted answer, user question, video, and global tool call trajectory generated based on the updated context information are transmitted to the reflective agent.

[0071] When the orchestration agent determines that the updated contextual information is sufficient to answer the user's question, it enters the final stage of the reasoning task. The orchestration agent invokes a visual language model, and given the user's question and all accumulated contextual information, integrates all visual evidence for semantic integration and reasoning to generate a predicted response. This predicted response can be a natural language answer text, a summary that meets the user's requirements, or a description of the search results.

[0072] Simultaneously, the orchestration agent packages the complete data for this video inference task and transmits it to the reflexive agent to trigger subsequent capability evolution processes. The transmitted data package includes: the user's original question, video data, generated predicted responses, and a global tool call trajectory recording the complete inference process for this round.

[0073] By decoupling task execution from capability evolution, the orchestration agent can focus on efficient online reasoning, while delegating computationally expensive tasks such as trajectory analysis and tool abstraction to a reflective agent to complete them offline or asynchronously. This architectural design enables the video understanding system to ensure real-time responsiveness while possessing a mechanism to continuously learn from task experience and improve its capabilities.

[0074] For example, the solution loop process of an orchestration agent includes an initialization phase and an iterative loop phase.

[0075] During initialization: Input: Long video User issues State: Empty context Initial toolset Global tool call trajectory (Currently an empty trajectory), iteration rounds Maximum number of iterations .

[0076] In the iterative loop (when) )stage: Step 1, Task Decomposition and Planning: Analysis and The gap, generating the first Wheel task sequence . The subscript t indicates the current iteration round. This represents the natural language execution instruction for the j-th subtask, such as "detect all people in the range of frames 50 to 100 in the video". This indicates the identifier of at least one tool that needs to be invoked to execute the subtask j. This tool identifier can point to a micro tool or a macro tool.

[0077] Step 2, Parallel Execution: Each executor executes according to... Invoke the specified tool Process video and obtain results .

[0078] Step 3, Context Update and Track Recording: The orchestration agent constructs a structured round record unit from all tool call details of this iteration. And append this unit to the global tool call trajectory. Within this unit, the execution instructions for each subtask in this round are contained. The tool identifier being called The passed parameters (if any) and the returned execution result Subsequently, iterative updates are performed: .

[0079] Step 4, terminate the judgment: Scenario 1: Sufficient information → Generate answer In this case, the current solution loop ends. The orchestration agent will then process the complete data packet for this video inference task, including the video v, the user question q, and the generated predicted response. And a complete global tool call history This information is then transmitted to the reflecting agent to trigger subsequent evolutionary stages of processing.

[0080] Regarding the acquisition and transmission of the preset true answer, in one optional implementation, the orchestration agent can also be configured to acquire the preset true answer y corresponding to the video reasoning task, such as a manually annotated standard answer. This preset true answer y can be provided by an external knowledge base or dataset interface along with the video v and user question q during task creation. When outputting the data required for the evolution phase, the orchestration agent will include the preset true answer y along with the video v, user question q, predicted answer y', and global tool call trajectory. Together, they are transmitted to the reflecting agent. In another alternative implementation, the orchestrating agent only transmits the video v, the user question q, the predicted response y', and the trajectory. The preset true response y is obtained by the reflective agent through an independent data retrieval channel based on the task identifier. Regardless of the transmission path used, the role of the preset true response y in the evolutionary stage is to provide the reflective agent with a true basis for judging the success or failure of the task and for conducting error attribution analysis.

[0081] Scenario 2: Insufficient information but not exceeded limit → Return to step 1.

[0082] Case 3: Reaching the maximum iteration limit → forced termination, generating a low-confidence answer based on the current context.

[0083] Through the aforementioned cyclical mechanism, the orchestration agent can systematically extract key visual evidence from long videos in an iterative manner, generate predictive responses, and provide complete reasoning trajectory data for the evolution of the reflective agent's capabilities.

[0084] 2. Regarding the execution of intelligent agents.

[0085] The execution agent, as a lightweight cluster of execution units, consists of a tool scheduling engine and a logic encapsulation layer. Each execution agent instance receives a single subtask instruction and a specified tool from the orchestration agent. It first extracts the operation object and parameters through an instruction parser, and then assigns an execution path based on the tool type. If a micro-tool is specified, the execution agent directly calls the corresponding atomic vision processing interface, such as a lens segmentation module, object detector, or OCR engine, and performs actual calculations on the video data. If a macro-tool is specified, the execution agent loads the internal workflow definition of the macro-tool and automatically schedules the underlying micro-tools to execute step-by-step according to the predefined micro-tool call chain, appearing externally as a single-step call. The execution agent standardizes and encapsulates the execution result of each subtask, returning a structured result object containing the result type, data content (such as border coordinates, text strings, timestamp ranges, etc.), and confidence level, and then sends it back to the orchestration agent.

[0086] 3. On reflective agents.

[0087] Please see Figure 3 This specification also provides a video understanding method for a reflective agent, which can be implemented based on a visual language model with frozen parameters. The method includes: In S300, the execution data of the video inference task is acquired. The execution data includes: video, user questions for the video, predicted answers, preset real answers, and global tool call trajectories generated during the generation of predicted answers. The tools recorded in the global tool call trajectories are selected from the currently maintained toolset, which includes: microtools for performing single visual perception operations, and macrotools that have been built and are composed of multiple microtools to achieve specific visual functions.

[0088] The preset true response serves as a baseline label for determining the success of the inference task. In one optional implementation, this true response can be provided by the task dataset, transmitted to the system along with the video and user question during task creation, and included in the execution data packet by the orchestration agent when outputting the execution result. In another optional implementation, the orchestration agent only transmits the video, user question, predicted response, and trajectory, while the reflective agent can independently retrieve the corresponding true response from an external knowledge base using a separate data retrieval channel based on the task identifier.

[0089] The global tool call trajectory records, in chronological order, the tools invoked and their execution details for each round and subtask during this solution process. The tools recorded in this trajectory are all selected from the toolset currently maintained by the video understanding system. As mentioned earlier, this toolset is a dynamic collection containing micro-tools for performing single visual perception operations, as well as macro-tools composed of multiple micro-tools to achieve specific visual functions. The data structure of the global tool call trajectory is designed in a format that can be precisely parsed by the reflexive agent, where each tool call step includes fields such as tool identifier, input parameters, execution status, and return result.

[0090] In S302, if the predicted response is consistent with the actual response, the visual language model, referring to the video and user questions, identifies and eliminates tool call steps that do not contribute to the generation of the predicted response from the global tool call trajectory, extracts the target tool call sequence, and generates a new macro tool based on the call order and parameter dependencies of each micro tool in the target tool call sequence. The new macro tool includes a function description, input and output interface specifications, and the execution logic of the internal micro tools.

[0091] This step is triggered only when the reflective agent determines that the video reasoning task has been successfully executed. The criterion for successful task execution is that the predicted response and the preset true response are semantically consistent. This consistency determination can be performed by the reflective agent calling a visual language model for semantic comparison, or it can be achieved through automatic comparison based on text similarity metrics.

[0092] Once the task execution is confirmed to be successful, the reflective agent initiates the macro tool abstraction process. The reflective agent calls the visual language model, inputting the user question, the predicted response, the keyframe sequence of the video, and the complete global tool call trajectory. It then guides the model to perform the following analysis task: among all the tool call steps recorded in the global tool call trajectory, based on the degree of causal correlation between the execution result of each tool call step and the final predicted response, it identifies and eliminates those tool call steps that do not contribute to generating the predicted response, retaining only the minimal target tool call sequence that is sufficient to support the derivation of the predicted response from the video information. Furthermore, based on the call order and parameter dependencies of each micro-tool in the target tool call sequence, it generates a new macro tool.

[0093] The generated new macro tool comprises the following components: a functional description, which describes the composite visual functions that the macro tool can achieve in natural language; an input / output interface specification, which clarifies the input parameter definitions of the macro tool (e.g., including parameter names, types, and semantic descriptions) and the format mode of the output results; and the execution logic of the internal microtools, which defines the scheduling order of the internal microtools, the parameter passing relationship between each step, and the conditional branching logic when the macro tool is invoked.

[0094] For example, a new macro tool's function description could be "track a specified entity and read the text on its handheld object." Its internal micro-tool execution logic could sequentially include: target detection and location of the specified entity, target tracking to obtain the entity's motion trajectory, keyframe extraction to locate the preferred view frame where the entity appears, and optical character recognition to read the text in the entity's handheld object region within that frame.

[0095] In one possible implementation, the process of generating a new macro tool through a visual language model in step S302 may include the following two sub-stages.

[0096] The first sub-stage is trajectory cleaning and critical path extraction. The reflective agent invokes the visual language model, inputting the user question, the predicted response, a representative frame sequence of the video, and the complete global tool call trajectory. The model is then guided to perform the following analysis task: Among all the tool call steps recorded in the global tool call trajectory, it distinguishes which steps constitute the necessary reasoning chain for successfully deriving the final predicted response, and which steps are exploratory attempts, redundant verifications, or bypass operations that do not substantially contribute to the final answer. Based on the causal relationship between the execution results of each tool call step and the final predicted response, the visual language model identifies and eliminates tool call steps that do not contribute to generating the predicted response, retaining only the minimal core tool call step sequence sufficient to support the derivation of the predicted response from the video information, and uses this as the target tool call sequence. In an optional implementation, this target tool call sequence may also include the call order of each micro-tool and the parameter dependencies between steps, i.e., how the output of the previous step serves as the input of the next step.

[0097] The second sub-stage is pattern generalization and macro-tool definition generation. The reflective agent again invokes the visual language model or pure language model, inputting the target tool invocation sequence and corresponding sub-task instructions. This guides the model to abstract the specific instances represented by the sequence into a widely reusable general pattern. For example, the model is prompted to replace specific entity names involved in the target tool invocation sequence with configurable variable parameters, such as replacing "the person wearing a red shirt" with "target entity description," and replacing specific time intervals with general time range parameters, thereby transforming the target tool invocation sequence into a general macro-tool definition structure.

[0098] In another possible implementation, the process of generating new macro tools through a visual language model in step S302 can be achieved using a guided generation method based on multimodal prompts.

[0099] The reflective agent constructs multimodal cue words. These cue words are used to convey all the context and instructions required for the macro tool generation task to the visual language model in one go. The multimodal cue words consist of: a sequence of keyframes from the video, a user question in text form, a predicted response in text form, a global tool call trajectory in text form, and macro tool generation instructions.

[0100] Among them, the keyframe sequence of the video is a set of representative frame images extracted from the original video. It is used to provide visual reference for the visual language model, so that when the analysis tool calls the trajectory, it can judge the rationality and necessity of each step in combination with the actual visual content.

[0101] The user question and the predicted response provide the target semantics and the final inference conclusion for this inference task. By inputting these along with the tool call trajectory, the visual language model can evaluate each step in the trajectory from the perspective of "what necessary operations need to be performed to derive the answer from the question".

[0102] The global tool call trajectory is represented by structured text, which transforms the tool call records, originally stored as structured data, into a text sequence suitable for input to a visual language model. For example, each step in the trajectory can be represented in text format as "round number - subtask number: instruction description | tool called: tool name | execution result: result summary".

[0103] A macro tool generation instruction is a predefined set of instructions that explicitly directs the visual language model to perform macro tool generation tasks according to a specified sequence of steps. This instruction, in the form of natural language or structured prompts, specifies the analysis, filtering, verification, abstraction, and generation steps that the visual language model must complete sequentially.

[0104] The reflective agent inputs multimodal cue words into the visual language model, which then performs the following processing sequentially according to the steps specified by the macro tool's generation instructions: The first step involves combining the semantic intent of the user's question with visual evidence from the keyframe sequence to score the relevance of each tool invocation step in the global tool invocation trajectory. The visual language model's task in this step is to evaluate, one by one, whether each tool invocation step recorded in the trajectory is substantially related to the final answer to the user's question. The evaluation criteria include: whether the visual information acquired in this step is directly cited or used as a basis for reasoning in the predicted response; whether the object of operation in this step matches the key semantic entities in the user's question; and whether the execution result of this step is consumed in subsequent steps and generates meaningful progress. Based on this multi-dimensional comprehensive judgment, the visual language model outputs a relevance score for each tool invocation step. This score can be a discrete gradation system (e.g., high, medium, low) or a continuous numerical system (e.g., a value between 0 and 1).

[0105] The second step involves identifying and eliminating tool call steps with relevance scores below a preset threshold, resulting in a candidate step set. The preset score threshold can be set based on experimental experience; its purpose is to filter out exploratory calls, redundant confirmation calls, or purely erroneous attempts that do not contribute to the final answer. In a specific implementation, if the relevance score uses a numerical range of 0 to 1, the threshold can be set to 0.5; steps below this threshold are considered non-contributing and eliminated. The steps retained after this filtering step constitute the candidate step set.

[0106] The third step involves analyzing the data dependencies and temporal logic among the tool invocation steps in the candidate step set to verify the existence of causal chains between input and output. In this step, the visual language model does not view each step in isolation, but rather treats the candidate step set as a holistic sequence of operations, examining the information flow relationships between steps. For example, the model determines whether the output of the previous step constitutes the input parameter or execution prerequisite for the next step. For instance, is the bounding box coordinate output by an object detection step used as the initial tracking box parameter in a subsequent object tracking step? Is the timestamp interval output by a shot segmentation step used as the sampling range in a subsequent frame extraction step? Through this analysis, the data flow and causal dependencies between steps are identified.

[0107] The fourth step involves removing isolated steps based on the verification results, retaining the sequence of steps that constitute the causal chain between input and output, forming the target tool invocation sequence. In the causal analysis described above, if some steps in the candidate step set neither depend on the output of other steps nor are their own outputs used by other steps, these steps are considered isolated steps. Although isolated steps may score well in relevance scoring, their lack of integration into the overall causal network indicates limited actual contribution to the final result or that they are redundant. After removing isolated steps, the retained sequence of steps constitutes the target tool invocation sequence, forming a complete causal chain from the original video information to the final answer. Each step in this sequence is a necessary node in the causal chain.

[0108] The fifth step is to variableize the operation objects in the target tool call sequence. To ensure that the macro tool abstracted from this specific task instance can be reused in future tasks, the specific information bound to the particular instance in the sequence needs to be replaced with configurable general parameters.

[0109] Variableization includes: First, abstracting visual entity descriptions into generic type parameters. For example, replacing the specific entity description "middle-aged man in a red shirt" in the step instructions with a string-type input parameter named "target entity description". Second, abstracting numerical parameters into dynamic configuration items. For example, replacing the specific frame range specified in the step, "frames 120 to 180", with an interval-type input parameter named "processing time interval", or replacing the specific coordinates of the OCR recognition area with an optional configuration item named "region of interest". Through variableization, the target tool call sequence is transformed from a specific instance that can only be used in a single specific scenario into a generic operation template that can be used for different instances of the same problem.

[0110] Step 6: Based on the variableized target tool call sequence, generate a new macro tool according to a preset macro tool template. The preset macro tool template defines the standard data structure of the macro tool, which may include the following fields: tool name, function description, input parameter mode, output result mode, and internal micro-tool execution logic. The visual language model maps each step in the variableized target tool call sequence to a call entry in the internal micro-tool execution logic, extracts input and output information from the sequence to populate the input parameter mode and output result mode fields, and automatically generates a concise function description text and tool name based on the overall function achieved by the step sequence.

[0111] Through the step-by-step generation method guided by the aforementioned multimodal prompts, the reflective agent can stably and interpretably extract structured macro-tool definitions from successful execution trajectories, providing a reliable technical path for the continuous evolution of the toolset's capabilities.

[0112] In S304, a usability check is performed on the new macro tool. If the usability check is passed, the new macro tool is added to the toolset.

[0113] Before formally incorporating the macro tools generated in step S302 into the toolset, their usability needs to be verified to ensure the quality of the newly generated macro tools and to prevent the toolset from becoming redundant or bloated.

[0114] In one possible implementation, availability verification includes functional redundancy detection. This functional redundancy detection aims to avoid multiple macro tools with highly overlapping functions within the toolset, thereby maintaining the toolset's simplicity and maintainability.

[0115] Reflect on the semantic similarity between the functional description of the newly generated macro tool computed by the intelligent agent and the functional description of each existing macro tool in the toolset. A preset similarity threshold can be set based on experimental experience, for example, 0.85 or 0.90, to define whether the two functional descriptions are highly similar semantically.

[0116] If any of the calculated semantic similarities is greater than or equal to a preset similarity threshold, it indicates that there is at least one existing macro tool in the toolset that is highly similar to the new macro tool in terms of functional semantics. In this case, the reflective agent does not directly discard the new macro tool, but performs further deep analysis to determine whether the new macro tool has substantial improvement value compared to the highly similar existing macro tool.

[0117] The in-depth analysis comprises two dimensions. The first dimension is the analysis of parameter space coverage. The reflective agent compares the input parameter definitions and output formats of the new macro tool with those of the target existing macro tool one by one. The comparison of input parameters includes the number of parameters, the type of each parameter, its semantics, and whether it is optional. The comparison of output results includes the completeness, granularity, and data type of the output fields. If the set of input parameters of the new macro tool completely covers the set of input parameters of the target existing macro tool—that is, if the new macro tool can receive every parameter configuration that the target existing macro tool can receive and provide equivalent or better processing—and the output results of the new macro tool are no less complete or granular than those of the target existing macro tool, then the new macro tool is determined to cover the target existing macro tool in parameter space.

[0118] The second dimension is the analysis of logical inclusion relationships in the internal micro-tool call chains. The reflective agent compares the execution logic of the custom internal micro-tools of the two macro tools. Logical inclusion refers to whether the internal micro-tool call chains of the new macro tool constitute a superset of the internal micro-tool call chains of the target macro tool in terms of functional semantics. For example, if the internal logic of the target macro tool is "target detection—target tracking," while the internal logic of the new macro tool is "target detection—target tracking—optical character recognition," and the new macro tool provides additional character recognition capabilities on top of completing all the functions of the target macro tool, then it is determined that the new macro tool logically includes the target macro tool.

[0119] Based on the analysis of the above two dimensions, the reflective agent further evaluates whether the performance metrics of the new macro tool are superior to those of the target existing macro tool. Performance metrics may include, but are not limited to: average execution time, resource consumption of tool calls, and confidence level of output results. These performance metrics can be obtained by conducting isolated tests on the two macro tools on a test sample set, or by extracting evidence from the successful execution trajectory of the new macro tool and comparing it with the historical execution statistics of the target existing macro tool.

[0120] If the comprehensive evaluation results show that the new macro tool can cover all parameters of the existing target macro tool, and at least one performance metric of the new macro tool is superior to the corresponding performance metric of the existing target macro tool, then the new macro tool is determined to be an upgraded version of the existing target macro tool. The reflexive agent replaces the existing target macro tool with the new macro tool, that is, it removes the existing target macro tool from the toolset and adds the new macro tool to the toolset. This replacement strategy allows the toolset to self-optimize during evolution through a process of survival of the fittest, maintaining the functional representativeness and execution efficiency of each macro tool in the toolset.

[0121] Conversely, if a new macro tool, while highly similar in function and semantics to an existing target macro tool, fails to cover all parameters of the existing target macro tool, or if its performance metrics are not superior to those of the existing target macro tool, then the new macro tool is not worth replacing. In this case, the reflexive agent discards the new macro tool, and the toolset remains unchanged. This conservative update strategy prevents the introduction of homogeneous but stagnant or even degraded tool variants into the toolset, thus avoiding the disorderly expansion of the toolset.

[0122] If the semantic similarity between the functional description of the new macro tool and the functional descriptions of all existing macro tools in the toolset is less than a preset similarity threshold, meaning there are no existing tools with highly similar functions, then the new macro tool contributes a new, previously unexplored composite capability to the toolset. In this case, the reflective agent determines that the new macro tool passes the usability check and directly adds it to the toolset. The toolset thus completes a functional boundary expansion.

[0123] In another possible implementation, usability verification may also include confirmatory testing. The reflexive agent may select a set of verification videos from a historical video library that are similar to the video scenes in the video reasoning task; using a new macro tool, perform simulated reasoning for each verification video in the set; calculate the success rate and average execution time of the simulated reasoning; and determine that the usability verification is passed only if the success rate is higher than a preset success threshold and the average execution time is lower than a preset execution time threshold.

[0124] After passing the availability check, the reflective agent adds the new macro tool to the currently maintained toolset of the video understanding system. This toolset is then updated, and subsequently, during video inference tasks, the orchestration agent can obtain the functional description of the expanded toolset, including the newly added macro tool, in step S200, and schedule it as an available tool option during the subtask planning phase. Thus, the system achieves continuous accumulation and dynamic growth of tool capabilities.

[0125] In some embodiments, to further improve the stability of macro tool evolution and avoid introducing new tools with unstable quality into the toolset due to the randomness of a single successful sample, a buffer and confirmation mechanism for a pool of candidate macro tools can be introduced before performing formal usability verification on newly generated macro tools.

[0126] The video understanding system maintains a pool of candidate macro tools within the reflexive agent or in a storage area accessible to the reflexive agent. This pool is used to temporarily store macro tool definitions that have been abstracted and generated in a single successful inference task but have not yet undergone stability verification across tasks. Each entry in the pool may contain the macro tool's definition structure, information about the most recently generated version, and a frequency counter.

[0127] After the reflective agent generates a new macro tool through the aforementioned step S302, it does not directly perform availability verification on the new macro tool, but first calculates its matching degree with each existing macro tool in the candidate macro tool pool.

[0128] The matching degree can be determined based on at least one of the following three dimensions. The first dimension is the semantic similarity of the functional descriptions. This can be achieved by extracting the semantic embedding vectors of the functional description text of the new macro tool and the functional description texts of existing macro tools in the candidate pool using a pre-trained text semantic encoder, and then calculating the cosine similarity between the two. The second dimension is the consistency of the input and output parameter spaces. This can be determined by comparing the degree of overlap between the two macro tools in terms of the number, name, data type, and semantic constraints of input parameters, as well as the field structure, data type, and semantic meaning of the output results. The third dimension is the consistency of the internal micro-tool execution logic. This can be determined by comparing the consistency of the types of micro-tools, the calling order, the parameter passing relationships, and the conditional branching structures in the micro-tool call chains defined within the two macro tools.

[0129] The matching score can be a comprehensive score derived from a weighted fusion of one or more of the above dimensions. In one implementation, the weight coefficients of each dimension can be preset, and the weighted sum can be calculated as the final matching score.

[0130] A preset matching threshold is used to determine whether two macro tools belong to the same category. If an existing macro tool has a matching degree higher than the preset matching threshold, it indicates that the newly generated macro tool is highly consistent with the existing macro tool in the candidate pool in terms of functional semantics, parameter interface, and execution logic, and the two belong to the same category of macro tool patterns. In this case, the reflexive agent does not add a new entry to the candidate pool, but instead increments the occurrence frequency counter of the existing macro tool. The incrementing of the occurrence frequency means that the same macro tool pattern has been repeatedly discovered and verified in multiple independent successful inference tasks, representing a stable, cross-task generalizable composite visual operation pattern.

[0131] When the cumulative frequency of occurrence reaches a preset stable threshold, it indicates that the macro tool pattern has been successfully applied and independently generated in a sufficient number of different task instances, and its effectiveness and versatility have been fully verified. At this point, the reflective agent triggers a formal usability verification process using the most recently generated version of the macro tool as the verification object. Using the most recently generated version as the verification object ensures that the macro tools that ultimately enter the toolset reflect the optimization details in the most recent successful task. In another implementation, when the frequency reaches a stable threshold, the version with better performance metrics can be selected as the verification object from the versions that have appeared multiple times.

[0132] If the number of macro tools has not accumulated to a stable threshold, they will remain temporarily in the candidate pool and will not enter the formal availability verification and toolset update process.

[0133] If the matching degree between the new macro tool and all existing macro tools in the candidate macro tool pool is not higher than the preset matching threshold, that is, there are no existing macro tools belonging to the same category, it indicates that the operation mode encapsulated by the new macro tool has not been recorded. In this case, the reflexive agent adds the new macro tool as a new category entry to the candidate macro tool pool and initializes its occurrence frequency counter to 1.

[0134] Through the cumulative confirmation mechanism of the aforementioned candidate macro tool pool, the video understanding system can effectively distinguish between "non-generalizable tool combinations resulting from accidental success in a single task" and "stable tool combinations repeatedly validated in multiple different tasks." This reduces the risk of unstable macro tools due to sample bias being prematurely introduced into the formal toolset, ensuring the functional quality and reliability of the toolset during its continuous evolution.

[0135] In some embodiments, please refer to Figure 4 It also provides a branch for reflexive agents to extract and accumulate experience from failed tasks during the evolutionary stage. Figure 3 The successful path processing flow is shown in parallel.

[0136] In S400, when the reflexive agent determines that the predicted response is inconsistent with the actual response, it performs attribution analysis on the global tool call trajectory through a visual language model, locates the tool call step that caused the incorrect predicted response, and extracts the context corresponding to the tool call step in order to generate constraints for the tools used in the tool call step.

[0137] After the reflective agent compares the predicted response with the preset real response, if the comparison results show that there is a substantial deviation between the two in terms of key facts, logical conclusions or information completeness, the video reasoning task is determined to have failed, and the failure attribution analysis process is triggered.

[0138] Failure attribution analysis is performed by the reflective agent invoking its internal parameter-frozen visual language model. The visual language model's input includes: the user question, the actual response, the predicted response, a sequence of keyframes from the video, and a structured textual representation of the complete global tool call trajectory. The visual language model is prompted to perform the following analytical task: starting from the difference between the predicted and actual responses, it traces backward along the information flow path recorded in the global tool call trajectory to identify which tool call step(s) introduced incorrect information, omitted key information, or produced misleading intermediate conclusions, ultimately leading to the bias in the predicted response.

[0139] After locating the specific tool invocation step, the reflective agent extracts the context information of that step during execution. This context information can include: the subtask instructions when the step is invoked, the input parameters, the timestamps and visual feature descriptions of the video frames or video segments on which the step is based, the execution result returned by the step, and how the execution result is used in downstream steps. By synthesizing the above context information, the reflective agent drives the visual language model to analyze the reasons why the tool malfunctions in this specific scenario. For example, the tool may be applied to a visual scene beyond its applicable conditions, the input parameter configuration may not match the tool's expected input range, or the tool's perception algorithm may have failure modes in this type of data distribution.

[0140] Based on the cause of the error derived from attribution analysis, the reflective agent generates a constraint on the tool. This constraint can be expressed as a declarative rule, describing under what conditions the tool should be avoided, how the parameter configuration range of the tool should be limited, or what quality verification steps should be added to the output of the tool. For example, a constraint could be described as: when the brightness of a local area of ​​a video frame is lower than a preset threshold, the recognition result of the optical character recognition tool should not be used as the sole factual basis, but should be cross-validated in conjunction with the recognition results of other frames.

[0141] In S402, constraints are stored in the tool invocation experience base; the tool invocation experience base is used to guide or restrict the selection of tools and / or parameter configuration during the planning phase of subsequent video inference tasks.

[0142] The reflective agent stores the constraints generated in step S400 into the tool usage experience base maintained by the video understanding system, according to a preset data structure. This tool usage experience base is a persistent structured knowledge storage unit used to accumulate prior knowledge of tool usage learned by the system from historical failed tasks.

[0143] During storage, each constraint record can be associated with a corresponding tool identifier and includes information such as a description of the scenario to which the constraint applies, the original task identifier that generated the constraint, and the severity level or confidence level of the constraint. Constraints in the tool invocation experience base can be indexed by tool category to support efficient retrieval in the subsequent task planning stage.

[0144] The tool invocation experience base is used to guide or constrain tool selection and / or parameter configuration during the planning phase of subsequent video inference tasks. For example, when the orchestration agent executes step S200 or S202 of a new task, it can retrieve constraints matching the currently selected tool from the tool invocation experience base. If relevant constraints are found, the orchestration agent can incorporate these constraints as supplementary information into the tool's functional description or use them as exclusionary rules for tool selection, thereby proactively avoiding known failure modes during the planning phase. Through this mechanism, the video understanding system can not only abstract new macro tools from successes to expand its capability boundaries but also extract structured constraint knowledge from failures to shrink the error space, achieving bidirectional capability evolution.

[0145] In some embodiments, the constraints stored in the tool invocation experience base may become inapplicable as the system operating environment changes, the underlying visual language model version is updated, or the capabilities of the tools in the toolset iterate. Indiscriminately retaining historical lessons learned could lead to the orchestration agent being overly restricted by outdated constraints during the planning phase, thereby inhibiting its ability to explore better tool combinations. Therefore, this embodiment introduces a dynamic lifecycle management mechanism for experience.

[0146] After the constraints are stored in the tool's experience base, a separate experience management module in the reflexive agent or video understanding system sets two additional attributes for the constraints: an expiration counter and a confidence score. The expiration counter records the number of inference task rounds or natural time periods the constraint has undergone since its generation; its initial value can be set according to preset rules based on the constraint's severity level. The confidence score is a value that can be set between 0 and 1, used to quantify the validity and reliability of the constraint in the current system state; the initial confidence score can be assigned based on the degree of certainty in the error attribution analysis when the constraint was generated.

[0147] In subsequent video inference tasks, the system tracks the effect and updates the confidence score of each triggered constraint. When the orchestration agent avoids calling a specific tool or restricts specific parameter configurations based on a constraint during the planning phase, and the inference task is ultimately judged as successfully completed by the reflective agent, it indicates that the constraint is still valid in the current task scenario, and the system increases the confidence score of that constraint. The increase can be a preset fixed increment or dynamically adjusted according to the degree of task success.

[0148] Conversely, if the orchestration agent avoids calling a specific tool based on a constraint, leading to inference task failure or a significant decrease in task execution efficiency—for example, a significant increase in iterations due to the lack of key information that the tool could have provided—it indicates that the constraint may no longer be applicable in the current system state or may be overly restrictive. In this case, the system lowers the confidence score of the constraint. The magnitude of the reduction can be correlated with the severity of the negative consequences directly caused by avoiding the tool.

[0149] When the confidence score of a constraint falls below a preset elimination threshold after multiple updates, the system determines that the constraint no longer has guiding value and removes it from the experience base. This elimination mechanism ensures that the experience base always contains valid constraint knowledge that is adapted to the current system capability and has been continuously verified, giving the experience base the ability to self-correct and dynamically evolve.

[0150] The various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they are not described one by one. Therefore, the arbitrary combination of various technical features in the above embodiments is also within the scope of this specification.

[0151] Figure 5 This is a schematic structural diagram of a device provided in an exemplary embodiment. For example... Figure 5 As shown, device 500 mainly consists of a communication interface 502, a user interface 504, a processor 506, and a data storage 508. These components are interconnected and communicate with each other via a system bus, network, or other connection mechanism 510. The communication interface 502 enables device 500 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface 502 may include an antenna and related processing devices for wireless communication with a radio access network or access point. Furthermore, the communication interface 502 can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface 502 can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface 502 may also include multiple physical communication interfaces, such as Wi-Fi interfaces, Bluetooth interfaces, and wide-area wireless interfaces.

[0152] User interface 504 includes receiving user input and providing output to the user. Therefore, user interface 504 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. User interface 504 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, user interface 504 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, device 500 may support remote access from other devices via communication interface 502 or another physical interface (not shown). User interface 504 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. User interface 504 may also be configured as a display device for rendering or displaying text fragments.

[0153] Processor 506 may contain one or more general-purpose processors and / or special-purpose processors.

[0154] Data storage 508 may include one or more volatile and / or non-volatile storage components and may be integrated wholly or partially with processor 506. Data storage 508 may include removable and non-removable components.

[0155] Processor 506 is capable of executing program instructions 518 (e.g., compiled or uncompiled program logic and / or machine code) stored in data storage 508 to perform the various functions described herein. Data storage 508 may contain a non-transitory computer-readable medium on which program instructions are stored, which, when executed by device 500, enable device 500 to perform any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Execution of program instructions 518 by processor 506 may result in processor 506 using data 512.

[0156] For example, program instructions 518 may include an operating system 522 (e.g., an operating system kernel, device drivers, and / or other modules) installed on device 500 and one or more applications 520 (e.g., a browser, social application, or game application). Similarly, data 512 may include operating system data 516 and application data 514. Operating system data 516 is primarily accessible to the operating system 522, while application data 514 is primarily accessible to one or more applications 520. Application data 514 may reside in a file system visible or hidden from the user of device 500.

[0157] Application 520 can communicate with operating system 522 through one or more application programming interfaces (APIs). These APIs help application 520 read and / or write application data 514, transmit or receive information via communication interface 502, receive or display information on user interface 504, etc.

[0158] In some terminology, application 520 may be simply referred to as "app". Furthermore, application 520 can be downloaded to device 500 through one or more online app stores or app markets. However, applications can also be installed on device 500 in other ways, such as through a web browser or a physical interface on device 500 (e.g., a USB port).

[0159] In some embodiments, a video understanding device for a reflective agent and a video understanding device for an orchestration agent are provided, which can be applied to, for example... Figure 5 The device shown implements the technical solution described in this specification. For details on the specific implementation of the device, please refer to the descriptions in the relevant sections above; they will not be repeated here.

[0160] For ease of description, the above devices are described by dividing them into various modules or units based on their functions. Of course, when implementing one or more of these specifications, the functions of each module or unit can be implemented in the same or different software and / or hardware, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0161] Based on the same concept as the methods described above, this specification also provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor performs the steps of the method as described in any of the above embodiments by executing the executable instructions.

[0162] Based on the same concept as the methods described above, this specification also provides a computer-readable storage medium having computer instructions stored thereon that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0163] Based on the same concept as the methods described above, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the methods as described in any of the above embodiments.

[0164] What those skilled in the art will understand is: In this specification, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in a process, method, product, or apparatus that includes said elements is not excluded.

[0165] In this specification, “a,” “an,” and “the” do not specifically refer to the singular, but may also include the plural.

[0166] In this specification, ordinal numbers such as "first," "second," etc., do not necessarily indicate order; they are often used to distinguish between objects. For example, "first server" and "second server" usually refer to two servers. To differentiate between these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0167] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0168] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0169] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0170] Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is only one of many possible execution orders and does not represent the only execution order. Therefore, when the claims involve method steps, any changes or adjustments to the order of such steps, or the parallelism between steps, are also within the scope of protection of the claims.

Claims

1. A video understanding method, the method comprising: The execution data of the video inference task is obtained, including: the video, the user question for the video, the predicted answer, the preset real answer, and the global tool call trajectory generated during the generation of the predicted answer; wherein, the tools recorded in the global tool call trajectory are selected from the currently maintained toolset, which includes: micro-tools for performing single visual perception operations, and macro-tools that have been built and are composed of multiple micro-tools to achieve specific visual functions; If the predicted response is consistent with the actual response, a visual language model is used to identify and eliminate tool call steps that do not contribute to the generation of the predicted response from the global tool call trajectory, based on the video and the user question. The target tool call sequence is extracted, and a new macro tool is generated based on the call order and parameter dependencies of each micro tool in the target tool call sequence. The new macro tool includes a function description, input and output interface specifications, and the execution logic of the internal micro tools. The new macro tool is validated for usability. If the validation is passed, the new macro tool is added to the toolset.

2. The method according to claim 1, further comprising: If the predicted response is inconsistent with the actual response, the visual language model is used to perform attribution analysis on the global tool call trajectory to locate the tool call step that caused the predicted response to be incorrect, and the context corresponding to the tool call step is extracted to generate constraints for the tool used in the tool call step. The constraints are stored in the tool invocation experience base; wherein the tool invocation experience base is used to guide or limit the selection of tools and / or parameter configuration during the planning phase of subsequent video inference tasks.

3. The method according to claim 1, wherein the method for generating new macro tools comprises: Construct multimodal prompt words that include the keyframe sequence of the video, the user question, the predicted answer, the global tool call trajectory, and macro tool generation instructions; Input the multimodal cue words into the visual language model and execute the following macro tool generation steps: By combining the semantic intent of the user question with the visual evidence in the keyframe sequence, a relevance score is assigned to each tool call step in the global tool call trajectory. Identify and eliminate tool call steps with relevance scores below a preset score to obtain a candidate step set; Analyze the data dependencies and temporal logic among the tool invocation steps in the candidate step set to verify whether there is an input-output causal chain between the steps; Based on the verification results, isolated steps are removed, and the sequence of steps that constitute the causal chain of input and output is retained to form the target tool call sequence; The operation objects in the target tool call sequence are variableized; The variableization process includes: abstracting visual entity descriptions into generic type parameters, and abstracting numerical parameters into dynamic configuration items; Based on the variableized target tool call sequence, new macro tools are generated according to the preset macro tool template.

4. The method according to claim 1, wherein performing a usability verification on the new macro tool, and adding the new macro tool to the toolset if the usability verification is passed, comprises: Calculate the semantic similarity between the functional description of the new macro tool and the functional descriptions of existing macro tools in the toolset; If there is a target existing macro tool whose semantic similarity is greater than or equal to a preset similarity threshold, analyze the coverage relationship between the new macro tool and the target existing macro tool in the parameter space, as well as the logical inclusion relationship in the internal micro-tool call chain; If the new macro tool can cover all the parameters of the target existing macro tool, and at least one performance metric of the new macro tool is better than the corresponding performance metric of the target existing macro tool, then the target existing macro tool is replaced by the new macro tool; otherwise, the new macro tool is discarded. If the semantic similarity is less than the preset similarity threshold, the new macro tool is added to the toolset.

5. The method according to claim 4, further comprising: Maintain a pool of candidate macro tools; After generating a new macro tool, the matching degree between the new macro tool and each existing macro tool in the candidate macro tool pool is calculated; wherein the matching degree is determined based on at least one of the semantic similarity of the functional description, the consistency of the input and output parameter space, and the consistency of the internal micro-tool execution logic; If there is an existing macro tool with a matching degree higher than the preset matching threshold, the new macro tool is determined to belong to the same category as the existing macro tool, and the occurrence frequency of the existing macro tool is accumulated. When the cumulative occurrence frequency reaches a preset stable threshold, the availability check is triggered using the new macro tool or the most recently generated version of the existing macro tool as the object. If there is no existing macro tool with a matching degree higher than the preset matching threshold, the new macro tool is added to the candidate macro tool pool and its occurrence frequency is initialized to 1.

6. A video understanding method, the method comprising: The system receives a video reasoning task that includes video and user questions, obtains the currently accumulated context information and the functional descriptions of the tools in the currently available toolset, which includes: microtools for performing single visual perception operations, and macrotools that have been built and are composed of multiple microtools to achieve specific visual functions. Using a visual language model, joint reasoning is performed on the video, the context information, the user question, and the functional descriptions of the tools in the toolset to generate a subtask sequence for the current iteration round; wherein each subtask in the subtask sequence includes an execution instruction for the video and at least one tool called to execute the subtask; The subtask sequence is distributed to the execution agent for execution, and the execution results of each subtask are returned by the execution agent. The context information is updated based on each subtask and its execution result, and the tool call records of each subtask in this round are added to the global tool call trajectory. If the updated context information is insufficient to answer the user's question, return to the step of performing joint inference based on the updated context information; If the updated context information can answer the user question, the predicted response generated based on the updated context information, the user question, the video, and the global tool call trajectory are transmitted to the reflective agent so that the reflective agent performs the method described in any one of claims 1 to 5.

7. The method according to claim 6, wherein the information input to the visual language model includes: The keyframe sequence of the video, the context information, the user question, and the functional description of each tool in the candidate tool subset; Determining the subset of candidate tools includes: Determine the combined semantic vector of the user question and the context information, as well as the semantic vector of the function descriptions of each tool in the toolset; Determine the similarity between the combined semantic vector and the semantic vectors describing the functions of each tool; The N tools with the highest similarity are selected to form the candidate tool subset, where N is a positive integer.

8. The method of claim 6, further comprising, before generating the subtask sequence for the current iteration round: For each tool in the toolset, retrieve the constraints that match that tool from the tool invocation experience base; If a matching constraint is found, the constraint is concatenated into the original functional description of the corresponding tool to form an enhanced functional description; wherein, the functional description input to the visual language model is the enhanced functional description. If no matching historical call constraint is found, the input function description of the visual language model is the original function description.

9. A video understanding system, comprising an orchestration agent, an execution agent, and a reflective agent; The orchestration agent is used to perform the method according to any one of claims 6 to 8; The execution agent is used to receive subtasks, call at least one tool specified in the subtask to process the video according to the execution instructions in the subtask, obtain the execution result and return it to the orchestration agent; The reflective agent is used to perform the method described in any one of claims 1 to 5.

10. An electronic device, comprising: processor; A memory for storing processor-executable instructions; wherein the processor implements the steps of the method as described in any one of claims 1-8 by executing the executable instructions.

11. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-8.

12. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method as claimed in any one of claims 1-8.

Images