A video understanding method based on multi-modal agent reinforcement learning

Abstract

The application discloses a kind of video understanding methods based on multimodal agent reinforcement learning, it is related to computer vision processing technical field, and its technical solution key points include the following steps: obtaining video and query instruction, based on multimodal agent, query instruction is carried out semantic analysis, and the multidimensional constraint information related to video understanding task is extracted to generate target perception parameter for visual perception tool;Based on target perception parameter, video timing fragment screening, pixel-level target segmentation, cross-frame target tracking and identity association are completed;Effect is that the hybrid training mode of supervised fine-tuning combined with reinforcement learning is used, and the tool call and iterative decision strategy of multimodal agent are optimized by multidimensional joint reward function.The scheme can output fine-grained visual evidence carrying unique identity, pixel mask, spatial boundary, timing trajectory and perception confidence, realize the closed-loop collaborative optimization of perception and reasoning, effectively improve the accuracy of video understanding result in open scene.

Description

Technical Field

[0001] This invention relates to the field of computer vision processing technology, and more specifically, to a video understanding method based on multimodal agent reinforcement learning. Background Technology

[0002] Multimodal large-scale model-based video understanding technology is widely used in business scenarios such as video question answering, video behavior analysis, and time-series event retrieval, with significant progress in both research and practical applications. Current video understanding methods mostly rely on large models to complete global feature matching for content parsing. However, the overall framework lacks refined object-level spatial perception capabilities. The models can only output global feature vectors for the entire video segment, failing to generate fine-grained data such as pixel-level segmentation masks, cross-frame motion trajectories, and frame-by-frame confidence levels for individual targets within the frame. This makes it difficult to form traceable and verifiable accurate visual evidence. When faced with complex video footage containing target occlusion, multiple overlapping targets, and blurred small targets, global features are prone to feature confusion, making it impossible to distinguish the spatial boundaries and motion states of different objects, directly leading to misjudgments and omissions in subsequent semantic reasoning.

[0003] Furthermore, existing technologies all employ a one-time static feature extraction coupled with a single inference process. The visual evidence collection and semantic reasoning processes are unidirectional and irreversible, lacking a bidirectional iterative closed loop. The model can only complete one round of video sampling, target detection, and feature extraction according to preset fixed parameters. After the evidence verification is completed in the inference stage, even if problems such as missing spatiotemporal evidence, temporal retrieval interval deviation, or incomplete target attribute capture occur, it is impossible to reverse-adjust the perception parameters and re-collect and supplement cross-frame spatiotemporal evidence. For complex natural language query tasks with multiple constraints, the limited visual evidence collected in a single instance is insufficient to cover all inference elements, resulting in generally low confidence levels in the final analytical conclusions. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a video understanding method based on multimodal agent reinforcement learning.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A video understanding method based on multimodal agent reinforcement learning, comprising the following steps:

[0007] The system acquires video and query commands, performs semantic parsing of the query commands based on a multimodal intelligent agent, extracts multidimensional constraint information related to the video understanding task, and generates target perception parameters for visual perception tools.

[0008] Based on target perception parameters, the video is used for temporal segment selection, target perception, pixel-level target segmentation and cross-frame target association tracking, and object-level spatiotemporal visual evidence including target identity, pixel mask, spatial boundary, temporal trajectory and location confidence is constructed.

[0009] The object-level spatiotemporal evidence is input into the multimodal intelligent agent, and multi-step reasoning is performed in combination with query semantics to determine whether the current evidence meets the reasoning requirements with preset evaluation criteria;

[0010] If it is determined that the current evidence has covered all the reasoning elements required by the query command, and the confidence level of the reasoning result reaches the preset threshold, then the reasoning requirements are confirmed to be met, the loop is terminated, and the final video understanding result is output.

[0011] If the evidence is determined not to meet the reasoning requirements, the target perception parameters are corrected. Based on the corrected target perception parameters, spatiotemporal evidence at the target object level is generated iteratively. The evidence completeness assessment and parameter correction are continuously completed, forming an iterative closed loop of visual perception and semantic reasoning until the output conditions are met or the maximum number of iterations is reached.

[0012] Preferably, the multidimensional constraint information includes object category, attribute characteristics, action interaction relationships, and time constraints;

[0013] The target perception parameters include semantic cue words adapted to visual perception tools and candidate time intervals.

[0014] Preferably, the process involves acquiring the video and query instructions, performing semantic parsing of the query instructions based on a multimodal intelligent agent, extracting multidimensional constraint information related to the video understanding task, and generating target perception parameters for visual perception tools. This specifically includes the following steps:

[0015] The query command is decomposed into a structured task semantic set containing multidimensional constraints.

[0016] The acquired video is sampled and processed to construct a semantic input frame sequence covering global temporal sequence;

[0017] By fusing semantic input frame sequences with structured task semantic sets, target perception parameters for visual perception tools are generated through multimodal intelligent agent reasoning.

[0018] Preferably, based on target perception parameters, the video undergoes temporal segment selection, target perception, pixel-level target segmentation, and cross-frame target association tracking to construct object-level spatiotemporal visual evidence containing target identity, pixel mask, spatial boundary, temporal trajectory, and location confidence. Specifically, this includes the following steps:

[0019] Based on the candidate time intervals in the target perception parameters, valid video segments are extracted. Based on semantic prompts, visual perception tools are driven to complete pixel-level target segmentation and target localization, and the single-frame target pixel mask, spatial boundary and localization confidence are output.

[0020] Cross-frame association matching is performed on the target regions of consecutive video frames. Based on the overlap and position change characteristics of the target regions, the binding of the same source targets is completed, and a unique identifier is assigned to the same target to obtain a temporally continuous cross-frame pixel sequence.

[0021] By combining the target's unique identifier, cross-frame pixel mask sequence, frame-by-frame spatial boundary, and positional confidence, the temporal position and morphological change patterns of the target are fitted to construct standardized object-level spatiotemporal visual evidence.

[0022] Preferably, the iterative closed loop of visual perception and semantic reasoning is formed by judging the eligibility of spatiotemporal evidence at the target object level based on a multimodal intelligent agent, specifically including the following steps:

[0023] If the corrected target object-level spatiotemporal evidence is consistent with the object-level spatiotemporal evidence, then the loop terminates and the final video understanding result is output.

[0024] If the cumulative execution rounds of the visual perception and semantic reasoning iteration process reach the pre-set tool call limit, the loop will terminate and the final video understanding result will be output.

[0025] Preferably, it further includes:

[0026] The training trajectory is formed by acquiring multi-round semantic reasoning processes, visual tool retrieval behaviors, object-level spatiotemporal evidence content, and standard analysis conclusions.

[0027] Supervised fine-tuning training of the multimodal agent is carried out based on the complete training trajectory to strengthen the basic semantic reasoning and tool calling capabilities of the multimodal agent for video understanding tasks, and to build a multi-dimensional joint reward mechanism that includes result accuracy, calling standardization, and target localization quality.

[0028] A multi-level collaborative feedback incentive mechanism guides multimodal intelligent agents to adapt to visual perception control and semantic reasoning in video query tasks.

[0029] Compared with existing technologies, this invention has the following advantages: By generating customized target perception parameters through multimodal intelligent agent parsing queries, and constructing standardized object-level spatiotemporal evidence through pixel-level segmentation and cross-frame association matching, it avoids the situation of relying solely on global feature matching and lacking fine-grained object perception capabilities. This solution can output fine-grained visual evidence carrying unique identifiers, pixel masks, spatial boundaries, temporal trajectories, and perception confidence, completely distinguishing the spatial and motion features of different targets in occluded and overlapping scenes, providing traceable and verifiable accurate visual evidence for multi-step semantic reasoning, and significantly reducing the probability of target confusion and reasoning errors in complex scenes. After the intelligent agent completes the evidence completeness judgment, it can adaptively correct the target perception parameters according to the reasoning deviation, re-drive the visual tool to supplement the collection of missing spatiotemporal evidence, and cyclically complete the evidence collection and verification until the termination condition is met. It can actively supplement all target, attribute, action, and temporal constraint evidence required for composite queries, significantly improving the completeness and confidence of reasoning results in complex multi-condition query scenarios. Attached Figure Description

[0030] Figure 1 This invention provides a schematic diagram illustrating the steps of a video understanding method based on multimodal agent reinforcement learning, as described in an embodiment of the invention.

[0031] Figure 2 This is a schematic diagram illustrating the steps of forming standardized object-level spatiotemporal visual evidence in a video understanding method based on multimodal agent reinforcement learning, as provided in an embodiment of the present invention. Detailed Implementation

[0032] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0033] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0034] Secondly, the term "an embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places throughout this specification does not necessarily refer to the same embodiment, nor is it a single embodiment or an embodiment selectively excluded from other embodiments.

[0035] Reference Figures 1-2 As shown.

[0036] Example 1 further illustrates the video understanding method based on multimodal agent reinforcement learning proposed in this invention.

[0037] A video understanding method based on multimodal agent reinforcement learning, comprising the following steps:

[0038] The system acquires video and query commands, performs semantic parsing of the query commands based on a multimodal intelligent agent, extracts multidimensional constraint information related to the video understanding task, and generates target perception parameters for visual perception tools.

[0039] Based on target perception parameters, the video is used for temporal segment selection, target perception, pixel-level target segmentation and cross-frame target association tracking, and object-level spatiotemporal visual evidence including target identity, pixel mask, spatial boundary, temporal trajectory and location confidence is constructed.

[0040] The object-level spatiotemporal evidence is input into the multimodal intelligent agent, and multi-step reasoning is performed in combination with query semantics to determine whether the current evidence meets the reasoning requirements with preset evaluation criteria;

[0041] If it is determined that the current evidence has covered all the reasoning elements required by the query command, and the confidence level of the reasoning result reaches the preset threshold, then the reasoning requirements are confirmed to be met, the loop is terminated, and the final video understanding result is output.

[0042] If the evidence is determined not to meet the reasoning requirements, the target perception parameters are corrected. Based on the corrected target perception parameters, spatiotemporal evidence at the target object level is generated iteratively. The evidence completeness assessment and parameter correction are continuously completed, forming an iterative closed loop of visual perception and semantic reasoning until the output conditions are met or the maximum number of iterations is reached.

[0043] Multimodal intelligent agents rely on reinforcement learning to form autonomous control-driven logic. They can retrieve all information stored in the past reasoning process and simultaneously absorb object-level spatiotemporal evidence feedback information from each round of visual perception output. Based on the information stored in the entire reasoning process and the object-level spatiotemporal evidence feedback information, they comprehensively adjust the deduction order and judgment level of semantic reasoning. Relying on the dynamically adapted reasoning logic, they reversely correct the target perception parameters corresponding to visual perception, so that the high-order semantic reasoning process and the low-order pixel-scale image extraction process form a two-way linkage closed-loop collaborative operation mode.

[0044] The multidimensional constraint information includes object category, attribute characteristics, action interaction relationship and time constraint;

[0045] The target perception parameters include semantic cue words adapted to visual perception tools and candidate time intervals.

[0046] The process involves acquiring video and query commands, performing semantic parsing of the query commands based on a multimodal intelligent agent, extracting multidimensional constraint information related to the video understanding task, and generating target perception parameters for visual perception tools. This includes the following steps:

[0047] The query command is decomposed into a structured task semantic set containing multidimensional constraints.

[0048] The acquired video is sampled and processed to construct a semantic input frame sequence covering global temporal sequence;

[0049] By fusing semantic input frame sequences with structured task semantic sets, target perception parameters for visual perception tools are generated through multimodal intelligent agent reasoning.

[0050] The multimodal agent first performs basic sampling and input construction operations on the video to be understood. The complete video to be understood can be abstractly defined as a set of continuous frames, and the corresponding mathematical expression is: In the formula, I t Let represent the t-th frame in the video sequence, where T represents the total number of frames in the original video. Since original videos typically have a large number of frames and high temporal redundancy, directly feeding all original frames into the agent would result in a significant waste of computational resources. Therefore, this scheme uses a uniform interval sampling method to downsample the original video, thereby constructing a global semantic input frame sequence that can cover the entire temporal range of the video. After uniform downsampling, a simplified video frame sequence is obtained, calculated using the following formula: In the formula, N is the total number of initial sampling frames pre-set by the system, and τn represents the frame index number corresponding to a single sampling frame in the original video. The frame index calculation formula is: This formula relies on linear division combined with floor function to ensure that all sampled frames are evenly distributed across the complete temporal range of the video, avoiding imbalances such as frame accumulation in local time periods or complete absence of sampled frames in some time sequences. For example, if the original total number of frames T in a surveillance video is 2400 and the preset initial total number of sampled frames N is 240, substituting these two sets of values ​​into the frame index calculation formula will solve for 240 independent τn values. The resulting index values ​​are evenly distributed across the entire frame range from 1 to 2400, fully covering the temporal stages of the video's beginning, middle, and end. This allows the multimodal agent to fully capture the inherent semantics of the global scene, the unchanging background images, and the positional and morphological changes of the main subject over time. After all sampling operations are completed, the agent sends the generated global semantic input frame sequence and the unmodified original query command into the internal processing unit to complete the construction of the model input context. The context integrates the global temporal image information of the video with the original query text information, providing complete global prior information for subsequent analysis of query semantics and extraction of multidimensional constraints.

[0051] After video sampling and processing, the multimodal agent initiates the task semantic parsing process for the query command. Let Q be the query command input by the user. This command is unstructured natural language text and cannot be directly recognized and invoked by the visual segmentation and tracking tool. The multimodal agent, relying on its built-in text semantic understanding capabilities, performs a structured decomposition of the query command, extracting all multidimensional constraint information strongly related to the current video understanding task, and integrating them to form a structured task semantic set. The corresponding mathematical expression is: In the formula, C is the set of all target object categories involved in the query command, which is used to limit the target semantic category that the visual perception process needs to identify; A is the set of attribute features of the target object, covering the visual attributes such as object color, appearance texture, clothing features, geometric shape, spatial position, and motion state, which is used to refine the distinction judgment conditions between different targets; R is the set of semantic relationships corresponding to the task, covering the interaction relationship between objects, action execution relationship, spatial position relationship, event sequence relationship, etc., which is used to constrain the behavioral logic and association rules corresponding to the target. This is the set of time constraints corresponding to the query command, used to limit the time interval of the target event, the sequence of actions, and the effective appearance period of the target, etc., as determined by time sequence rules. For example, when the input query command is to locate all the footage of a pedestrian in a red coat running continuously on the right side of the second half of a video, the multimodal agent, after hierarchical semantic decomposition, will fill in four types of data within the structured task semantic set: category set C represents the pedestrian; attribute feature set A represents the spatial position and running motion state of the red coat relative to the right side of the screen; semantic relationship set R represents the single-object action relationship of the pedestrian independently performing the running action; and the time constraint set... For the time interval corresponding to the second half of the video, the four types of constraints are fully incorporated into the structured task semantic set. The various constraints within the set maintain logical connections, fully restore all reasoning requirements of the query instruction, and complete the conversion from natural language query to machine-recognizable structured constraints, providing a complete semantic basis for the subsequent generation of perception parameters.

[0052] Once both the global semantic input frame sequence and the structured task semantic set are generated, the multimodal agent initiates feature fusion operations on the two data streams. It performs a deep weighted fusion of the video global temporal feature vector carried by the semantic input frame sequence and the four-dimensional constraint semantic features carried by the structured task semantic set. After the fusion operation, target perception parameters for use by visual perception tools are generated. These parameters simultaneously carry multiple control indicators, including the video global temporal coverage range, query-defined target detection categories, image retrieval spatial boundaries, pixel segmentation accuracy thresholds, and effective time windows for cross-frame target matching. Upon receiving these parameters, the visual perception tool strictly adheres to all the constraints within the parameters to perform subsequent target perception, pixel-level segmentation, and cross-frame association matching operations. This achieves a pre-scheduling logic that relies on the bidirectional collaborative control of video global temporal information and query multidimensional constraint information to regulate the scope and accuracy of visual evidence collection. This completes the entire technical process of acquiring video and query commands, using the multimodal agent to complete video sampling to construct a temporal frame sequence, query structured decomposition to extract multidimensional constraints, and fusion of the two data streams to generate target perception parameters adapted to the visual perception tool.

[0053] Based on target perception parameters, the video undergoes temporal segment selection, target perception, pixel-level target segmentation, and cross-frame target correlation tracking to construct object-level spatiotemporal visual evidence that includes target identity, pixel mask, spatial boundary, temporal trajectory, and location confidence. Specifically, this includes the following steps:

[0054] Based on the candidate time intervals in the target perception parameters, valid video segments are extracted. Based on semantic prompts, visual perception tools are driven to complete pixel-level target segmentation and target localization, and the single-frame target pixel mask, spatial boundary and localization confidence are output.

[0055] Cross-frame association matching is performed on the target regions of consecutive video frames. Based on the overlap and position change characteristics of the target regions, the binding of the same source targets is completed, and a unique identifier is assigned to the same target to obtain a temporally continuous cross-frame pixel sequence.

[0056] By combining the target's unique identifier, cross-frame pixel mask sequence, frame-by-frame spatial boundary, and positional confidence, the temporal position and morphological change patterns of the target are fitted to construct standardized object-level spatiotemporal visual evidence.

[0057] First, the process of defining candidate time intervals and extracting corresponding continuous video segments based on target perception parameters is performed. The timing constraints within the tool call instruction of the k-th round are calculated using the following formula: In the formula, This represents the starting frame index of the k-th round of temporal constraint interval, corresponding to the original video frame position where visual forensics processing needs to begin in this round, and is the left boundary threshold for video segment extraction. k indicates the k-th iteration, and s represents the semantic identifier from the beginning, used to distinguish the start and end points of the interval. represents the end frame index value of the k-th round of temporal constraint interval, corresponding to the original video frame position where visual evidence processing stops in this round, is the right boundary threshold for video segment extraction, and 'e' represents the semantic identifier of self-end. Based on this temporal constraint, target video segments containing only task-related images are extracted from the complete original video. The formula for calculating the extracted segment set is as follows: This extraction operation can filter out all time-series frames within the video that are irrelevant to the current query task, significantly reducing the total amount of image data that the visual tool needs to process, reducing redundant computing power consumption, and allowing each round of visual perception operations to focus only on the video segment corresponding to the query requirement, improving the accuracy and targeting of visual forensics. For example, if the current iteration is the 3rd round, The value is 200. If the value is 800, then for This means that only the 200th to 80th frames of the original video are processed in this round, and all other time-series frames are filtered out and not involved in the perception calculation. All subsequent segmentation and tracking calculations are carried out only within this 600-frame segment, avoiding the waste of resources caused by frame-by-frame calculation of the entire video.

[0058] After the target video segment is extracted, the system calls the Open Vocabulary Video Segmentation and Tracking Tool to perform pixel-level segmentation, identifying the object concept cues corresponding to the semantic constraints in this round. With the selected target video clips Common input visual tools The tool combines textual semantic constraints to distinguish foreground and background in each frame of the segment, divides the pixel area covered by the target frame by frame, and simultaneously outputs the visual reliability value corresponding to each pixel area. The calculation formula for the complete perception output result of this round is: The complete formula for calculating standardized perception results in multi-target scenarios is as follows: In the formula, This represents the total number of all valid targets identified in the k-th round of perception. The m-th target-specific unique identifier is used to bind the same entity across frames. The binary pixel mask corresponding to the m-th target in frame t is used to accurately represent the target's contour region. The confidence score, representing the target perception result of a single frame, is used to measure the reliability of the segmentation and detection in the current frame. This scheme preferentially uses the SAM3 model as the segmentation and tracking tool, while also being compatible with other visual models that have open vocabulary detection, pixel segmentation, and cross-frame tracking capabilities. After frame-by-frame computation, the tool can output the mask and spatial boundary range of all target pixels in each frame, which is the bounding box of each target. It fully preserves the spatial morphology and confidence level information of the target at the single-frame level. For example, if the query requirement is to identify a cyclist in the scene, the tool will divide the foreground pixel mask corresponding to the cyclist's body in each frame of the captured video clip based on the prompt words corresponding to the cyclist, and simultaneously output the confidence value corresponding to the cycling target in each frame. At the same time, it will generate a spatial bounding box that surrounds the cyclist, and fully record the outline and spatial position of the cycling target in a single frame.

[0059] After obtaining all the target pixel masks and spatial boundary information for each single frame, the process enters the cross-frame same-source target association and matching stage. The system compares all target regions in adjacent frames sequentially according to the original temporal order of the video, calculates the overlap ratio of the target pixel masks between the two frames and the positional change of the spatial bounding box. Two targets in the same frame with a high overlap ratio and a positional change within a preset threshold are determined to be the same visual entity, and the same unique identifier is bound to the successfully matched same-source entities. The system integrates and concatenates all single-frame pixel masks carrying the same identifier to generate a coherent pixel mask sequence covering the entire candidate time interval. Based on the matched identifiers and coherent pixel sequences, the system calculates the coordinate offset and mask contour changes of each target's bounding box as the image progresses, frame by frame, to reconstruct the complete temporal dynamics of the target. To further reduce background noise and highlight task-related target features, the system simultaneously performs frame-level target visual enhancement processing, setting a background preservation coefficient α with a value between zero and one. The standard calculation formula for the enhanced frame is: The target area covered by the mask in the formula retains all pixel information of the original image, while the background area not covered by the mask will have its image information attenuated synchronously according to the background retention coefficient, thus achieving the effect of highlighting the visual features of the target and suppressing irrelevant background information, and supplementing the spatiotemporal evidence with enhanced visual image information.

[0060] Finally, the system integrates all computational outputs to construct complete object-level spatiotemporal evidence. First, it constructs a single-target-specific spatiotemporal trajectory based on unique identifiers, pixel mask sequences, frame-by-frame spatial boundary ranges, and temporally dynamic change data. The complete calculation formula for the target's spatiotemporal trajectory is: This trajectory comprehensively records all dynamic features of the target entity within the candidate video time period, including the precise time of the target's appearance and disappearance, frame-by-frame spatial position offset, mask contour morphology changes, and fluctuations in perception confidence for each round. This enables the conversion from discrete single-frame segmentation results to a temporally continuous entity state model. The system integrates and encapsulates all the unique identifiers of all targets, their overall positional variation patterns, target pixel masks, spatial boundaries, temporally dynamic change data, and frame-level enhanced video clips, forming standardized object-level spatiotemporal evidence. This evidence simultaneously carries both static spatial pixel features and dynamic temporal motion features of the target, providing fine-grained, traceable, and complete visual evidence for subsequent multi-step semantic reasoning by multimodal agents.

[0061] Based on multimodal intelligent agents, an iterative closed loop of visual perception and semantic reasoning is formed by judging the eligibility of spatiotemporal evidence at the target object level. Specifically, it includes the following steps:

[0062] If the corrected target object-level spatiotemporal evidence is consistent with the object-level spatiotemporal evidence, then the loop terminates and the final video understanding result is output.

[0063] If the cumulative execution rounds of the visual perception and semantic reasoning iteration process reach the pre-set tool call limit, the loop will terminate and the final video understanding result will be output.

[0064] First, the system performs operations to integrate spatiotemporal visual evidence into the reasoning context. After completing the k-th round of visual perception processing, the system summarizes the spatiotemporal visual evidence corresponding to all candidate target entities in this round, and integrates them to obtain the exclusive evidence set for this round, marked as follows. Simultaneously retrieve evidence data retained from all historical iterations, and overlay it with the accumulated evidence set covering the entire process in this round of evidence construction, marked as... The accumulated spatiotemporal evidence set, the initial global semantic frame sequence of the video, and the original query command are input into the inference unit inside the multimodal agent to complete the deep feature fusion of visual pixel evidence and text query semantics. Based on the complete fused context, this round of multi-step semantic reasoning is carried out. The standard calculation formula corresponding to the reasoning result of this round is: In the formula, This represents a multimodal agent inference network trained through reinforcement learning. Q represents the global semantic input frame sequence, and Q represents the original natural language query command. This represents a cumulative set of spatiotemporal evidence encompassing all historical iterations. This inference operation can fully inherit the spatiotemporal evidence information generated in each iteration, ensuring logical continuity and consistency throughout multiple iterations and avoiding information gaps or loss in single-round inference. For example, when the iteration process reaches the third round, the system aggregates all spatiotemporal trajectory evidence retained from the first and second rounds, superimposing it with newly collected object spatiotemporal evidence from the third round to form a cumulative evidence set. This set, along with sampled video sequences and query text, is fed into the inference network to solve for the third round's inference result. The inference process can retrieve target location and action information identified in the first two rounds, without losing effective visual clues obtained in the early stages.

[0065] After completing the current round of inference and outputting the inference result, the multimodal agent initiates a spatiotemporal visual evidence completeness judgment operation. The agent simultaneously retrieves four types of reference information: structured multidimensional semantic constraints corresponding to the query command, visual features of the enhanced target image, the spatiotemporal trajectory change patterns of the object throughout the entire process, and historical tool call records for each round. It comprehensively compares the completeness of the current accumulated spatiotemporal evidence set to determine whether the existing evidence fully meets all the inference output conditions corresponding to the video understanding task. The judgment process is divided into two types. The first is evidence qualification: when the accumulated evidence set completely covers all target entities, object attributes, interactive actions, and temporal relationships included in the query requirement, and can support stable, accurate, and unbiased video understanding inference conclusions, the system determines that the current evidence is fully qualified and directly solidifies the current round of inference results as the final conclusion to be output. The second is evidence defects: when the existing spatiotemporal evidence has various defects such as missing target information, deviations in the temporal retrieval range, fuzzy target segmentation features, and missed identification of object interaction relationships, and cannot support credible and effective inference output, the system automatically triggers an iterative update mechanism, generating a new round of global perception planning results based on the inference network. The calculation formula is: In the formula, S represents the structured task semantic set obtained by decomposing the query instruction. The new planning result carries the corrected set of perception control parameters and automatically enters the next round of visual perception optimization process. For example, if the query requires the identification of a pedestrian's complete running sequence, but the current spatiotemporal evidence only captures the pedestrian's initial image and lacks the middle and final movement trajectories, the agent determines that the evidence has a defect of missing temporal information, triggers the iterative update mechanism to generate corrected perception planning parameters, and starts a new round of visual evidence collection operations.

[0066] After triggering the iterative update mechanism, the system enters the computation phase of dynamically correcting perception parameters and collecting updated spatiotemporal evidence. The agent matches the corresponding parameter correction strategy based on the type of evidence bias identified in this round of inference computation. Available correction strategies include adding target object attribute constraints, supplementing object interaction relationship constraints, eliminating target concepts irrelevant to the query, scaling candidate video temporal intervals, and shifting the boundaries of temporal retrieval intervals—multiple optimization methods. Each strategy specifically corrects semantic and temporal range biases that occurred in the previous round of visual perception. After all parameter correction operations are completed, a new round of standardized tool call instructions is generated based on the updated global planning results, marked as... The instruction is sent to the refined visual evidence acquisition module, which drives the segmentation and tracking tool to re-extract video segments according to the corrected constraint range, perform pixel-level segmentation and cross-frame homologous matching, and acquire a new round of updated object-level spatiotemporal evidence. The new evidence is then sent back to the multimodal agent to conduct completeness judgment again, thus forming a closed-loop iterative cycle of visual perception evidence acquisition, semantic reasoning judgment, and perception parameter correction.

[0067] After each round of evidence discrimination and parameter update, the system synchronously executes the closed-loop iteration termination condition control logic. This scheme sets multiple independent termination judgment conditions. Meeting any one of these conditions during the process will immediately stop all tool calls and iterative cycles, directly outputting the final, solidified video understanding result. This corresponds to two types of core termination rules. The first type of termination rule is the consistency judgment between two rounds of evidence. When the spatiotemporal evidence of the target object level re-collected after the current round of correction is completely consistent with the object-level spatiotemporal evidence retained from the previous iteration, it means that the perceptual parameter adjustment can no longer supplement new effective visual cues, and the completeness of the evidence has reached its upper limit. The system then terminates the loop and outputs the inference result. For example, the spatiotemporal evidence collected after correcting the time series interval and target constraints in the 4th iteration contains the exact same target trajectory, pixel mask, time series interval, and confidence information as the spatiotemporal evidence generated in the 3rd iteration. There is no new effective evidence content, and the system determines that the evidence no longer has room for optimization, directly ending the iteration and outputting the result. The second type of termination rule is the upper limit judgment of the iteration rounds. The system pre-sets a global maximum number of tool call rounds as... The system continuously tracks the cumulative execution rounds of the visual perception and semantic reasoning iteration process. When the cumulative number of rounds reaches the preset maximum limit, to avoid continuous computational consumption caused by infinite iteration, the system forcibly terminates the loop and outputs the currently solidified video understanding and reasoning result. For example, the system presets a maximum tool call count of ten rounds. When the iteration process has completed ten perception evidence collection and reasoning judgment operations, regardless of whether there are minor flaws in the current spatiotemporal evidence, the system will stop the iteration loop and output the reasoning conclusion supported by the current accumulated evidence. These two types of termination conditions work together to balance the accuracy of video understanding and reasoning with the overall efficiency of computational resource consumption. After the iteration terminates, the final video understanding result is fully output, and the entire two-way closed-loop video understanding process of perception and reasoning is completed.

[0068] Also includes:

[0069] The training trajectory is formed by acquiring multi-round semantic reasoning processes, visual tool retrieval behaviors, object-level spatiotemporal evidence content, and standard analysis conclusions.

[0070] Based on the complete training trajectory and the multi-agent interaction optimization mechanism, the self-regulation logic of the multimodal intelligent agent is adjusted, and a multi-level collaborative feedback incentive mechanism is constructed.

[0071] Supervised fine-tuning training of multimodal agents is performed based on the complete training trajectory to solidify the basic semantic reasoning and tool invocation capabilities of multimodal agents for video understanding tasks, and a multi-dimensional joint reward mechanism including result accuracy, invocation standardization, and target localization quality is constructed.

[0072] The process begins with the acquisition and construction of the complete trajectory. The system first constructs training samples consisting of pairs of original videos and corresponding query commands. Each training sample is uniformly labeled as follows: For each independent training sample, the entire iterative closed-loop process of preceding visual perception and semantic reasoning is fully simulated. This fully replicates the multi-round semantic reasoning operation, the visual tool's command retrieval and output behavior, and the object-level spatiotemporal evidence content collected in each round. Simultaneously, pre-annotated standard analytical conclusions are introduced, integrating all temporal step information to form a standardized and complete training trajectory. The calculation formula for a single training trajectory is: In the formula, R represents the output of a single round of semantic reasoning, A represents the visual tool invocation command issued in a single round, E represents the object-level spatiotemporal evidence collected in a single round, the superscript numbers distinguish the iteration rounds, and Y represents the standard analytical conclusion corresponding to the sample. The complete training trajectory fully retains all temporal interaction information from the initial parameter generation, multiple rounds of perception evidence collection, multi-step semantic reasoning to the final standard conclusion, providing complete temporal supervision data for subsequent supervision fine-tuning and reward calculation. For example, for a surveillance video sample containing a pedestrian running event, coupled with a query command to retrieve the pedestrian's movement trajectory, the system fully simulates a six-round iterative closed loop, sequentially recording the reasoning output, tool invocation command, and spatiotemporal evidence set of each round, and then combining it with the standard event analytical conclusion obtained from manual annotation to integrate and generate a complete trajectory containing six rounds of interaction information.

[0073] After collecting a sufficient amount of complete training trajectories, the process enters the stage of adjusting the agent's control logic based on the trajectories and building a multi-level collaborative feedback incentive mechanism. This stage is divided into two sub-steps: supervised fine-tuning and the construction of a multi-dimensional joint reward function. These two steps are executed sequentially to complete the incentive mechanism construction. The first step involves supervised fine-tuning training of the multimodal agent. Gradient update operations are performed based on all collected training trajectories, using a dedicated supervised loss function. The formula for the loss function is: In the formula, The mask identifier is used to shield the gradient update signals generated by low-quality or erroneous intermediate steps within the trajectory, retaining only compliant and valid supervision signals to participate in parameter iteration. This strengthens the agent's basic task planning capabilities and standardized tool call output capabilities, and completes the basic correction of the agent's initial control logic.

[0074] This scheme constructs a multi-dimensional reward function corresponding to multi-level collaborative feedback. It sets three independent reward components: the first is a video understanding result reward, used to constrain the matching accuracy between the agent's final output reasoning conclusion and the standard analytical conclusion; the second is a tool invocation format reward, used to standardize the agent's output of structured perception parameters and tool instructions; the third is a cross-frame target localization reward with a penalty term, which evaluates the evidence quality of each round of visual evidence collection by comparing the target temporal intersection-union ratio and the pixel mask intersection-union ratio, and simultaneously introduces a penalty value to suppress ineffective and excessively biased target localization behaviors. The three reward components are weighted and fused to generate a unified joint reward function, calculated as follows: In the formula, , , These are the weight coefficients corresponding to the three types of rewards, which are dynamically and adaptively adjusted during the training process. The accuracy of the result is represented by a reward component. Representative tool format specifications and reward weights, This represents the cross-frame localization reward component with a penalty term. This joint reward function forms the core quantitative basis of the multi-level collaborative feedback incentive mechanism, providing simultaneous feedback on the quality of each step of the agent's regulatory behavior across three levels: final inference accuracy, tool output standardization, and visual evidence collection quality. This provides quantitative feedback signals for subsequent policy optimization. For example, if the agent's output tool call instruction format is missing in one round, or the intersection-union (IUU) value of the collected spatiotemporal evidence target mask is low, the joint reward function will output a smaller reward value, simultaneously triggering a penalty term to reduce the overall reward score, thus indicating a defect in the agent's regulatory logic for that round.

[0075] Based on the established multi-level collaborative feedback incentive mechanism, the system executes a group relative policy optimization algorithm to iteratively update agent parameters, thereby guiding the multimodal agent to autonomously adapt to the visual perception control and semantic reasoning logic of various video query tasks. The algorithm relies on multiple shared training trajectories to calculate the relative benefit advantages between different agent control strategies, and uses KL divergence constraints to ensure stable convergence during the policy update process. It continuously iteratively optimizes the entire set of control logic within the agent, including perception parameter correction strategies for visual tools, multi-step semantic reasoning logic, and iteration termination condition determination strategies. After multiple rounds of policy optimization iterations, the multimodal agent can autonomously complete adaptation and control based on any new video and query command. For complex queries, it automatically adjusts perception control operations such as temporal interval scaling, target attribute constraint supplementation, and cross-frame matching threshold modification, simultaneously completing coherent and stable multi-step progressive semantic reasoning, fully realizing the optimization goal of the reinforcement learning training chain for the agent's perception and reasoning collaborative capabilities. For example, after optimizing training trajectories for a large number of pedestrians, vehicles, and animals, when a new video query task of retrieving the temporal interaction behavior of cyclists is input, the agent can autonomously determine that it is necessary to broaden the candidate temporal interval, supplement dynamic motion attribute constraints, automatically correct perception parameters to complete fine-grained visual evidence collection, and then perform multi-step temporal semantic reasoning based on complete spatiotemporal evidence to output a video understanding conclusion that matches the task requirements with high precision.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A video understanding method based on multimodal agent reinforcement learning, characterized in that, The method includes the following steps: The process involves acquiring video content and query commands, performing semantic parsing of the query commands based on a multimodal intelligent agent, extracting multidimensional constraint information related to the video understanding task, and generating target perception parameters for visual perception tools. This includes the following steps: The query command is decomposed into a structured task semantic set containing multidimensional constraint information. The acquired video is sampled and processed to construct a semantic input frame sequence covering global temporal sequence; By fusing semantic input frame sequences with structured task semantic sets, target perception parameters for visual perception tools are generated through multimodal intelligent agent reasoning. The multidimensional constraint information includes object category, attribute characteristics, action interaction relationship and time constraint; The target perception parameters include semantic cue words adapted to visual perception tools and candidate time intervals; Based on target perception parameters, the video undergoes temporal segment selection, target perception, pixel-level target segmentation, and cross-frame target correlation tracking to construct object-level spatiotemporal visual evidence that includes target identity, pixel mask, spatial boundary, temporal trajectory, and location confidence. Specifically, this includes the following steps: Based on the candidate time intervals in the target perception parameters, valid video segments are extracted. Based on semantic prompts, visual perception tools are driven to complete pixel-level target segmentation and target localization, and the single-frame target pixel mask, spatial boundary and localization confidence are output. Cross-frame association matching is performed on the target regions of consecutive video frames. Based on the overlap and position change characteristics of the target regions, the binding of the same source targets is completed, and a unique identifier is assigned to the same target to obtain a temporally continuous cross-frame pixel sequence. By combining the target's unique identifier, cross-frame pixel sequence, frame-by-frame spatial boundary, and positional confidence, the temporal position and morphological change patterns of the target are fitted to construct standardized object-level spatiotemporal visual evidence. The object-level spatiotemporal visual evidence is input into a multimodal agent, and multi-step reasoning is performed in combination with query semantics. The current evidence is judged to meet the reasoning requirements by a preset evaluation criterion. The preset evaluation criteria include the confidence level of comparative reasoning, the completeness of key evidence and the matching degree of temporal relationships; The key evidence integrity means that the current evidence completely covers all target entities, object attributes, interactive actions, and temporal relationships included in the query instruction; If it is determined that the current evidence has covered all the reasoning elements required by the query command, and the confidence level of the reasoning result reaches the preset threshold, then the reasoning requirements are confirmed to be met, the loop is terminated, and the final video understanding result is output. If the current evidence is determined not to meet the reasoning requirements, the target perception parameters are corrected. Based on the corrected target perception parameters, the visual perception tool is driven to re-execute the temporal segment screening, target perception, pixel-level target segmentation, and cross-frame target correlation tracking to generate target object-level spatiotemporal evidence. The evidence completeness assessment and parameter correction are continuously completed to form an iterative closed loop of visual perception and semantic reasoning until the output conditions are met or the maximum iteration round is reached.

2. The video understanding method based on multimodal agent reinforcement learning according to claim 1, characterized in that, Based on multimodal intelligent agents, an iterative closed loop of visual perception and semantic reasoning is formed by evaluating the completeness of spatiotemporal evidence at the target object level. Specifically, it includes the following steps: If the corrected object-level spatiotemporal evidence matches the object-level spatiotemporal visual evidence, then the loop terminates and the final video understanding result is output. If the cumulative execution rounds of the visual perception and semantic reasoning iteration process reach the pre-set tool call limit, the loop will terminate and the final video understanding result will be output.

3. The video understanding method based on multimodal agent reinforcement learning according to claim 2, characterized in that, Also includes: The training trajectory is formed by acquiring multi-round semantic reasoning processes, visual perception tool retrieval behaviors, object-level spatiotemporal visual evidence content, and standard analysis conclusions. Supervised fine-tuning training of the multimodal agent is carried out based on the complete training trajectory to strengthen the basic semantic reasoning and tool calling capabilities of the multimodal agent for video understanding tasks, and to build a multi-dimensional joint reward mechanism that includes result accuracy, calling standardization, and target localization quality. A multi-level collaborative feedback incentive mechanism guides multimodal intelligent agents to adapt to visual perception control and semantic reasoning in video query tasks.

Images