Multi-agent reinforcement learning optimization method and system for complex video understanding

Abstract

The application discloses a kind of multi-agent reinforcement learning optimization methods and systems for complex video understanding, it relates to multimodal video understanding technical field.The application includes: obtaining video to be processed and query text, input pre-trained video understanding model to video to be processed and query text;Task planning is carried out to video to be processed and query text, and positioning plan is obtained;Based on positioning plan, cross-modal feature fusion is carried out to the visual feature of video to be processed and the text feature of query text, and time stamp perception mark trigger timing boundary decoding is generated, and initial timing positioning segment is obtained;With confidence as reward signal, multi-scale neighborhood search is carried out to initial timing positioning segment, and fine-tuned timing positioning segment is obtained;Depth inference is carried out to fine-tuned timing positioning segment, and final answer is obtained.The application effectively compensates local error of initial localizer under noise interference, and significantly improves the performance of model in long video timing positioning, video question and answer and other tasks.

Description

Technical Field

[0001] This invention relates to the field of multimodal video understanding technology, and in particular to a multi-agent reinforcement learning optimization method and system for complex video understanding. Background Technology

[0002] With the rapid development of multimodal large-scale models, video understanding has gradually evolved from simple video question answering to a higher-order task requiring accurate temporal localization and complex causal reasoning. In long video scenarios, the sparse distribution of key events, severe background redundancy and noise, and complex temporal dependencies make it difficult for traditional models to simultaneously achieve high-precision segment localization and reliable logical reasoning.

[0003] Current mainstream video understanding methods generally adopt a one-time temporal localization strategy. The implementation process includes: uniformly sampling or extracting keyframes from the input video, extracting global video features using a pre-trained video encoder; converting text queries into embedding vectors using a text encoder; determining the similarity matrix between global video features and text query features, and obtaining preliminary temporal candidate segments through similarity peak or threshold segmentation; and outputting these candidate segments as localization results all at once.

[0004] However, one-time temporal localization strategies rely solely on static similarity matching between global features and text queries, lacking the ability to perceive dynamic changes in the temporal context of videos. When faced with complex long videos, they are highly susceptible to noise interference, leading to the easy omission of key segments or significant boundary shifts. Furthermore, without a correction mechanism, the initial error is continuously amplified by subsequent steps, severely reducing the localization accuracy of understanding complex long videos. Summary of the Invention

[0005] Therefore, it is necessary to provide a multi-agent reinforcement learning optimization method and system for complex video understanding to address the aforementioned technical problems.

[0006] This invention provides a multi-agent reinforcement learning optimization method for complex video understanding, comprising: Retrieve the video to be processed and the query text; The video to be processed and the query text are input into a pre-trained video understanding model, which includes a planner, a localizer, a fine tuner, and a deep thinker connected in sequence. The planner performs task planning for the video to be processed and the query text, thereby decoupling the video understanding task into the localization stage and the inference stage, and obtaining a localization plan to guide the temporal localization of the video to be processed. Based on the localization plan, the visual features of the video to be processed and the text features of the query text are fused across modalities by the locator, and a timestamp-aware marker is generated to trigger temporal boundary decoding in order to initially determine the video temporal boundary related to the query text and obtain the initial temporal localization segment. Using the confidence score of the initial temporal localization segment as the reward signal, a multi-scale neighborhood search is performed on the initial temporal localization segment through a fine tuner to correct the boundary offset of the video segment in the localization stage, and the fine-tuned temporal localization segment is obtained. By using a deep thinker to perform deep reasoning on the finely tuned temporal location segments, causal reasoning can be performed using precisely located temporal evidence to obtain the final answer corresponding to the query text.

[0007] Optionally, training a video understanding model includes: Obtain the training sample set, which includes videos, query text, and corresponding time-series annotations and answer labels; The locator and fine tuner are trained in a supervised manner using the training sample set to obtain the trained locator and fine tuner. The training loss of the locator is obtained by weighting the foreground classification loss for supervised foreground classification, the L1 loss for supervised boundary regression, and the frame-query contrast loss for enhancing the alignment of video frames and query text. The deep thinker was cold-started with supervised fine-tuning and then optimized using temporal-aware reinforcement learning (T-GRPO) to obtain the trained deep thinker.

[0008] Optionally, the deep thinker undergoes supervised fine-tuning-based cold-start training, and time-aware reinforcement learning (T-GRPO) is used for policy optimization to obtain the trained deep thinker, specifically including: The deep thinker was fine-tuned in a supervised manner using a dataset containing image and video reasoning samples and long thought chain annotations to obtain a pre-trained deep thinker. In the policy optimization process of Temporally Aware Reinforcement Learning (T-GRPO), for the same query, two sets of inference outputs are obtained by using temporally ordered video frame sequences and randomly shuffled video frame sequences, respectively. A temporal comparison reward is assigned if and only if the inference accuracy of the ordered frame sequence group is greater than or equal to the inference accuracy of the shuffled frame sequence group; the temporal comparison reward is combined with the task correctness reward to determine the advantage function; the update magnitude of the policy is limited by the importance sampling ratio of the current policy and the old policy, and relative entropy is introduced as a regularization term to update the policy parameters. When both sets of reasoning outputs are correct and the length of each set of reasoning outputs is within the preset range, an additional length bonus is awarded. The rewards for T-GRPO reinforcement learning include task correctness rewards and temporal comparison rewards. The temporal comparison rewards are determined by comparing the inference accuracy of temporally ordered frame sequences with that of randomly shuffled frame sequences.

[0009] Optionally, a planner performs task planning on the video to be processed and the query text to decouple the video understanding task into a localization phase and an inference phase, resulting in a localization plan to guide the temporal localization of the video to be processed, specifically including: The query text is parsed into a target description for the localization phase and a question description for the reasoning phase, resulting in a localization plan containing temporal search constraints to guide the locator in searching for video segments related to the target description in the video to be processed.

[0010] Optionally, based on the localization plan, the locator performs cross-modal feature fusion on the visual features of the video to be processed and the textual features of the query text, and generates a timestamp-aware marker to trigger temporal boundary decoding, so as to initially determine the video temporal boundary related to the query text and obtain the initial temporal localization segment, specifically including: Learnable modal embeddings are added to the visual features of the video to be processed, sinusoidal positional encodings are added to the text features of the query text, and features are fused through Transformer to obtain cross-modal joint features; A timestamp-aware marker is generated on the cross-modal joint features. The start and end times corresponding to the time sequence boundaries are decoded based on the timestamp-aware marker to obtain the initial time sequence localization segment.

[0011] Optionally, using the confidence score of the initial temporal localization segment as the reward signal, a multi-scale neighborhood search is performed on the initial temporal localization segment through a fine-tuner to correct the boundary offset of the video segment during the localization stage, resulting in a fine-tuned temporal localization segment, specifically including: Centered on the initial temporal localization segment, neighborhood sampling is performed using a step size that includes multiple scales to generate multiple candidate segments; For each candidate segment, the confidence level is calculated using magnification verification, and a length penalty term based on the ratio of segment duration to total video duration is introduced to construct a fine-tuning objective function; The candidate segment that maximizes the fine-tuning objective function is used as the fine-tuned temporal localization segment, and the search process is controlled by budget constraints, step size decay, and confidence gain threshold.

[0012] Optionally, a deep thinker is used to perform deep reasoning on the refined temporal location segments to utilize precisely located temporal evidence for causal reasoning and obtain the answer corresponding to the query text. Specifically, this includes: Extract the video frame sequence within the finely tuned temporal localization segment, and input the video frame sequence and query text into a policy network trained by temporal-aware reinforcement learning. The policy network performs causal reasoning based on the temporal relationships in the video frame sequence, and obtains the answer corresponding to the reasoning process and the query text.

[0013] This invention provides a multi-agent reinforcement learning optimization system for complex video understanding, comprising: The data acquisition module is used to acquire the video to be processed and the query text; The data input module is used to input the video to be processed and the query text into the pre-trained video understanding model. The pre-trained video understanding model includes a planner, a locator, a fine tuner and a deep thinker connected in sequence. The task planning module is used to plan tasks for the video to be processed and the query text through the planner, so as to decouple the video understanding task into the localization stage and the inference stage, and obtain a localization plan to guide the temporal localization of the video to be processed. The localization module is used to perform cross-modal feature fusion of the visual features of the video to be processed and the text features of the query text through the locator based on the localization plan, and generate a timestamp-aware marker to trigger temporal boundary decoding in order to initially determine the video temporal boundary related to the query text and obtain the initial temporal localization segment. The fine-tuning module uses the confidence score of the initial temporal localization segment as the reward signal and performs a multi-scale neighborhood search on the initial temporal localization segment through the fine-tuner to correct the boundary offset of the video segment in the localization stage, thereby obtaining the fine-tuned temporal localization segment. The reasoning module is used to perform deep reasoning on the finely tuned temporal location segments through a deep thinker, so as to use the precisely located temporal evidence to perform causal reasoning and obtain the final answer corresponding to the query text.

[0014] The multi-agent reinforcement learning optimization method and system for complex video understanding provided in this invention have the following advantages compared with the prior art: This invention uses confidence level as the reward signal. The fine-tuner no longer relies on external one-time annotations, but instead judges the uncertainty of the initial localization result itself, giving the correction process a clear optimization direction. This technique enables multi-scale neighborhood search to systematically explore better solutions near the initial boundary at different time granularities. It can discover missed keyframes and remove redundant segments that are mistakenly included, effectively compensating for the local errors of the initial localizer under noise interference. It can solve the problems of one-time localization error accumulation, inference reliance on shallow visual shortcuts, and training instability in traditional methods, significantly improving the performance of the model in tasks such as long video temporal localization and video question answering. Attached Figure Description

[0015] Figure 1A comparison diagram of failure mechanisms of a multi-agent reinforcement learning optimization method for complex video understanding provided in one embodiment; Figure 2 This is a flowchart illustrating a multi-agent reinforcement learning optimization method for complex video understanding, provided in one embodiment. Figure 3 This is a qualitative reasoning effect diagram on a video question answering task provided in one embodiment of a multi-agent reinforcement learning optimization method for complex video understanding. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0017] This invention provides a multi-agent reinforcement learning optimization method for complex video understanding, such as... Figure 3 As shown, the method includes: Obtain the video to be processed and the query text.

[0018] The video to be processed and the query text are input into a pre-trained video understanding model, which includes a planner, a locator, a fine tuner, and a deep thinker connected in sequence.

[0019] The planner performs task planning for the video to be processed and the query text, thereby decoupling the video understanding task into a localization phase and an inference phase, resulting in a localization plan to guide the temporal localization of the video to be processed.

[0020] Based on the localization plan, the locator performs cross-modal feature fusion on the visual features of the video to be processed and the textual features of the query text, and generates a timestamp-aware marker to trigger temporal boundary decoding, so as to initially determine the video temporal boundary related to the query text and obtain the initial temporal localization segment.

[0021] Using the confidence score of the initial temporal localization segment as the reward signal, a multi-scale neighborhood search is performed on the initial temporal localization segment through a fine tuner to correct the boundary offset of the video segment in the localization stage, thereby obtaining the finely tuned temporal localization segment.

[0022] By using a deep thinker to perform deep reasoning on the finely tuned temporal location segments, causal reasoning can be performed using precisely located temporal evidence to obtain the final answer corresponding to the query text.

[0023] This invention provides a specific embodiment of a multi-agent reinforcement learning optimization method (LoVRR) for complex video localization and reasoning, comprising four core mechanisms: a multi-agent collaborative architecture, timestamp-aware temporal localization, VG-MSR multi-scale self-fine-tuning, and T-GRPO temporal-aware reinforcement learning. The multi-agent collaborative architecture decouples the video understanding process into four roles: planner, locator, fine-tuner, and deep thinker, achieving "first localizing evidence, then deep reasoning."

[0024] Planner: Receives the video to be processed and the query text, parses and maps the original query into a structured action plan, and generates a rewritten query that is more suitable for time-series retrieval (e.g., extracting event descriptions from complex problems).

[0025] Grounder: Based on the generation of specific tags ( <reg>A token is used to trigger the timestamp decoding task. Visual and textual features are fused by a Transformer and then used to predict frame-level probabilities and boundary offsets through a multi-scale temporal pyramid, using a classification head and a boundary regression head, to generate Top-N initial candidate temporal boundaries.

[0026] Refiner: Employs a 'zoom-inverification' mechanism, expanding candidate segments by 50% to both sides to expose true boundaries and inserting special boundary markers into the sequence.<SEG_START> and<SEG_END> Finally, the output fragment determines whether it contains the target event using binary logits, which are then converted into a confidence score.

[0027] DeepThinker: Receives finely tuned, precise evidence fragments and analyzes them in the output. <think>and <answer>Part of it performs deep thinking based on temporal logic, and finally outputs a reasoned answer with explanation.

[0028] The planner is trained using a strategy of independently training specific role data. Specifically, a LoRA adapter with a rank and alpha value of 64 is applied to the backbone model. In terms of training configuration, the global batch size is set to 32, the AdamW optimizer is used, and the learning rate for the planner is set to 2e-5. Overall training lasts for one epoch, with a linear warmup within the first 3% of the steps. The planner maps the original problem to structured action plans, which are represented as JSON-style call sequences in the implementation.

[0029] Positioner via <reg>The marker triggers timestamp decoding, transforming time-series boundary prediction into a timestamp-aware generation task.

[0030] VG-MSR uses the fine-tuner confidence as a reward signal to perform multi-scale neighborhood search and boundary iteration optimization on the initial fragment.

[0031] The Deep Thinker employs a two-stage training approach: supervised fine-tuning cold start and T-GRPO reinforcement learning. It also introduces temporal contrastive rewards to force the model to rely on temporal information rather than static visual features, while combining length rewards to achieve stable and efficient optimization.

[0032] As a preferred embodiment, such as Figure 2 As shown, the locator fuses video visual features with query text features. The feature fusion formula is as follows: ; In the formula: For video visual features, To query text features, , For learnable modal embedding, The location is encoded using sinusoidal positioning; the locator training loss is a weighted average of the foreground classification loss, boundary regression loss, and frame-query contrast loss. ; As a preferred embodiment, such as Figure 2 As shown, the VG-MSR fine-tuning objective function incorporates a length penalty to suppress fragment over-coverage: ; In the formula: For the confidence level of the segment, For the duration of the segment, Total video duration The penalty coefficient is used; a multi-scale step size {0.5Δ, Δ, 2Δ} and 8 local transformations are used to complete the neighborhood search, and risk control is achieved through budget constraints, step size decay, and gain threshold.

[0033] As a preferred embodiment, such as Figure 2 As shown, to guide the model in learning temporal logic, a temporal comparison reward is introduced: ; In the formula: For time-series comparison rewards, As the reward coefficient, For the inference accuracy of time-ordered frame sequences, The accuracy rate of randomly shuffling the sequence.

[0034] In this preferred embodiment, the total reward consists of a basic task reward and a time-series reward: ; In the formula: For the first i Total reward for each output Rewards will be given for correct answers and proper formatting. Rewards are based on time-series comparisons.

[0035] As a preferred embodiment, the advantage function is calculated from the total reward: ; In the formula: As the dominant value, This is the state value function.

[0036] In this preferred embodiment, the objective function for optimizing the TGRPO time-aware strategy is: ; ; In the formula: Importance sampling ratio, This is the cutting factor. Here are the KL divergence regularity coefficients. As the current strategy, This is a strategy for reference.

[0037] As a preferred embodiment, a length bonus is introduced to balance reasoning length and quality: ; In this preferred embodiment, the fine-tuning device employs amplified verification, and the output confidence level is: ; In this preferred embodiment, the reasoning process follows planning → positioning → fine-tuning → deep reasoning, and finally outputs the answer and time-series evidence fragments.

[0038] Another specific embodiment of the present invention is provided: This embodiment performs performance verification on multiple video benchmarks covering general video understanding, temporal localization and inference tasks, including MVBench, Video-MME, LVBench, CG-Bench, NExT-GQA, and Charades-STA.

[0039] This embodiment is trained based on the Qwen2-VL-7B-Instruct model, using eight NVIDIA L20 GPUs and a Chain-of-LoRA lightweight adapter. Both the localization module and the deep thinker employ a two-stage training strategy. During the training phase, video frames are sampled at a fixed frame rate; during the inference phase, VG-MSR multi-scale fine-tuning is used to improve temporal localization accuracy.

[0040] The experimental results of this example on video benchmark tests verify the effectiveness of LoVRR in video temporal localization and complex video inference tasks. The results are shown in Table 1.

[0041] Table 1 Performance Comparison of General Video Understanding on MVBBench As can be seen from Table 1, Example 1 is significantly more effective than other methods in video temporal localization and complex reasoning tasks.

[0042] To visually and intuitively evaluate the effectiveness of the method of the present invention, such as Figure 1 As shown, the failure mechanisms of the method of this invention and traditional one-time localization methods when facing boundary ambiguity are compared; at the same time, the invention is applied to a specific video question-answering task for qualitative analysis. The qualitative results show that this embodiment can accurately locate temporal evidence fragments and complete reliable reasoning based on temporal relationships.

[0043] Based on the same inventive concept, embodiments of the present invention also provide a multi-agent reinforcement learning optimization system for complex video understanding, the system comprising: The data acquisition module is used to acquire the video to be processed and the query text.

[0044] The data input module is used to input the video to be processed and the query text into the pre-trained video understanding model. The pre-trained video understanding model includes a planner, a locator, a fine tuner, and a deep thinker connected in sequence.

[0045] The task planning module is used to plan tasks for the video to be processed and the query text through a planner, so as to decouple the video understanding task into the localization stage and the inference stage, and obtain a localization plan to guide the temporal localization of the video to be processed.

[0046] The localization module is used to perform cross-modal feature fusion of the visual features of the video to be processed and the textual features of the query text through the locator based on the localization plan, and generate a timestamp-aware marker to trigger temporal boundary decoding in order to initially determine the video temporal boundary related to the query text and obtain the initial temporal localization segment.

[0047] The fine-tuning module uses the confidence level of the initial temporal localization segment as a reward signal and performs a multi-scale neighborhood search on the initial temporal localization segment through a fine-tuner to correct the boundary offset of the video segment during the localization stage, thereby obtaining the fine-tuned temporal localization segment.

[0048] The reasoning module is used to perform deep reasoning on the finely tuned temporal location segments through a deep thinker, so as to use the precisely located temporal evidence to perform causal reasoning and obtain the final answer corresponding to the query text.

[0049] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.< / reg> < / answer> < / think> < / reg>

Claims

1. A multi-agent reinforcement learning optimization method for complex video understanding, characterized in that, include: Retrieve the video to be processed and the query text; The video to be processed and the query text are input into a pre-trained video understanding model, which includes a planner, a locator, a fine tuner and a deep thinker connected in sequence. The planner performs task planning on the video to be processed and the query text to decouple the video understanding task into a localization stage and an inference stage, thereby obtaining a localization plan to guide the temporal localization of the video to be processed. Based on the positioning plan, the locator performs cross-modal feature fusion on the visual features of the video to be processed and the text features of the query text, and generates a timestamp-aware marker to trigger temporal boundary decoding, so as to initially determine the video temporal boundary related to the query text and obtain the initial temporal positioning segment. Using the confidence score of the initial temporal localization segment as the reward signal, the fine-tuner performs a multi-scale neighborhood search on the initial temporal localization segment to correct the boundary offset of the video segment in the localization stage, thereby obtaining the fine-tuned temporal localization segment. The deep thinker performs deep reasoning on the finely tuned temporal location segment to perform causal reasoning using precisely located temporal evidence, thereby obtaining the final answer corresponding to the query text.

2. The multi-agent reinforcement learning optimization method for complex video understanding as described in claim 1, characterized in that, Training the video understanding model specifically includes: Obtain a training sample set, which includes videos, query text, and corresponding time-series annotations and answer labels; The locator and the fine tuner are trained in a supervised manner using the training sample set to obtain the trained locator and the trained fine tuner; the training loss of the locator is obtained by weighting the foreground classification loss for supervised foreground classification, the L1 loss for supervised boundary regression, and the frame-query contrast loss for enhancing the alignment of video frames and query text. The deep thinker is subjected to supervised fine-tuning-based cold start training, and the policy is optimized using temporal-aware reinforcement learning (T-GRPO) to obtain the trained deep thinker.

3. The multi-agent reinforcement learning optimization method for complex video understanding as described in claim 1, characterized in that, The process of performing supervised fine-tuning-based cold-start training on the deep thinker and optimizing the policy using Temporal Reinforcement Learning (T-GRPO) to obtain the trained deep thinker specifically includes: The deep thinker is fine-tuned in a supervised manner using a dataset containing image and video reasoning samples and long thought chain annotations to obtain a pre-trained deep thinker. In the policy optimization process of Temporally Aware Reinforcement Learning (T-GRPO), for the same query, two sets of inference outputs are obtained by using temporally ordered video frame sequences and randomly shuffled video frame sequences, respectively. A temporal comparison reward is granted if and only if the inference accuracy of the ordered frame sequence group is greater than or equal to the inference accuracy of the shuffled frame sequence group; the temporal comparison reward is combined with the task correctness reward to determine the advantage function; the update magnitude of the policy is limited by the importance sampling ratio of the current policy and the old policy, and relative entropy is introduced as a regularization term to update the policy parameters; When both sets of reasoning outputs are correct and the length of each set of reasoning outputs is within the preset range, an additional length bonus is awarded. The reward for T-GRPO reinforcement learning includes the task correctness reward and the temporal comparison reward. The temporal comparison reward is determined by comparing the inference accuracy of the temporally ordered frame sequence with that of the randomly shuffled frame sequence.

4. The multi-agent reinforcement learning optimization method for complex video understanding as described in claim 1, characterized in that, The process involves using the planner to plan the task for the video to be processed and the query text, decoupling the video understanding task into a localization phase and an inference phase, and obtaining a localization plan to guide the temporal localization of the video to be processed. Specifically, this includes: The query text is parsed into a target description for the localization phase and a question description for the reasoning phase, resulting in a localization plan containing temporal search constraints, which guides the locator to search for video segments related to the target description in the video to be processed.

5. The multi-agent reinforcement learning optimization method for complex video understanding as described in claim 1, characterized in that, Based on the positioning plan, the locator performs cross-modal feature fusion on the visual features of the video to be processed and the textual features of the query text, and generates a timestamp-aware marker to trigger temporal boundary decoding, so as to initially determine the video temporal boundary related to the query text and obtain an initial temporal positioning segment, specifically including: Learnable modal embeddings are added to the visual features of the video to be processed, sinusoidal positional encodings are added to the text features of the query text, and feature fusion is performed through Transformer to obtain cross-modal joint features; A timestamp-aware marker is generated on the cross-modal joint features. The start and end times corresponding to the time sequence boundaries are decoded based on the timestamp-aware marker to obtain the initial time sequence localization segment.

6. The multi-agent reinforcement learning optimization method for complex video understanding as described in claim 1, characterized in that, The step of using the confidence score of the initial temporal localization segment as a reward signal and performing a multi-scale neighborhood search on the initial temporal localization segment through the fine-tuning unit to correct the boundary offset of the video segment during the localization stage, thereby obtaining a fine-tuned temporal localization segment, specifically includes: Centered on the initial temporal localization segment, neighborhood sampling is performed using a step size that includes multiple scales to generate multiple candidate segments; For each candidate segment, the confidence level is calculated using magnification verification, and a length penalty term based on the ratio of segment duration to total video duration is introduced to construct a fine-tuning objective function; The candidate segment that maximizes the fine-tuning objective function is selected as the fine-tuned temporal localization segment, and the search process is controlled by budget constraints, step size decay, and confidence gain threshold.

7. The multi-agent reinforcement learning optimization method for complex video understanding as described in claim 1, characterized in that, The process of performing deep reasoning on the finely tuned temporal location segment using the deep thinker, and using precisely located temporal evidence for causal reasoning to obtain the answer corresponding to the query text, specifically includes: Extract the video frame sequence within the finely tuned temporal localization segment, and input the video frame sequence and the query text into a policy network trained by temporal-aware reinforcement learning; The policy network performs causal reasoning based on the temporal relationships in the video frame sequence to obtain the answer corresponding to the reasoning process and the query text.

8. A multi-agent reinforcement learning optimization system for complex video understanding, characterized in that, include: The data acquisition module is used to acquire the video to be processed and the query text; The data input module is used to input the video to be processed and the query text into a pre-trained video understanding model. The pre-trained video understanding model includes a planner, a locator, a fine tuner, and a deep thinker connected in sequence. The task planning module is used to perform task planning on the video to be processed and the query text through the planner, so as to decouple the video understanding task into a localization stage and an inference stage, and obtain a localization plan to guide the video to be processed to perform temporal localization. The positioning module is used to perform cross-modal feature fusion on the visual features of the video to be processed and the text features of the query text through the locator based on the positioning plan, and generate a timestamp-aware marker to trigger temporal boundary decoding, so as to initially determine the video temporal boundary related to the query text and obtain an initial temporal positioning segment. The fine-tuning module is used to perform a multi-scale neighborhood search on the initial temporal positioning segment using the confidence score of the initial temporal positioning segment as a reward signal, so as to correct the boundary offset of the video segment in the positioning stage and obtain the fine-tuned temporal positioning segment. The reasoning module is used to perform deep reasoning on the finely tuned temporal location segment through the deep thinker, so as to use the precisely located temporal evidence to perform causal reasoning and obtain the final answer corresponding to the query text.

Images