A Dynamic Multi-Scale Optimization Method for Multimodal Large Language Models in Long Videos

Abstract

This invention discloses a dynamic multi-scale optimization method for a multimodal large language model used in long videos, comprising the following steps: S1, acquiring an uncropped long video and a natural language query; S2, based on the long video and the natural language query, constructing a video thought tree reasoning process using a dynamic tree structure search paradigm, generating multiple candidate time segments through multi-scale sampling within the tree structure; S3, optimizing the candidate time segments through an iterative process of feedback diagnosis, multi-scale adjustment, and candidate re-ranking, generating a set of time segments corresponding to the semantics of the natural language query. This dynamic multi-scale optimization method for a multimodal large language model used in long videos significantly improves the temporal positioning accuracy of long videos by constructing a video thought tree and introducing a dynamic multi-scale sampling and feedback-driven iterative optimization mechanism.

Description

Technical Field

[0001] This invention relates to the field of multimodal perception and video understanding technology, specifically to a dynamic multi-scale optimization method for a multimodal large language model for long videos. Background Technology

[0002] The development of multimodal perception technology has driven the improvement of video understanding capabilities. Long video temporal localization, as one of the core tasks in this field, aims to accurately locate target video segments that are semantically aligned with natural language queries in application scenarios such as intelligent transportation and video surveillance.

[0003] In existing technologies, role-based video language frameworks have achieved certain processing results by decomposing complex tasks through modular workflows. However, such methods have significant limitations when processing long videos: on the one hand, input at a single scale cannot simultaneously take into account global contextual information and fine-grained temporal details, leading to deviations in temporal boundary positioning; on the other hand, static workflows lack effective feedback mechanisms and cannot correct initial positioning errors, affecting positioning accuracy.

[0004] To address these issues, some methods attempt to improve performance through time-series pooling or by introducing heavy modules. However, time-series pooling operations tend to ignore fine-grained time information, while the introduction of heavy modules significantly increases computational overhead, making it difficult to meet the dual requirements of efficiency and accuracy in practical applications.

[0005] The baseline framework VideoMind employs a role-based workflow and a Chain-of-LoRA role switching mechanism, achieving a good balance between efficiency and performance. However, this framework is still limited by a fixed single-scale input pattern and lacks an error correction feedback loop, making it difficult to meet the high-precision requirements of long-video temporal localization tasks. Summary of the Invention

[0006] To overcome the shortcomings of existing technologies, the present invention aims to provide a dynamic multi-scale optimization method for multimodal large language models in long videos. By constructing a video mind tree and introducing a dynamic multi-scale sampling and feedback-driven iterative optimization mechanism, the temporal positioning accuracy of long videos is significantly improved.

[0007] To achieve the objective of this invention, the following solution is adopted: A dynamic multi-scale optimization method for multimodal large language models in long videos includes the following steps: S1. Obtain the uncropped long video and natural language query; S2. Based on the long video and the natural language query, a video thinking tree reasoning process is constructed using a dynamic tree structure search paradigm, and multiple candidate time segments are generated in the tree structure through multi-scale sampling. S3. Through an iterative process of feedback diagnosis, multi-scale adjustment, and candidate re-ranking, the candidate time segments are optimized to generate a set of time segments corresponding to the semantics of the natural language query; wherein, the calculation formula for the set of time segments is: in, Represents a set of time segments. Indicates the first The start timestamp of a time segment Indicates the first The end timestamp of a time segment This indicates the total number of time segments.

[0008] Furthermore, prior to step S2, the process also includes constructing a Chain-of-LoRA agent architecture: Based on a frozen multimodal large language model backbone network, three sets of lightweight low-rank adaptation modules are trained as planners, localizers and validators, respectively. The planner is used to decompose the natural language query into an executable search plan, the locator is used to receive the current visual input and generate the candidate time segments, and the validator is used to evaluate the semantic alignment between the candidate time segments and the natural language query and diagnose whether the current visual resolution meets the confidence decision requirements.

[0009] Furthermore, the video mind tree reasoning process is defined as a tuple. ,in, The state space represents all possible reasoning nodes; Represents a set of reasoning actions; Indicates the state transition probability; This represents the set of state assessment results; The states in the state space are represented as follows: in, Indicates the first The state nodes of step-by-step reasoning This indicates the video segment to be processed in the current state. This represents the query context after the planner restatement. Indicates the sampling scale. This indicates the depth of the current state in the search tree; Furthermore, in step S2, the locator generates a set of candidate time segments based on the current state. The generation process is represented as follows: in, This indicates the number of candidate fragments generated in a single batch. Indicates the first One candidate time segment, Indicates the start timestamp of the candidate segment. Indicates the end timestamp of the candidate segment. This represents the locator function. This represents the backbone network of the multimodal large language model. This indicates the parameters of the low-rank adapter module corresponding to the locator.

[0010] Further, in step S3, the verifier uses a dual-signal evaluation mechanism to evaluate each candidate segment, obtaining a semantic matching confidence score and an uncertainty flag. The evaluation process is expressed as follows: in, This represents the semantic matching confidence score. Indicates uncertainty. Represents the validator function. This represents a video cropping function, based on candidate time segments. The time interval from the raw video data Extract the corresponding video clips.

[0011] Furthermore, step S3 also includes an active multi-scale refinement step: When the validator outputs a semantic matching confidence score satisfy If the candidate time segment is determined to be in a blurred state, a magnification operation is triggered, and the candidate time segment is resampled at a higher sampling scale to generate a new child node. The process is represented as follows: in, This indicates a low confidence threshold. This indicates a high confidence threshold. This represents the video resampling function. Indicates high sampling scale parameters; Switch to the locator and place the new child node. The corresponding video clips are re-run to generate corrected candidate time segments.

[0012] Furthermore, a depth-constrained depth-first search algorithm is used for reasoning, and the specific steps include: a. The planner parses the natural language query, initializes the root state, and pushes the root state onto the stack; b. Use a stack to store the states to be processed, and perform the following operations in a loop until the stack is empty: pop a current state from the stack, switch to the position where the locator generates candidate time segments in that current state. c. Switch to the validator to evaluate each candidate time segment, and record the best candidate time segment and the highest confidence level. ; d. If If so, output the result in advance; if If the current search depth has not reached the maximum depth, then multi-scale resampling is performed on the optimal candidate time segment and the generated new state is pushed onto the stack; e. After the iteration terminates, output the final determined optimal candidate time segment.

[0013] Furthermore, step S3 also includes candidate fusion and reordering steps: Candidate time segments generated at different sampling scales are fused across scales, and deduplication is performed by the intersection-union ratio threshold to select a preset number of candidate time segments. The verifier re-evaluates and sorts the selected candidate time segments to generate the final time location result.

[0014] Furthermore, the confidence high threshold The confidence threshold is set to 0.8. The maximum depth is set to 3, the crossover ratio threshold is set to 0.7, and the preset number of candidate time segments is 20.

[0015] Furthermore, the diagnostic mechanism of the verifier identifies defective time-localization results and triggers directional error correction in the dynamic tree structure search paradigm; And / or, through a dynamic multi-scale adjustment module that adaptively adjusts the sampling scale, complementary candidate time segments are generated to capture fine-grained temporal details in the video.

[0016] The dynamic multi-scale optimization method for multimodal large language models in long videos of this invention includes a Chain-of-LoRA agent architecture, a video thought tree reasoning process, an active multi-scale refinement mechanism, and a depth-constrained depth-first search algorithm, as detailed below: Chain-of-LoRA agent architecture: based on a single frozen MLLM backbone network Train three lightweight LoRA modules It implements planning, positioning, and verification functions respectively, and dynamically switches adapters during inference, taking into account both functional diversity and lightweight requirements.

[0017] in, This represents a frozen multimodal large language model backbone network (such as Qwen2-VL-2B), providing basic visual-language alignment and perception capabilities. This represents the parameters of the LoRA module in the planner, specifically designed to optimize complex query decomposition and search strategy formulation. This indicates the parameters of the LoRA module for the locator, optimizing time segment generation and accurate timestamp decoding. This represents the parameters of the LoRA module for validators, which optimizes the semantic matching evaluation and resolution diagnosis of candidate segments.

[0018] Video-ToT reasoning defines three core components: state space, thought generation, and state evaluation. Through iterative expansion of tree-structured nodes, it achieves multi-path reasoning and candidate optimization. The core formulas and symbols are as follows: The formula for generating candidate fragments is as follows: in, Grounder represents the set of candidate fragments generated by the locator, and Grounder represents the locator function that receives the current state. and model parameters Output candidate segments. This represents the current inference state node, which includes the video segment to be processed, the query context, the sampling scale, and the search depth. The condition assessment formula is as follows: in, This represents the semantic matching confidence score, ranging from [0,1]. A higher score indicates a closer match between the candidate and the query. This indicates uncertainty and is calculated from the output distribution entropy value. The higher the value, the more blurred the visual evidence (such as low resolution leading to a lack of detail). Indicates based on candidate fragments The time range from the original video A partial clip cut from the image. This indicates the query context that is appropriate for the current reasoning step.

[0019] Active multi-scale refinement mechanism: for fuzzy candidates labeled by validators The process involves resampling to generate high-fidelity input, correcting boundary errors, and capturing fine-grained visual cues. The formula for generating the new state through resampling is as follows: in, This represents a new inference state node, corresponding to a video segment after high-scale resampling. This represents the resampling function for candidate segments. Perform high frame rate or high resolution resampling. This indicates a high sampling scale, such as a frame rate of 1.5 FPS (high temporal scale) or a resolution of 40-80 px (high spatial scale). This indicates a low confidence threshold (0.3). The confidence threshold (0.8) is used to determine whether candidates need to be refined.

[0020] Depth-Constrained First-Solve (DFS) algorithm: Balances exploration breadth with computational efficiency, focusing resources on optimizing high-potential video segments and avoiding the high overhead of full-scale high-resolution processing. The key parameters are as follows: This indicates the maximum search depth (3), preventing excessive iteration from causing a surge in computational overhead; This represents the optimal candidate segment, which is the time segment with the highest semantic matching degree after multiple rounds of refinement and verification. The IoU threshold (0.7) is used for candidate segment deduplication, removing redundant candidates with a temporal overlap of more than 70%.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention overcomes the limitations of single-scale input. By employing a dynamic tree structure search paradigm to construct the video thought tree reasoning process and introducing a multi-scale sampling mechanism into the tree structure, the model can process video content at different sampling scales. This preserves the integrity of the global context while capturing fine-grained temporal details, effectively overcoming the temporal boundary positioning deviation problem caused by fixed single-scale input in existing technologies.

[0022] 2. This invention introduces a feedback-driven iterative optimization mechanism. Through an iterative process of feedback diagnosis, multi-scale adjustment, and candidate reordering, this invention continuously optimizes the initially generated candidate time segments, achieving proactive correction of the positioning results. This solves the technical defects of traditional static workflows, which lack error correction capabilities and struggle to correct initial positioning errors.

[0023] 3. This invention improves temporal positioning accuracy. Through the synergistic effect of the aforementioned dynamic tree structure search and multi-scale refinement strategy, this invention enables small-scale (2B) models to outperform larger-scale models such as GPT-4o in core temporal positioning metrics, significantly improving the accuracy of long-video temporal positioning tasks and meeting the practical needs for high-precision positioning in scenarios such as intelligent transportation and video surveillance.

[0024] 4. This invention balances computational efficiency and positioning performance. By refining blurred regions within a tree structure, this invention avoids full high-resolution processing of the entire long video, achieving precise positioning while maintaining lightweight advantages and optimizing the allocation efficiency of computational resources. Attached Figure Description

[0025] Figure 1 This is a flowchart of a dynamic multi-scale optimization method for a multimodal large language model for long videos, as described in an embodiment of the present invention. Figure 2 This is a schematic diagram comparing the static chain execution and dynamic self-refinement methods in an embodiment of the present invention; Figure 3 This is an overall flowchart of the Video-ToT intelligent agent framework in this embodiment of the invention; Figure 4 This is a schematic diagram of the multi-scale localization and cross-scale candidate fusion process in an embodiment of the present invention; Figure 5 This is a flowchart illustrating the execution of the depth-constrained first search algorithm in an embodiment of the present invention. Detailed Implementation

[0026] The present invention will now be further described in conjunction with the accompanying drawings and specific embodiments. It should be noted that, without conflict, the various embodiments or technical features described below can be arbitrarily combined to form new embodiments.

[0027] This invention provides a dynamic multi-scale optimization method for multimodal large language models used in long videos. It addresses the core issues of single-scale input and lack of feedback mechanisms through a feedback-driven multi-scale refinement strategy, achieving both accurate and lightweight temporal localization in long videos. This method is applicable to scenarios such as intelligent transportation, surveillance, and multimedia retrieval.

[0028] like Figure 1-5 As shown in the figure, the dynamic multi-scale optimization method for multimodal large language models for long videos according to an embodiment of the present invention includes the following steps: S1. Obtain the uncropped long video and natural language query; S2. Based on the long video and the natural language query, a video thinking tree reasoning process is constructed using a dynamic tree structure search paradigm, and multiple candidate time segments are generated in the tree structure through multi-scale sampling. S3. Through an iterative process of feedback diagnosis, multi-scale adjustment and candidate re-ranking, the candidate time segments are optimized to generate a set of time segments corresponding to the semantics of the natural language query.

[0029] The following is a detailed description of the dynamic multi-scale optimization method for multimodal large language models used in long videos according to embodiments of the present invention.

[0030] The present invention provides a dynamic multi-scale optimization method for multimodal large language models (Video-ToT) for long videos. This method focuses on the temporal localization task of long videos and requires uncropped long videos. and natural language query Determine the set of time segments corresponding to the query semantics. Employing a dynamic tree-structured search paradigm, and through an iterative process of feedback diagnosis, multi-scale adjustment, and candidate re-ranking, precise optimization of time positioning is achieved. This includes the set of time segments. The calculation formula is: in, This represents the raw, uncropped data of a long video. This represents a natural language query used to describe the video content to be located. This represents a set of time segments containing all target video segments aligned with the query semantics. Indicates the first The start timestamp of each target segment. Indicates the first The end timestamp of each target segment This indicates the total number of target segments.

[0031] Furthermore, the dynamic multi-scale optimization method for multimodal large language models for long videos in this embodiment of the invention adopts a Chain-of-LoRA agent architecture, based on a single frozen multimodal large language model (MLLM) backbone network. Three lightweight low-rank adaptation (LoRA) modules were trained. These correspond to the three cognitive roles of planning, localization, and verification, respectively. During inference, functional switching is achieved through dynamic adapter switching, without the need to reload backbone network parameters. Planner Break down complex queries into executable search plans and decide on a global scan or targeted search strategy; Positioner : Receives the current visual input to generate candidate time segments, and is equipped with a timestamp decoder header to output continuous coordinates; Validator : Evaluate the semantic alignment between candidate fragments and the query, and diagnose whether the current visual resolution meets the confidence decision requirements; in, This represents a frozen multimodal large language model backbone network (such as Qwen2-VL), providing basic multimodal perception capabilities. This represents the LoRA adaptation module parameters corresponding to the planner, used to optimize query decomposition and search strategy planning functions. This indicates the parameters of the LoRA adaptation module corresponding to the locator, used to optimize the candidate time segment generation and timestamp decoding functions. This represents the LoRA adaptation module parameters corresponding to the validator, used to optimize candidate fragment evaluation and resolution diagnostic functions.

[0032] Furthermore, in the process of constructing the Video-ToT reasoning tree, a tuple is defined and represented as follows: in, The state space of the video mind tree contains all possible reasoning nodes. This represents a set of reasoning actions, including actions such as "Zoom-in" and "Verify". This represents the state transition probability, describing the probability of transitioning from one state to another through a reasoning action. This represents the set of state evaluation results, including the confidence and uncertainty signals output by the validator; Continuous spatiotemporal domain for video understanding: 1. State Space :state The following formula is used to represent the nodes of the reasoning tree: in, Indicates the first The state nodes of step-by-step reasoning This indicates the video segment to be processed in the current state (which can be a complete video or a cropped segment). This represents the query context after the planner restatement. This indicates the sampling scale (including frame rate (fps) and spatial resolution (px)). Indicates the depth of the current state in the search tree; root state The corresponding complete video at the coarse sampling rate; 2. Thought Generation (Location): The locator is based on the current state. generate Candidate fragments Each candidate is defined as a time interval. During training, the positioning accuracy is ensured by minimizing the boundary regression loss. The formula used in the generation process is as follows: in, This indicates the number of candidate fragments generated in a single batch. Represents the set of candidate segments. Indicates the first One candidate segment, Indicates the start timestamp of the candidate segment. Indicates the end timestamp of the candidate segment. This represents the locator function, which takes the current state and model parameters as input and outputs a set of candidate segments. 3. State Assessment (Diagnosis): The validator employs a dual-signal assessment mechanism, including semantic matching confidence score. (Probability of matching fragment with query) and uncertainty flag (Output distribution entropy value, high entropy indicates blurred visual evidence), the formula used in the evaluation process is as follows: in, This represents the semantic matching confidence score, with a value range of [0,1]. A higher value indicates a higher semantic match between the candidate fragment and the query. This indicates uncertainty and is calculated from the entropy value of the validator output distribution. A higher value indicates that the visual evidence is more ambiguous. This represents the validator function, which takes the cropped video segment, query context, and model parameters as input and outputs an evaluation signal. This represents a video cropping function, based on candidate segments. The time range from the original video Extract the corresponding segment from it. This represents the raw video data.

[0033] Furthermore, the dynamic multi-scale optimization method for multimodal large language models for long videos in this embodiment of the invention introduces an active multi-scale refinement mechanism, when the validator marks candidate segments as "fuzzy". When this is triggered, a zoom-in operation is performed: returning to the original video source. For candidate fragments Resample at a higher frame rate or spatial resolution to generate new child nodes. The formulas used in the process are as follows: in, This represents the low confidence threshold, with a value of 0.3, used to determine whether a candidate segment is in a "fuzzy" state. This represents the high confidence threshold, with a value of 0.8, used to determine whether a candidate fragment is in a "confidence match" state. Indicates candidate fragments The start timestamp (and) (meaning is consistent) Indicates candidate fragments End timestamp (and) (meaning is consistent) Indicates the first The state node of the step-by-step reasoning is the current state. child nodes, This represents a video resampling function that takes the original video, candidate segments, and target scale as input and outputs a high-fidelity video segment. This indicates high sampling scale parameters, including high frame rate (e.g., 1.5 FPS) and high spatial resolution (e.g., 40-80 px). The locator re-runs on high-fidelity input, correcting boundary errors caused by low resolution and capturing momentary visual cues; if the validator refuses to refine the candidate, it backtracks to explore other branches.

[0034] Furthermore, the dynamic multi-scale optimization method for multimodal large language models in long videos according to embodiments of the present invention employs a depth-constrained depth-first search (DFS) algorithm to balance exploration breadth and computational efficiency. The reasoning process includes: (1) Planner parsing query Initialize root state (Full video + low resolution); (2) The stack structure stores the state to be processed, and the current state is popped during iteration and switched to the LoRa locator to generate candidate fragments; (3) Switch to the validator LoRA to evaluate each candidate and record the best candidate and the highest confidence level; (4) If confidence level Then output the result in advance; if And it did not reach the maximum depth. If so, multi-scale resampling is performed and the new state is pushed onto the stack; (5) Output the optimal candidate after the iteration terminates. ; in, The semantic matching confidence score of the candidate segment (and) (meaning is consistent) This represents the maximum depth of the search tree, with a value of 3, used to limit the number of iterations for refinement. This represents the optimal time segment selected in the final filtering, i.e., the video segment that best matches the query semantics.

[0035] Furthermore, set thresholds for key parameters: high confidence threshold. Low threshold Maximum search depth Candidate deduplication uses an IoU threshold of 0.7, and the validator evaluates the Top-20 candidates across scales. The multi-scale configuration includes standard scale, high temporal scale (to improve frame rate), and high spatial scale (to improve resolution). The final result is generated through cross-scale candidate fusion, IoU deduplication, and validator reordering. in, This represents the high confidence threshold (0.8). If the score exceeds this value, it is considered a confidence match, and the result is output directly. This represents a low confidence threshold (0.3). Candidates below this value are considered invalid and are pruned. The maximum search depth (3) is defined, which limits the maximum number of iterations for refinement. IoU (Intersection over Union) represents the intersection-over-union ratio, which measures the temporal overlap between two candidate segments. The threshold of 0.7 is used to filter non-redundant candidates. Top-20 represents the upper limit of the number of candidates evaluated by the validator, which means selecting the top 20 candidates from all cross-scale candidates for reordering.

[0036] Furthermore, the dynamic multi-scale optimization method for multimodal large language models for long videos in this embodiment of the invention employs a candidate diagnostic mechanism, which can identify defective temporal localization results and trigger targeted error correction without additional overhead; the dynamic multi-scale adjustment module generates complementary candidates to capture fine-grained temporal details by adaptively adjusting the sampling rate and resolution.

[0037] The dynamic multi-scale optimization method for multimodal large language models in long videos according to embodiments of the present invention has the following advantages: 1. This invention solves the limitations of single-scale input in long video temporal localization by using a feedback-driven multi-scale refinement strategy, significantly improving boundary localization accuracy. The 2B model outperforms larger-scale models such as GPT-4o in temporal localization metrics. 2. This invention adopts a Chain-of-LoRA architecture to maintain the advantage of lightweight design, avoids additional computational overhead, and can take into account both global context and fine-grained details without processing a complete high-resolution long video. 3. This invention adopts a dynamic tree structure search and active perception paradigm, enabling the model to adaptively allocate computing resources, accurately refine fuzzy areas, and improve the robustness of long video understanding in complex scenarios. 4. This invention employs a dual-signal evaluation mechanism and an iterative feedback loop to effectively correct initial positioning errors and reduce the risk of misjudgment caused by low-confidence predictions. It is applicable to practical application scenarios such as intelligent transportation and monitoring.

[0038] In one optional embodiment, the performance of the invention is validated on four benchmark datasets covering long video temporal localization and comprehensive understanding tasks, including CG-Bench, VideoMME, MLVU, and LVBench. CG-Bench is a grounded video question-answering benchmark with an average duration of 27 minutes, and temporal localization accuracy is evaluated using mIoU, R@IoU, and A@IoU metrics. The other three are general long video understanding benchmarks, covering diverse video lengths and inference challenges.

[0039] The experimental environment was built based on the VideoMind framework, using Qwen2-VL-2B as the backbone MLLM (i.e., ), directly reuse its pre-trained role-specific LoRA module ( No additional training was performed; the feedback-driven multi-scale refinement mechanism of this invention was integrated only during the inference phase. The experimental hardware consisted of a single server equipped with an NVIDIA A100 GPU. During the training phase, the resolution of each video frame was set to 36-64px (standard scale), 32-56px (high temporal scale), and 40-80px (high spatial scale). During the inference phase, the resolution configuration remained unchanged, and multi-scale switching was achieved through dynamic resampling.

[0040] The specific implementation steps are as follows: 1. Parameter configuration initialization: Set the core parameter thresholds and the high confidence threshold. Low threshold Maximum search depth The candidate deduplication IoU threshold is 0.7, and the validator evaluates the Top-20 candidates across scales. The multi-scale sampling parameter configuration is shown in the table below: Table 1 Multi-scale sampling parameter configuration table 2. Multi-scale candidate generation and fusion: The localizers share weights and process the video at three scales to generate candidate segments (Z); for example... Figure 4 As shown, IoU-based deduplication (threshold 0.7) is performed on all cross-scale candidates. After removing redundant segments, the Top-20 candidates are retained and enter the fusion stage. The global validator loads high-resolution video segments to re-score the candidates and sorts them by confidence to generate the final candidate list.

[0041] 3. Reasoning Flow Execution: Input a long video v and a natural language query Q (e.g., "Where in the video is the scene where the protagonist drinks red wine and eats Auvergne pork tenderloin?"). Figure 3 As shown, the planner parses the query and generates a search plan; in the root state s0 (complete video + standard scale), the locator generates initial candidates (e.g., 1.24 minutes), and the validator evaluates their confidence. (between) and If the fragment is between [a certain value], it is identified as a "fuzzy" candidate, triggering a magnification operation; the candidate fragment is then resampled at a high temporal scale (1.5fps) using a formula to generate a new state. The formulas used are as follows: in, This represents a new inference state node, corresponding to a video segment after high-scale resampling. This represents the resampling function for candidate segments. Perform high frame rate or high resolution resampling. This indicates a high sampling scale, such as a frame rate of 1.5 FPS (high temporal scale) or a resolution of 40-80 pixels (high spatial scale). This indicates a low confidence threshold (0.3). The confidence threshold (0.8) is used to determine whether candidates need to be refined. Positioner in The candidate is then regenerated and refined (e.g., at 1.43 minutes), and the validator evaluates the confidence level. (higher than) This confirms that the candidate is the optimal result. The final time segment is output by the responder.

[0042] 4. Performance Verification and Comparative Analysis: The performance of this embodiment on various benchmark datasets is shown in the table below, and it is also compared with existing mainstream models: Table 2. Performance Comparison of Each Model on Long Video Benchmark Dataset As shown in the table above, the 2B model of this invention significantly outperforms the baseline model VideoMind in the core temporal localization metrics (mIoU, R@IoU) of CG-Bench, and surpasses larger-scale models such as ShareGPT4Video and Qwen2-VL. On the general long video understanding benchmark, the overall score is higher than that of the baseline model, which verifies the synergistic improvement effect of this invention in temporal localization accuracy and overall video understanding capability.

[0043] To further verify the effectiveness of the core module, this embodiment conducted ablation experiments for multi-scale configuration, IoU threshold, and validator range. The results are shown in the table below: Table 3 Ablation Experiment Results of Core Module The ablation experiment results show that the integrity of the multi-scale configuration, a reasonable IoU threshold (0.7) and a sufficient validator evaluation range (Top-20) are the key to achieving optimal performance, further demonstrating the rationality and effectiveness of the technical solution of this invention.

[0044] In one optional embodiment, the focus is on long-video temporal localization tasks in intelligent transportation scenarios to verify the performance of the invention in practical applications. Experimental data uses real road surveillance videos (single segment length 30-60 minutes), including typical events such as vehicles illegally changing lanes and pedestrians crossing the road. A query example is "Where in the video between 10:00 and 11:00 is the segment where vehicle with license plate number XXX illegally crossed the solid line?".

[0045] The experimental setup is consistent with the aforementioned embodiments, such as... Figure 5 The inference process shown, through the dynamic multi-scale refinement mechanism of this invention, involves the locator first rapidly scanning the entire video at a standard scale, identifying the 10:23-10:25 minute segment as a suspected segment (confidence level). The validator diagnoses that the license plate details are blurred due to insufficient frame rate, triggering a high-time-scale (1.5fps) resampling; after refinement, the locator outputs a precise segment from 10:23:45 to 10:24:12, and the validator evaluates the confidence level. Ultimately, this allowed for the accurate identification of the violations.

[0046] According to statistics, the time positioning accuracy of this embodiment reached 89.7% in 100 test videos, with an average positioning error of ≤2.3 seconds. Compared with the baseline model VideoMind (accuracy of 76.3%, average error of ≤4.8 seconds), the positioning accuracy and efficiency are significantly improved, meeting the actual application needs of intelligent transportation scenarios.

[0047] This invention overcomes the limitations of single-scale and static process defects of existing methods through a dynamic multi-scale adjustment and iterative feedback mechanism, providing a lightweight and high-precision solution for long-video multimodal perception, which can be widely used in intelligent transportation, video surveillance, multimedia retrieval and other fields.

[0048] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A dynamic multi-scale optimization method for multimodal large language models in long videos, characterized in that, Includes the following steps: S1. Obtain the uncropped long video and natural language query; S2. Based on the long video and the natural language query, a video thinking tree reasoning process is constructed using a dynamic tree structure search paradigm, and multiple candidate time segments are generated in the tree structure through multi-scale sampling. S3. Through an iterative process of feedback diagnosis, multi-scale adjustment, and candidate re-ranking, the candidate time segments are optimized to generate a set of time segments corresponding to the semantics of the natural language query; wherein, the calculation formula for the set of time segments is: in, Represents a set of time segments. Indicates the first The start timestamp of a time segment Indicates the first The end timestamp of a time segment This indicates the total number of time segments.

2. The method for dynamic multi-scale optimization of multimodal large language models for long videos according to claim 1, characterized in that, Before step S2, the process also includes building the Chain-of-LoRA agent architecture: Based on a frozen multimodal large language model backbone network, three sets of lightweight low-rank adaptation modules are trained as planners, localizers and validators, respectively. The planner is used to decompose the natural language query into an executable search plan, the locator is used to receive the current visual input and generate the candidate time segments, and the validator is used to evaluate the semantic alignment between the candidate time segments and the natural language query and diagnose whether the current visual resolution meets the confidence decision requirements.

3. The method for dynamic multi-scale optimization of multimodal large language models for long videos according to claim 2, characterized in that, The video mind tree reasoning process is defined as a tuple. ,in, The state space represents all possible reasoning nodes; Represents a set of reasoning actions; Indicates the state transition probability; This represents the set of state assessment results; The states in the state space are represented as follows: in, Indicates the first The state nodes of step-by-step reasoning This indicates the video segment to be processed in the current state. This represents the query context after the planner restatement. Indicates the sampling scale. This indicates the depth of the current state in the search tree.

4. The method for dynamic multi-scale optimization of multimodal large language models for long videos according to claim 3, characterized in that, In step S2, the locator generates a set of candidate time segments based on the current state. The generation process is represented as follows: in, This indicates the number of candidate fragments generated in a single batch. Indicates the first One candidate time segment, Indicates the start timestamp of the candidate segment. Indicates the end timestamp of the candidate segment. This represents the locator function. This represents the backbone network of the multimodal large language model. This indicates the parameters of the low-rank adapter module corresponding to the locator.

5. The method for dynamic multi-scale optimization of multimodal large language models for long videos according to claim 4, characterized in that, In step S3, the verifier uses a dual-signal evaluation mechanism to evaluate each candidate segment, obtaining a semantic matching confidence score and an uncertainty flag. The evaluation process is expressed as follows: in, This represents the semantic matching confidence score. Indicates uncertainty. Represents the validator function. This represents a video cropping function, based on candidate time segments. The time interval from the raw video data Extract the corresponding video clips.

6. The method for dynamic multi-scale optimization of multimodal large language models for long videos according to claim 5, characterized in that, Step S3 also includes an active multi-scale refinement step: When the validator outputs a semantic matching confidence score satisfy If the candidate time segment is determined to be in a blurred state, a magnification operation is triggered, and the candidate time segment is resampled at a higher sampling scale to generate a new child node. The process is represented as follows: in, This indicates a low confidence threshold. This indicates a high confidence threshold. This represents the video resampling function. Indicates high sampling scale parameters; Switch to the locator and place the new child node. The corresponding video clips are re-run to generate corrected candidate time segments.

7. The method for dynamic multi-scale optimization of multimodal large language models for long videos according to claim 6, characterized in that, The inference is performed using a depth-constrained depth-first search algorithm, and the specific steps include: a. The planner parses the natural language query, initializes the root state, and pushes the root state onto the stack; b. Use a stack to store the states to be processed, and perform the following operations in a loop until the stack is empty: pop a current state from the stack, switch to the position where the locator generates candidate time segments in that current state. c. Switch to the validator to evaluate each candidate time segment, and record the best candidate time segment and the highest confidence level. ; d. If If so, output the result in advance; if If the current search depth has not reached the maximum depth, then multi-scale resampling is performed on the optimal candidate time segment and the generated new state is pushed onto the stack; e. After the iteration terminates, output the final determined optimal candidate time segment.

8. The method for dynamic multi-scale optimization of multimodal large language models for long videos according to claim 7, characterized in that, Step S3 also includes candidate fusion and reordering steps: Candidate time segments generated at different sampling scales are fused across scales, and deduplication is performed by the intersection-union ratio threshold to select a preset number of candidate time segments. The verifier re-evaluates and sorts the selected candidate time segments to generate the final time location result.

9. The method for dynamic multi-scale optimization of multimodal large language models for long videos according to claim 8, characterized in that, The high confidence threshold The confidence threshold is set to 0.

8. The maximum depth is set to 3, the crossover ratio threshold is set to 0.7, and the preset number of candidate time segments is 20.

10. The method for dynamic multi-scale optimization of multimodal large language models for long videos according to any one of claims 1-9, characterized in that, The diagnostic mechanism of the verifier identifies defective time-localization results and triggers directional error correction in the dynamic tree structure search paradigm. And / or, through a dynamic multi-scale adjustment module that adaptively adjusts the sampling scale, complementary candidate time segments are generated to capture fine-grained temporal details in the video.

Images