A video timing positioning method based on attention consistency enhancement

Abstract

The application discloses a video time sequence positioning method based on attention consistency enhancement, comprising the following steps: first, sampling pretreatment and feature extraction are performed on a long video to obtain a plurality of video clips; and then, the plurality of video clips are input into a trained video time sequence positioning model based on attention consistency enhancement to realize accurate positioning. In the fine-tuning stage of the multi-modal large language model, an attention loss supervision function is introduced to enhance the consistency between the highlighted part of the attention graph video and the time segment to be positioned of the video, and an inference-time intervention method is used to enhance the consistency between the answer of the trained video time sequence positioning model and the highlighted part of the attention graph video, so that the attention consistency can be effectively enhanced while the time positioning accuracy is maintained, and the accuracy of time positioning is improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and video understanding, and relates to a video temporal localization method based on attention consistency enhancement. Background Technology

[0002] Faced with the challenges of retrieval and understanding brought about by the massive amounts of unedited videos in the new media era, Temporal Video Grounding (TVG) technology has become crucial and is developing rapidly. Among them, TVG methods based on multimodal large language models (Qwen, GPT, etc.) break through the limitations of traditional methods through three core advantages: ① Stronger multimodal semantic understanding of video text, which can deeply align natural language queries with video temporal features, parse fuzzy descriptions (such as "the look in the eyes changed from hesitation to determination") and accurately capture dynamic associations, resulting in a higher semantic matching accuracy than traditional methods; ② More convenient user interaction through question-and-answer methods, supporting natural language conversational queries (such as "when did the person in red start running"), and returning location results and context in real time; ③ Stronger zero-shot capability, which can handle new cross-domain scenarios (such as sports game-winning shots and film monologues) without fine-tuning. However, despite the significant advantages of the TVG method in multimodal large language models, its design generally neglects the issue of attention consistency. Attention consistency refers to the high degree of overlap between the semantic key information of the model-generated answer and the highlighted area (or weight distribution) focused by the attention mechanism, that is, the core content of the answer is consistent with the area where attention weights are significantly concentrated.

[0003] Currently, video temporal localization methods based on multimodal large language models are mainly divided into two categories: First, by preprocessing the video (such as adding video prompts, incorporating additional information, and other auxiliary means), the multimodal large language model is directly called for processing without additional training; Second, the multimodal large language model is fine-tuned and trained on a task-specific dataset to achieve video temporal localization.

[0004] Methods that directly call multimodal large language models in preprocessed videos have evolved to add visual cues to video frames, such as outlining objects and marking frame numbers. For example, Numpro (Wu, Yongliang, et al. "Numberit: Temporal grounding videos like flipping manga." Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.) adds unique numerical identifiers to video frames. Essentially, it transforms the continuous dynamic visual flow into a static "cartoon panel" with temporal markers: by using OCR-friendly fonts (such as red 40-point font marked in the lower right corner of the frame), each frame is assigned an incrementing sequence number (such as Frame 001, Frame 002, etc.), turning the originally disordered sequence of video frames into a "numbered panel" similar to the pages of a comic book—the panel order is the chronological order, and the number itself becomes a "natural anchor" between the visual content and the timeline. When processing numbered comics, multimodal large language models approach this type of content as intuitively as humans flipping through a comic book: understanding the visual content frame by frame in numerical order (e.g., "a character raises their hand in Frame 005" and "an object falls in Frame 012"), while simultaneously using built-in OCR capabilities to recognize numbers, thus binding the "number-visual information-time sequence" together. When a query to "locate the start and end times of an event" is received, the model only needs to match the corresponding number range of the event in its memory (e.g., "raising a hand" corresponds to Frames 005-008), and then map the number to the actual timestamp (e.g., 0.5 seconds per frame corresponds to 2.5 to 4 seconds), achieving a seamless transition from "visual understanding" to "temporal location." The ingenuity of this approach lies in replacing complex temporal modeling with the simplest "number equals sequence" logic, allowing the model to rely on existing sequence understanding and OCR capabilities to locate the content like "finding the plot in a comic book," preserving the temporal information of the video while significantly reducing the learning cost of temporal reasoning.

[0005] Another approach involves fine-tuning training directly on task-specific datasets. For example, LLaVA-MR (Lu, Weiheng, et al. "Llava-mr: Large language-and-vision assistant for videomoment retrieval." arXiv preprint arXiv:2411.14505(2024).) addresses the three core pain points in video temporal localization tasks: weak temporal awareness, missing keyframes, and context overload. It proposes an innovative framework based on a multimodal large language model. This framework achieves strong binding between video frames and precise timestamps through dense frame temporal coding, solving the problem of temporal information loss caused by traditional sparse sampling. It filters redundant static frames and retains keyframes such as action changes through information frame selection, ensuring that details of the target event are not lost. Dynamic token compression merges tokens for similar frames and retains complete tokens for keyframes, preserving key information while controlling the context length.

[0006] In general, while current video temporal localization methods have made progress in areas such as feature fusion and temporal modeling, they generally suffer from insufficient utilization of the intrinsic mechanisms of multimodal large language models. Firstly, existing research largely focuses on external feature alignment, failing to fully explore the dynamic evolution of attention maps during model inference. Secondly, the semantic relationship between attention maps and the final localization response lacks systematic analysis, resulting in the ineffective utilization of deep semantic constraints in cross-modal interactions. Thirdly, existing frameworks often treat multimodal large language models as black-box tools, neglecting the potential for adaptation to task characteristics and spatiotemporal awareness priors formed during self-supervised pre-training. Therefore, constructing a video temporal localization framework based on attention consistency constraints by dynamically calibrating visual-text attention distribution and enhancing the semantic interpretability of cross-modal interactions is of great significance. Summary of the Invention

[0007] To address the problem that existing video temporal localization methods based on multimodal large language models fail to fully utilize attention consistency capabilities due to neglecting the dynamic changes of the attention map during internal inference, this invention provides a video temporal localization method based on enhanced attention consistency. This method introduces an attention loss supervision function during the fine-tuning stage of the multimodal large language model to enhance the consistency between the highlighted parts of the model's attention map and the time segment to be located in the video. Furthermore, by using inference-time intervention, the consistency between the trained multimodal large language model's response and the highlighted parts of the attention map is enhanced. This effectively improves attention consistency while maintaining temporal localization accuracy, thereby increasing the overall accuracy of temporal localization.

[0008] The present invention is achieved by at least one of the following technical solutions.

[0009] A video temporal localization method based on attention consistency enhancement includes the following steps: First, the long video is sampled, preprocessed, and its features are extracted to obtain multiple video segments; Then, multiple video clips are input into a pre-trained video temporal localization model based on attention consistency enhancement to achieve accurate localization.

[0010] Furthermore, the sampling preprocessing includes extracting keyframes from the original video stream and uniformly scaling the resolution of each frame to pixels.

[0011] Furthermore, the training process of the video temporal localization model includes the following steps: (1) A subset is extracted from the training data proportionally to serve as the dataset for the decoding layer of the multimodal large language model; (2) Based on a multimodal large language model with pre-trained weights, inference is performed on a subset and attention maps of each decoding layer are extracted. After selecting key decoding layers, attention consistency loss function is applied for supervision. (3) Fine-tune the multimodal large language model with pre-trained weights on the training data.

[0012] Furthermore, the key decoding layer is selected as follows: First, the sequence is normalized to the [0,1] interval. Then, the mean and standard deviation of the normalized sequence are calculated. Subsequently, thresholding and connected component detection are performed. Elements in the sequence that are greater than or equal to the threshold are marked as 1 and the rest are marked as 0. The start and end points of the intervals with consecutive 1s are detected by using difference operations to obtain the initial high-scoring interval list. After identifying high-scoring intervals, adjacent intervals are merged, and the interval with the highest average score is selected as the prediction interval. The Temporal IoU between the prediction interval and the real interval is calculated and accumulated. After processing all videos at each layer, the average IoU is calculated, and finally, a list of average IoU for each layer is returned to measure the effect of attention of different layers on the localization of key intervals in the video. During training, decoding layers with attention performance higher than the average are selected, and attention consistency loss is added to them to optimize performance.

[0013] Furthermore, the attention consistency loss function Loss due to mean square error With variance regularization term The weighted average is calculated using the following formula:

[0014] in The number of samples contained in each training batch is the number of video-text pairs; For the model to the first iThe predicted output value of the sample, for a video temporal localization task, represents the _i_th sample. i Each sample is a video segment score derived from the model's attention distribution; For the first i The true label or target value of the nth sample, i.e., the nth i The true score of the video segment in each sample; This is a regularization hyperparameter used to control the degree of constraint. For the model to the first i The average of the predicted output values ​​of each sample is the average score of the video segment.

[0015] Furthermore, the weights of the attention consistency loss function are adjusted during the training process, and the fine-tuning is achieved through the following formula:

[0016] in the formula Let cross-entropy be the loss function. For the dataset, For input-output samples, Represents conditional probability. To output the sequence length, For the first t Bit Token, For prefix context, These are the model parameters.

[0017] Furthermore, the inference-time intervention phase of the video temporal localization model includes: 1) Select samples from the training dataset that meet the following conditions as positive samples: the peak frames of the attention scores of the video are located within the time range of the response segment and the target localization segment of the multimodal large language model. The remaining samples are used as negative samples. Finally, the effective samples are extracted from the training data in proportion to form a sample set. 2) Use the trained video localization model to infer the sample set, extract the last token feature of the inference result of each sample, and then calculate the average token features of the positive and negative samples respectively. The difference between the two is the offset correction vector. 3) Quantitatively evaluate the discrimination ability of each attention head in the multimodal large language model using a linear regression classifier, and add offset correction vectors to the attention heads with high accuracy.

[0018] The system for implementing the video temporal localization method based on attention consistency enhancement includes: The video temporal localization model decoding layer supervised selection module is used to select key layers and add attention consistency loss to optimize performance; The video temporal localization model fine-tuning training module is used to train the video localization model using parameter freezing and low-rank adaptive fine-tuning strategies.

[0019] A computer device according to the present invention includes a memory and a processor, the memory being electrically connected to the processor, the memory storing a computer program, which, when executed by the processor, causes the processor to implement the method described herein.

[0020] The present invention provides a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor implements the method described herein.

[0021] Compared with the prior art, the present invention has the following beneficial effects: The method proposed in this invention can effectively parse the inference relationships between different decoding layers and attention heads in multimodal large language models. Furthermore, the analysis method proposed in this invention can effectively solve the attention consistency problem in multimodal large language models and improve the localization performance of video temporal localization models, thereby promoting the application of video understanding algorithms in complex scenarios. Attached Figure Description

[0022] Figure 1 This is a diagram illustrating the implementation process of a video temporal localization framework based on attention consistency enhancement, and its system. Figure 2 This is a flowchart illustrating the implementation of decoding layer selection for a multimodal large language model based on attention maps. Figure 3 This is a flowchart illustrating the implementation process of fine-tuning training of some parameters in a video temporal localization model, as shown in the example. Figure 4 This is a flowchart illustrating the implementation of intervention during inference in a video temporal localization model, as shown in the example. Figure 5 This is a complete flowchart of long video temporal positioning in the embodiment. Detailed Implementation

[0023] The present invention will be further described in detail below through specific embodiments, but the embodiments of the present invention are not limited thereto.

[0024] The principle of this invention: While existing video temporal localization methods have initially demonstrated the powerful cross-modal reasoning potential of multimodal large language models, a key limitation remains in practical applications—they fail to delve into the dynamic relationship between the model's external responses and internal reasoning processes. The root cause of this problem lies in the fact that existing research generally neglects the dynamic evolution of the internal representations of multimodal large language models (especially the attention maps generated by different decoding layers and attention heads) as the input video content and query instructions change during answer generation. Specifically, when a multimodal large language model processes a video temporal localization task, its attention heads at each layer capture the correlation information of different time segments, spatial regions, and text queries in the video, forming attention maps with a specific distribution. Ideally, if a segment of video content is precisely the target to be localized, the model's higher layers or key attention heads should mark its corresponding time region as "highlighted." However, due to the lack of explicit modeling of the causal chain of "attention map highlighting pattern - final answer generation," the model often exhibits the phenomenon of "attention and answer misalignment": that is, although the highlighted area of ​​the attention map accurately covers the target video segment, when the model integrates multi-level features and generates the final time boundary, the output result may be inconsistent with the optimal segment indicated by the attention due to the loss of intermediate layer information, noise interference, or decision mechanism bias. To solve the attention inconsistency problem in video temporal localization tasks, this invention adopts a phased optimization strategy. In the fine-tuning training phase, a special attention-supervised loss function is introduced. This function aims to guide the model to learn so that the highlighted areas on the attention map generated by the model (i.e., the focus of the model's attention) can be more accurately aligned with the target predicted time segment. In the inference phase, an inference-time intervention technique is used to selectively select some key attention heads and apply small offset correction vectors to them. The purpose of this is to dynamically fine-tune the attention distribution to ensure that the predicted time segment output by the final model is consistent with the highlighted area indicated by its attention map. Through this training and inference co-optimization mechanism, the present invention not only effectively alleviates the phenomenon of attention drift or inconsistency, but also further utilizes the natural correspondence between attention focus and target time period, significantly enhancing the model's ability to accurately locate specific moments in videos.

[0025] like Figure 1 , Figure 5 As shown ( Figure 5 The video temporal localization model in the text represents a multimodal large language model trained with partial parameter fine-tuning. A video temporal localization method based on attention consistency enhancement includes the following steps: (1) To achieve refined preprocessing and feature extraction of long videos and transform them into standardized inputs that the model can process, a systematic processing flow is required. The processing quality at this stage directly determines the performance ceiling of the subsequent model. The specific steps are as follows: Temporal sampling: Experiments have verified that extracting keyframes from the original video stream using a sampling rate of 2 frames per second achieves the optimal balance between preserving the integrity of temporal information and controlling computational overhead. For example, a 30-second video can obtain 60 valid input frames through this sampling, which maintains the continuous perception of the video while effectively controlling computational complexity.

[0026] Spatial processing: The resolution of each frame is uniformly scaled down to 128×128 pixels. This size has been carefully evaluated to ensure that sufficient visual detail is preserved for video temporal localization while meeting the input specifications of the subsequent visual encoder.

[0027] Feature Extraction: The visual encoder and text encoder of the pre-trained multimodal large language model Qwen3-VL (Qwen3-VL is the latest generation of visual language large model launched by Alibaba's Tongyi Qianwen team at the Yunqi Conference in September 2025, aiming to achieve a key leap from "perception" to "cognition-execution" in multimodal AI. This model significantly improves long video understanding, multimodal reasoning, and cross-modal alignment capabilities through three core technologies: interleaved MRoPE, DeepStack, and textual timestamps) are used to extract video features from the spatially processed images.

[0028] (2) Accurate positioning is achieved based on the video temporal positioning model with attention consistency enhancement.

[0029] The video temporal localization model in this embodiment uses Qwen3-VL-8B as the multimodal backbone network, and achieves accurate localization through hierarchical temporal modeling and cross-modal alignment strategies. This model introduces attention consistency loss to improve attention alignment quality and enhance video localization performance through training optimization.

[0030] The training process of the video temporal localization model is as follows: like Figure 2As shown, for 20% of the samples in the Charades-STA dataset, high-score intervals are segmented by preprocessing the smoothed attention score sequence. The process is as follows: First, the sequence is normalized to the [0,1] interval, with the formula s_norm=(s -s_min) / (s_max - s_min + 1e-12) (s is the smoothed sequence, s_min / s_max are the minimum / maximum values ​​of the sequence, and 1e-12 is used to prevent division by zero); then, the mean μ of the normalized sequence s_norm is calculated using np.mean, and the standard deviation σ is calculated using np.std. The threshold uses the adaptive formula thr=μ + k*σ (default k=0, simplified to thr=μ); then, thresholding and connected component detection are performed—elements in the sequence ≥thr are marked as 1 (representing high-score regions), and the rest are marked as 0. The start and end points of continuous 1-value intervals are detected by np.diff difference operation to obtain the initial prediction interval list. After identifying the predicted intervals, if two intervals are separated by one video segment, they are considered adjacent. Adjacent intervals are merged, and the average attention score within each interval is calculated. The interval with the highest score is selected as the final predicted interval. The Temporal IoU between the predicted interval and the ground truth interval is calculated and accumulated. For 20% of the samples in the Charades-STA dataset, the average IoU is calculated after each layer of the decoding layer is processed. Finally, the average IoU of each layer is returned to measure the effect of different layers' attention on the localization of key video intervals. During training, model decoding layers with attention performance higher than the average (the overall mean of the average IoU of all decoding layers in the model) are selected, and attention consistency loss is added to them to optimize performance.

[0031] like Figure 3 As shown, this module aims to efficiently adapt to downstream localization tasks while retaining the multimodal capabilities of the pre-trained model. It employs a parameter freezing + low-rank adaptive (LoRA) fine-tuning strategy: First, all parameters of the video encoder and text encoder are frozen—since both have been pre-trained on large-scale multimodal data and possess mature visual-language joint representation capabilities, freezing avoids catastrophic forgetting and reduces training overhead; LoRA fine-tuning is performed only on the decoding layer parameters by introducing low-rank decomposition matrices (assuming the original parameters are W0∈Rd×k, and new low-rank matrices A∈Rd×r, B∈Rr×k, where r...). min(d,k) is the rank. By constraining the parameter update amount to ΔW=AB, training only A and B while fixing W0, the number of trainable parameters is reduced to less than 1% of the total fine-tuning while maintaining performance, significantly reducing memory usage and computational cost. Here, Rd represents a d-dimensional real vector space (or a set of vectors in the d-dimensional real field), used here to define the row dimensions of the original parameter matrix W0 and the low-rank matrix A (W0∈Rd×k, A∈Rd×r), where d is a positive integer corresponding to the dimension of the feature or output (such as the row dimension of the Transformer layer weight matrix). Rr represents an r-dimensional real vector space (or a set of vectors in the r-dimensional real field), used here to define the column dimensions of the low-rank matrix A and the row dimensions of the low-rank matrix B (A∈Rd×r, B∈Rr×k), where r is the rank (the core parameter of the low-rank constraint) much smaller than min(d,k). D is a positive integer representing the row dimension of the original parameter matrix W0 (which is also the row dimension of the low-rank matrix A), reflecting the dimensionality of the features or output (such as the dimension of the input features or the dimension of the layer output).

[0032] The specific fine-tuning process is achieved through the following formula:

[0033] in the formula To supervise the fine-tuning itself, the loss function supervision is needed, namely the cross-entropy loss function. For the dataset, For input-output samples, Represents conditional probability. To output the sequence length, For the first t Bit Token, For prefix context, These are the model parameters. An attention consistency loss function is then added to these parameters, as shown in the following formula:

[0034] in The number of samples contained in each training batch is the number of video-text pairs; For the model to the first i The predicted output value of the sample, for a video temporal localization task, represents the _i_th sample. i Each sample is a video segment score derived from the model's attention distribution; For the first i The true label or target value of the nth sample, i.e., the nth i The true score of the video segment in each sample; This is a regularization hyperparameter used to control the degree of constraint. For the model to the first iThe average of the predicted output values ​​of each sample is the average score of the video segment.

[0035] Attention Consistency Loss Function Loss due to mean square error With variance regularization term The weighted composition is used to optimize the attention distribution characteristics of the dual-objective collaborative optimization model.

[0036] Among them, mean square error loss The core objective is to ensure that the attention score of the video time segment to be localized output by the model's decoding layer reaches the highest level, that is, to force the highlighted area of ​​the attention map to highly overlap with the video time segment to be localized - this is equivalent to defining the "focus of attention" for the model, making the key time segment the absolute core of attention allocation, and avoiding localization deviation due to attention dispersion.

[0037] And the variance regularization term The uniformity of attention distribution within a specific time segment is then considered: it directly reflects the degree of fluctuation in attention scores within that segment. A smaller variance indicates that the attention scores are closer to the average and the changes are more gradual (i.e., "uniform change"). As a regularization term, the variance regularization term... By penalizing large fluctuations in attention scores, the model is forced to stabilize its attention distribution at localization moments, avoiding local over-focusing or under-focusing, thereby enhancing its ability to coherently perceive the overall semantics of time segments.

[0038] Combining the two, pass Anchoring to "where to focus", and then using variance regularization terms Standardize "how to focus" to ultimately achieve precise focusing and stable distribution of attention on target segments, thereby improving the accuracy and robustness of video temporal positioning.

[0039] The optimizer uses the AdamW algorithm, employing a cosine learning rate decay strategy during training to stabilize the convergence process. In one embodiment, the number of training epochs is set to 3, and the batch size and gradient accumulation steps are configured to achieve an effective batch size of 16. This entire setup balances training stability—AdamW's weight decay mechanism and cosine decay work together to suppress parameter oscillations, while by reasonably controlling the effective batch size, it balances computational efficiency and gradient estimation accuracy while fully utilizing GPU memory resources, avoiding memory overflow caused by excessively large batches or gradient noise caused by excessively small batches.

[0040] For the trained video temporal localization model, this method introduces inference-time intervention (ITI, such as...) Figure 4As shown, ITI (Interference-In-The-Field) dynamically optimizes the inference process, aiming to guide the model to focus its responses more on the highlighted parts of the attention map. ITI is a technique that specifically improves the performance of a particular task by adjusting the model's internal state or output during the inference phase (outside of training). Its core lies in applying controllable external interventions (such as adding offset vectors, adjusting weight distribution, and guiding attention focus) to key model components (e.g., attention heads, intermediate activation layers, or output logits) during inference, making the output more aligned with the target task requirements. Specifically, the interventions implemented during the inference process of this model are as follows: When selecting samples from the training dataset, the core criterion is "consistency between attention focus and target localization": samples where the peak video attention score frame (i.e., the single frame with the highest weight in the model's attention graph, representing the moment the model focuses on most) falls within both the multimodal large language model's answer segment (the time range corresponding to the text description generated by the model based on the query) and the target localization segment (the actual time range to be localized marked in the dataset) are defined as positive samples. These samples intuitively reflect the ideal state of "the attention highlighting area coincides with the target time." Other samples that do not meet this condition are classified as negative samples to capture counterexamples of attention shift or focus deviation. To ensure balanced data distribution and training efficiency, a valid sample set is finally extracted from the entire dataset according to a preset ratio (e.g., a 1:1 ratio of positive to negative samples or adjusted according to task difficulty) to avoid the interference of extreme distributions on subsequent intervention effects.

[0041] The trained multimodal large language model is used to infer from the selected dataset samples. The focus is on extracting the last token feature of each sample's inference result—the feature vector corresponding to the last token output by the model's decoding layer (this vector aggregates the model's comprehensive representation of the entire output sequence and is strongly correlated with the global judgment of the localization task). Subsequently, the average token features of the positive and negative sample sets are calculated separately (the average of the last token features of all positive samples is used to obtain the positive sample center vector, and the same applies to negative samples). The difference between these two vectors is the offset correction vector. The physical meaning of this vector is to capture the feature differences between the "ideal attention state of positive samples" and the "biased state of negative samples," providing a basis for targeted adjustments in subsequent interventions.

[0042] The discriminative ability of each attention head is quantitatively evaluated by constructing a Logistic Regression Probe. Attention heads with high accuracy are selected and offset correction vectors are added. This classifier is lightweightly connected to the output layer of different attention heads in the model. Token implicit features are used as training input to the Logistic Regression Probe, and the positive and negative labels of samples (positive sample = 1, negative sample = 0) are used as supervision signals to train its ability to distinguish between positive and negative samples. The evaluation metric is classification accuracy; higher accuracy indicates a stronger discriminative ability regarding whether attention is focused on the target moment (i.e., the head pays more attention to temporal information related to target localization). Finally, attention heads with high accuracy (such as those in the top 20% or above a set threshold) are selected and their previously obtained offset correction vectors are applied to achieve the goal of "precise intervention with highly sensitive attention heads."

[0043] During the inference intervention phase of the video temporal localization model, a linear regression classifier dynamically selects different attention heads for the decoding layer of the multimodal large language model. In the training phase, a layer-wise selection strategy for the decoding layer is further combined to form a multi-granularity attention mechanism optimization path from layer to head. This dual selection mechanism effectively captures the dynamic evolution characteristics of the model's internal attention map: at the layer level, by selecting decoding layers of different depths, it can adapt to the differentiated needs of video temporal modeling for long-range dependencies and local details; at the head level, by selecting attention heads with complementary semantic perception capabilities, it can enhance the model's ability to resolve complex temporal relationships.

[0044] During the training of the video temporal localization model, the target time segment is used as a supervision signal. By optimizing the attention mechanism, the model can more accurately focus on the target segment to be localized.

[0045] The design of these two stages enables the output time segments of the video temporal localization model to highly overlap with the highlighted parts of the attention map, which not only effectively solves the attention consistency problem, but also enhances the localization effect of the model.

[0046] The video temporal localization model designed using the method of this invention enables end-to-end automated processing of video temporal localization inference: for a 1-hour video, the inference time can be controlled within 5 minutes. Application deployment supports various real-world scenarios, such as rapid retrieval of key information in surveillance videos and summarization and generalization of educational videos.

[0047] In this embodiment, to verify the effectiveness of the attention map-based multimodal large language model decoding layer selection algorithm, a comparative experiment was conducted using Table 1. The experiment selected Qwen3-VL-2B and Qwen2.5-VL-3B as comparison models, using the unified prompt "When does the event '{}' happen? Answer in the format: 'From x seconds to y seconds'.", and compared the performance using the Charades-STA dataset as the evaluation benchmark.

[0048] Table 1. Video temporal localization results from the Charades-STA dataset.

[0049] Performance evaluation uses the R1@IoU metric, where R1@0.3, R1@0.5, and R1@0.7 correspond to the recall rate of the top-ranked segment in the prediction results when the IoU thresholds are 0.3, 0.5, and 0.7, respectively. A higher threshold indicates a more stringent requirement for temporal localization accuracy. Ours of this invention states that, for the comparison model, the original model's response is not directly used; instead, the output is obtained after processing by a multimodal large language model decoding layer selection algorithm based on attention maps.

[0050] As shown in Table 1, the algorithm demonstrates effectiveness across different model series, with improvements across all metrics, validating its applicability to decoding layer selection. This result indicates that the proposed method can not only serve as a practical strategy for decoding layer optimization but also provide new insights for subsequent invention design and research on model internal reasoning mechanisms.

[0051] As a specific implementation, this embodiment implements the method of the present invention on Qwen3-VL models with different parameter counts (2B, 8B), using a unified prompt word template: "When does the event '{}' happen? Answer in the format: 'From x seconds to y seconds.'", and uses the Charades-STA dataset as the evaluation benchmark for performance comparison analysis. Specifically, this method constructs a complete video temporal localization solution by introducing attention consistency loss optimization during the training phase and combining inference-time intervention strategies during the inference phase. Experimental results are shown in Table 2, systematically verifying the effectiveness of the method of the present invention under different model scales.

[0052] Table 2 Charades-STA video temporal localization results

[0053] Table 2 presents the experimental results of the Charades-STA video temporal localization task. "Ours" indicates the application of the complete method of this invention to the corresponding comparative model (covering the addition of attention consistency loss optimization during training and the introduction of inference intervention during the inference phase). This table compares the performance of the method of this invention (Ours) with the baseline model Qwen3_VL in terms of R1@0.3, R1@0.5, and R1@0.7 (these three represent the top-ranked recall at different intersection-union thresholds; higher values ​​indicate stronger video temporal localization accuracy) using model size (2B, 8B) as the dimension, providing a quantitative analysis.

[0054] A horizontal comparison at the same model size shows that when the model size is 2B, Ours improves R1@0.3, R1@0.5, and R1@0.7 compared to Qwen3-VL, from 0.4212, 0.2804, and 0.1261 to 0.8048, 0.6866, and 0.4591, respectively. When the model size is 8B, Ours also achieves an improvement in R1@0.7 from 0.2844 to 0.5333. Therefore, Ours significantly outperforms Qwen3-VL at all IoU thresholds.

[0055] A longitudinal comparison of the same method across different model sizes reveals that Qwen3-VL shows an increasing performance trend across all IoU thresholds as the model parameters increase from 2B to 8B. While Ours also achieves performance improvement with increasing model size, its improvement is weaker than that of Qwen3-VL_2B (e.g., the improvement of 8B compared to 2B). This phenomenon demonstrates that Ours has a more significant performance gain advantage in small model (2B) scenarios—even with a smaller number of model parameters, the method of this invention can effectively compensate for the model's inherent limitations through a complete solution of "attention consistency loss + inference-time intervention".

[0056] In summary, the experimental results show that the complete method proposed in this invention, which involves "adding attention consistency loss optimization during the training phase and introducing inference intervention during the inference phase," can significantly improve the performance of video temporal localization tasks under different model sizes (2B / 8B) and IoU thresholds (0.3 / 0.5 / 0.7), effectively verifying the effectiveness and universality of the method.

[0057] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A video temporal localization method based on attention consistency enhancement, characterized in that, Includes the following steps: First, the long video is sampled, preprocessed, and its features are extracted to obtain multiple video segments; Then, multiple video clips are input into a pre-trained video temporal localization model based on attention consistency enhancement to achieve accurate localization.

2. The video temporal localization method based on attention consistency enhancement according to claim 1, characterized in that, The sampling preprocessing includes extracting keyframes from the original video stream and scaling the resolution of each frame to pixels.

3. The video temporal localization method based on attention consistency enhancement according to claim 1, characterized in that, The training process of a video temporal localization model includes the following steps: (1) A subset is extracted from the training data proportionally to serve as the dataset for the decoding layer of the multimodal large language model; (2) Based on a multimodal large language model with pre-trained weights, inference is performed on a subset and attention maps of each decoding layer are extracted. After selecting key decoding layers, attention consistency loss function is applied for supervision. (3) Fine-tune the multimodal large language model with pre-trained weights on the training data.

4. The video temporal localization method based on attention consistency enhancement according to claim 3, characterized in that, The key decoding layer is selected as follows: First, the sequence is normalized to the [0,1] interval. Then, the mean and standard deviation of the normalized sequence are calculated. Subsequently, thresholding and connected component detection are performed. Elements in the sequence that are greater than or equal to the threshold are marked as 1 and the rest are marked as 0. The start and end points of the intervals with consecutive 1s are detected by using difference operations to obtain the initial high-scoring interval list. After identifying high-scoring intervals, adjacent intervals are merged, and the interval with the highest average score is selected as the prediction interval. The Temporal IoU between the prediction interval and the real interval is calculated and accumulated. After processing all videos at each layer, the average IoU is calculated, and finally, the average IoU list of each layer is returned to measure the effect of different layers' attention on the localization of key intervals in the video. During training, decoding layers with above-average attention performance are selected and attention consistency loss is added to them to optimize performance.

5. The video temporal localization method based on attention consistency enhancement according to claim 3, characterized in that, Attention Consistency Loss Function Loss due to mean square error With variance regularization term The weighted average is calculated using the following formula: in The number of samples contained in each training batch is the number of video-text pairs; For the model to the first i The predicted output value of the sample, for a video temporal localization task, represents the _i_th sample. i Each sample is a video segment score derived from the model's attention distribution; For the first i The true label or target value of the nth sample, i.e., the nth i The true score of the video segment in each sample; This is a regularization hyperparameter used to control the degree of constraint. For the model to the first i The average of the predicted output values ​​of each sample is the average score of the video segment.

6. The video temporal localization method based on attention consistency enhancement according to claim 3, characterized in that, The weights of the attention consistency loss function are adjusted during training, and the fine-tuning process is achieved through the following formula: in the formula Let cross-entropy be the loss function. For the dataset, For input-output samples, Represents conditional probability. To output the sequence length, For the first t Bit Token, For prefix context, These are the model parameters.

7. The video temporal localization method based on attention consistency enhancement according to claim 3, characterized in that, The inference-time intervention phase of the video temporal localization model includes: 1) Select samples from the training dataset that meet the following conditions as positive samples: the peak frames of the attention scores of the video are located within the time range of the response segment and the target localization segment of the multimodal large language model. The remaining samples are used as negative samples. Finally, the effective samples are extracted from the training data in proportion to form a sample set. 2) Use the trained video localization model to infer the sample set, extract the last token feature of the inference result of each sample, and then calculate the average token features of the positive and negative samples respectively. The difference between the two is the offset correction vector. 3) Quantitatively evaluate the discrimination ability of each attention head in the multimodal large language model using a linear regression classifier, and add offset correction vectors to the attention heads with high accuracy.

8. A system for implementing the video temporal localization method based on attention consistency enhancement as described in claim 1, characterized in that, include: The video temporal localization model decoding layer supervised selection module is used to select key layers and add attention consistency loss to optimize performance; The video temporal localization model fine-tuning training module is used to train the video localization model using parameter freezing and low-rank adaptive fine-tuning strategies.

9. A computer device comprising a memory and a processor, the memory being electrically connected to the processor, the memory storing a computer program, characterized in that: When the computer program is executed by the processor, it causes the processor to implement the method as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the processor implements the method as described in any one of claims 1 to 8.

Images