Video temporal localization method based on multimodal large model prior semantic relationship alignment

Abstract

The application discloses a video time sequence positioning method based on multimodal large model prior semantic relationship alignment, constructs a video time sequence positioning model, respectively extracts text features and video features from a natural language query and a video, obtains interactive video features after alignment and interaction, extracts time sequence context features from the video features by a context reasoning module, fuses the time sequence context features and the interactive video features to obtain enhanced features, extracts time sequence features from the enhanced features, generates a segment relationship attention matrix based on the enhanced features, weights the time sequence features, and predicts the start and end time of a target segment based on the weighted time sequence features; the model is trained by using a training sample set, a loss term of a loss function is set by using a multimodal large model prior semantic relationship alignment strategy during the training process, and after the training is completed, time sequence positioning can be performed on an input video. The application can effectively improve the positioning accuracy and robustness of video time sequence positioning.

Description

Technical Field

[0001] This invention belongs to the field of video temporal localization technology, and more specifically, relates to a video temporal localization method based on prior semantic relationship alignment of a multimodal large model. Background Technology

[0002] Existing video temporal localization techniques typically take uncropped video and natural language queries as input. They first extract segment-level and word-level features using pre-trained and often frozen visual and text encoders, then perform semantic matching within a unified representation space through a cross-modal interaction module, and finally output start and end timestamps from a boundary regression head. Typical implementation paths fall into two main categories: one is the candidate proposal paradigm, which generates a large number of segment proposals on the timeline based on sliding windows, anchor boxes, or candidate sets, and scores and ranks the proposals using cross-modal attention or matching networks; the other is the no-proposal paradigm, which directly performs regression prediction on the target span in an end-to-end framework to reduce candidate redundancy and inference overhead. In recent years, DETR-like detection-based Transformers have been introduced into this task, replacing manual proposal components with set prediction and query-driven segment detection mechanisms to achieve a more unified training and inference process. To improve alignment stability and boundary discriminability, related methods further combine contrastive learning, multi-task joint optimization, saliency branching, boundary-sensitive modeling, and temporal context enhancement strategies to mitigate alignment shifts caused by modal differences and improve local discriminative capabilities.

[0003] Existing video temporal localization systems still commonly encounter three bottlenecks in engineering implementation and complex scenarios: First, cross-modal alignment is unstable. Natural language queries are typically short and have limited information density, making it difficult to cover fine-grained actions and state changes in videos. Under direct alignment strategies, irrelevant background segments are easily mistakenly pulled into the text semantic space, resulting in insufficient feature differentiation between adjacent segments and the target segment being submerged in similar backgrounds. Second, the supervision signal is sparse. Training often focuses on optimizing the "query-target segment" pair, lacking dense constraints on event semantics and constraints on local temporal relationships between segments. This leads to the model's perception of boundaries relying more on weakly correlated cues, which are prone to dispersion and jitter, resulting in start and end time shifts. Third, the relationship modeling level is shallow. Many methods only use first-order similarity or single-round attention interactions to characterize segment relationships, lacking structural constraints such as "similar segments should be aggregated, and dissimilar segments should be separated" and propagational relationship reasoning mechanisms. Therefore, in situations with highly similar scenes, redundant backgrounds, and semantic overlap, false detections and boundary drift are more likely to occur, leading to a decrease in overall robustness. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a video temporal localization method based on prior semantic relationship alignment of a multimodal large model. During the training phase, event semantic supervision and local relationship supervision are introduced to improve cross-modal alignment stability, segment separability and boundary cue concentration, thereby improving the localization accuracy and robustness of video temporal localization in complex backgrounds and semantically overlapping scenes.

[0005] To achieve the above-mentioned objectives, the video temporal localization method based on multimodal large model prior semantic relationship alignment includes the following steps:

[0006] S1: Obtain the training sample set according to actual needs. The input of each training sample includes video. and natural language query Video recording The number of video clips contained therein is Natural Language Query The number of words contained in it is The label is the start and end time of the target segment. , , These represent the start and end times of the target segment, respectively.

[0007] S2: Construct a video temporal localization model, including a text encoder, a video encoder, an alignment and interaction module, a contextual reasoning module, a semantic guidance enhancement module, a temporal feature extraction module, a relationship attention module, and a localization prediction module, wherein:

[0008] Text encoders are used for natural language queries Text features were extracted , Indicates the first Features of each word , Represent the feature dimensions, and then the text features Send to the alignment interaction module;

[0009] Video encoders are used to process video. Video features were extracted. , Indicates the first Features of a video clip Then, video features Send to the alignment interaction module and the context reasoning module;

[0010] The alignment interaction module is used to align text features and video features Perform alignment interaction to obtain interactive video features. And send it to the semantic guidance enhancement module;

[0011] The context reasoning module is used for video features Perform temporal context encoding to obtain temporal context features. And send it to the semantic guidance enhancement module;

[0012] The semantic guidance enhancement module is used to integrate temporal context features. and interactive video features By fusing, enhanced features are obtained. And send it to the time series feature extraction module and the relationship attention module;

[0013] The temporal feature extraction module is used to enhance features. Temporal feature extraction is performed to obtain temporal features. And send it to the relationship attention module;

[0014] The relational attention module is used to enhance features. Generate fragment relationship attention matrix Then, the temporal features Attention-weighted features are obtained. And send it to the localization prediction module. The segment relationship attention matrix is ​​included. The generation method is as follows:

[0015] Based on enhanced features Generate fragment relationship matrix ,in Indicates enhanced features The Middle Enhancement features of individual segments With the Enhancement features of individual segments Similarity between them For the fragment relation matrix Normalization yields the fragment relation attention matrix. ;

[0016] The localization prediction module is used to predict based on weighted temporal features. Predict the start and end times of the target segment;

[0017] S3: Train the video temporal localization model using the training sample set to obtain the trained video temporal localization model. During the training process, the loss function is set based on the prior semantic relationship alignment strategy of the multimodal large model. Its loss items include the location main loss. Significant loss Event semantic temporal alignment loss Alignment loss consistent with local relations The calculation formula is as follows:

[0018] ,

[0019] in, , , These represent the weights of the corresponding loss terms, and the event semantic temporal alignment loss. The calculation method is as follows:

[0020] A multimodal large model is set up according to actual needs, and the video is then processed using the multimodal large model. Divided into A semantically coherent set of events is obtained. ,in Indicates the first A textual description of the event. Indicates the first The time range of each event; the text encoder in the video temporal localization model is used to describe the event text. Encoding as event semantic features This constitutes a set of semantic features of the event. ;

[0021] For the temporal context features obtained by the context reasoning module in the video temporal localization model Perform interval pooling within the event time range to obtain the first... Contextual features of an event :

[0022] ,

[0023] in, Represents a linear mapping;

[0024] The event semantic temporal alignment loss is calculated using the following formula. :

[0025] ,

[0026] in, This indicates the calculation of feature similarity;

[0027] Local relational alignment loss The calculation method is as follows:

[0028] Multimodal large model for video Each video segment generates descriptive text, which is then encoded into a segment semantic vector using a text encoder in a video temporal localization model. Based on this, a text relation matrix is ​​constructed. ,in Indicates the first semantic vectors of fragments and the semantic vectors of fragments The similarity between them is calculated using the following formula to determine the local relation consistency alignment loss. :

[0029] ,

[0030] in, This indicates the search for the L2 norm;

[0031] S4: For videos requiring temporal positioning, divide them into... The system takes a video segment and inputs the video and the query to be located into a trained video temporal localization model to obtain the start and end times of the target segment.

[0032] This invention presents a video temporal localization method based on multimodal large-scale model prior semantic relation alignment. It constructs a video temporal localization model, extracting textual and video features from natural language queries and videos respectively. After alignment and interaction, interactive video features are obtained. A contextual reasoning module extracts temporal contextual features from the video features. These temporal contextual features and interactive video features are fused to obtain enhanced features. Temporal features are extracted from the enhanced features, and a segment relationship attention matrix is ​​generated based on these enhanced features. The temporal features are then weighted, and the start and end times of the target segment are predicted based on the weighted temporal features. The model is trained using a training sample set. During training, a multimodal large-scale model prior semantic relation alignment strategy is used to set the loss term in the loss function. After training, temporal localization of input videos can be performed.

[0033] The present invention has the following beneficial effects:

[0034] 1) This invention designs two types of text prior alignment constraints for training: event semantic temporal alignment and local relation consistency alignment, which provide denser semantic and structural signals, improve the performance of video temporal localization models, and thus improve the accuracy of video temporal localization.

[0035] 2) This invention improves the loss function by using a prior semantic relationship alignment strategy based on a multimodal large model. Without increasing the model overhead in the inference stage, it can improve cross-modal alignment stability, fragment separability and boundary cue concentration by introducing event semantic supervision and local relationship supervision only in the training stage, thereby improving the localization accuracy and robustness in complex backgrounds and semantically overlapping scenarios.

[0036] 3) This invention also proposes a decoupled interactive alignment strategy, which can achieve adaptive decoupling of the alignment process through a context-aware codebook, effectively absorb query-irrelevant semantic disturbances, form clearer boundary discrimination clues, make the cross-modal alignment process more stable, and comprehensively improve the video temporal positioning effect. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating a specific implementation of the video temporal localization method based on prior semantic relationship alignment of a multimodal large model according to the present invention.

[0038] Figure 2 This is a structural diagram of the video temporal localization model in this invention;

[0039] Figure 3 This is a structural diagram of the alignment interaction module in this embodiment;

[0040] Figure 4 This is a schematic diagram of the prior semantic relation alignment strategy for multimodal large models in this invention. Detailed Implementation

[0041] The specific embodiments of the present invention will now be described with reference to the accompanying drawings to enable those skilled in the art to better understand the invention. It should be particularly noted that in the following description, detailed descriptions of known functions and designs that might obscure the main content of the invention will be omitted here.

[0042] Example

[0043] Figure 1 This is a flowchart illustrating a specific implementation of the video temporal localization method based on multimodal large model prior semantic relationship alignment according to the present invention. Figure 1 As shown, the video temporal localization method based on prior semantic relationship alignment of a multimodal large model according to the present invention includes the following steps:

[0044] S101: Obtain the training sample set:

[0045] Obtain a training sample set according to actual needs. The input of each training sample includes a video. and natural language query Video recording The number of video clips contained therein is Natural Language Query The number of words contained in it is The label is the start and end time of the target segment. , , These represent the start and end times of the target segment, respectively.

[0046] S102: Constructing a video temporal localization model:

[0047] To improve the accuracy of video temporal localization under the prior semantic relationship alignment strategy of multimodal large model, this invention specifically designs a video temporal localization model. Figure 2 This is a structural diagram of the video temporal localization model in this invention. (See diagram below.) Figure 2 As shown, the video temporal localization model in this invention includes a text encoder, a video encoder, an alignment and interaction module, a contextual reasoning module, a semantic guidance enhancement module, a temporal feature extraction module, a relationship attention module, and a localization prediction module. Each module will be described in detail below.

[0048] Text encoders are used for natural language queries Text features were extracted , Indicates the first Features of each word , Represent the feature dimensions, and then the text features Send to the alignment interaction module.

[0049] Video encoders are used to process video. Video features were extracted. , Indicates the first Features of a video clip Then, video features Send to the alignment interaction module and the context reasoning module.

[0050] In practical applications, to improve the training efficiency of subsequent video temporal localization models, pre-trained and freezeable text encoders and video encoders can be used.

[0051] The alignment interaction module is used to align text features and video features Perform alignment interactions to obtain interaction features. And send it to the semantic guidance enhancement module.

[0052] Existing direct alignment interactions tend to mistakenly pull query-irrelevant background fragments into the text space, resulting in insufficient local separability. Therefore, this embodiment proposes a decoupled alignment interaction strategy. By constructing cross-modal differential context tokens to explicitly characterize the background bias, and introducing a context-aware codebook to generate auxiliary tokens to participate in cross-modal interactions to absorb query-irrelevant semantic disturbances and stabilize the alignment process. Figure 3 This is a structural diagram of the alignment interaction module in this embodiment. For example... Figure 3As shown, in this embodiment, the alignment interaction module includes a first linear layer, a first pooling layer, a second linear layer, a second pooling layer, a background context calculation module, an auxiliary token generation module, and a cross-modal attention interaction module, wherein:

[0053] The first linear layer is used for text features. Perform a linear transformation to obtain text features Send to the first pooling layer.

[0054] The first pooling layer is used to process text features. Pooling is performed on the word sequence to obtain text pooling features. And send it to the background context calculation module.

[0055] The second linear layer is used to process video features. Perform a linear transformation to obtain video features Send to the second pooling layer.

[0056] The second pooling layer is used to process video features. Pooling is performed along the time dimension to obtain video pooling features. And send it to the background context calculation module.

[0057] The background context calculation module is used to calculate background context features. And send it to the auxiliary token generation module.

[0058] The auxiliary token generation module is used to store the codebook. , Indicates the first Each code character , Indicate the number of codewords, and then receive the background context features. Then, the one with the highest similarity was found from it. Each codeword serves as an auxiliary token. , constitute auxiliary token features Then, the auxiliary token feature Send to the cross-modal attention interaction module.

[0059] The cross-modal attention interaction module is used for text features Video features and auxiliary token features Perform cross-modal attention interaction to obtain interactive video features. The specific method for cross-modal attention interaction in this embodiment is as follows:

[0060] Text features Video features and auxiliary token features Features obtained by splicing Then, a self-attention mechanism is used to obtain features. Self-attention characteristics Extract the features corresponding to the video features from it. As a feature of interactive video .

[0061] The context reasoning module is used for video features Perform temporal context encoding to obtain temporal context features. The data is then sent to the semantic guidance enhancement module. By extracting temporal context features, the dispersion of boundary clues caused by relying solely on interaction features can be effectively avoided.

[0062] The semantic guidance enhancement module is used to integrate temporal context features. and interactive video features By fusing, enhanced features are obtained. And send it to the time series feature extraction module and the relationship attention module.

[0063] In this embodiment, the semantic guidance enhancement module employs a two-level fusion strategy to highlight key segments and suppress redundant background, enabling subsequent localization to focus more on semantically consistent time regions. The specific expression for the two-level fusion is as follows:

[0064] ,

[0065] ,

[0066] in, This represents the cross-attention mechanism. These represent the query matrix, key matrix, and value matrix in the cross-attention mechanism, respectively.

[0067] The temporal feature extraction module is used to enhance features. Temporal feature extraction is performed to obtain temporal features. And send it to the relational attention module. In this embodiment, the temporal feature extraction module includes a Transformer encoder and a multilayer perceptron, wherein:

[0068] Transformer encoders are used to fuse features. Temporal encoding is performed to obtain temporal features, which are then sent to a multilayer perceptron.

[0069] A multilayer perceptron is used to process the received temporal features to obtain temporal characteristics. .

[0070] The relational attention module is used to enhance features. Generate fragment relationship attention matrix Then, the temporal features Attention-weighted features are obtained. And send it to the localization prediction module. The segment relationship attention matrix is ​​included. The generation method is as follows:

[0071] Based on enhanced features Generate fragment relationship matrix ,in Indicates enhanced features The Middle Enhancement features of individual segments With the Enhancement features of individual segments Similarity between them For the fragment relation matrix Normalization yields the fragment relation attention matrix. In this embodiment, cosine similarity is used for feature similarity, and the softmax activation function is applied to the fragment relationship matrix. Normalization is performed to generate attention relation attention matrix ,Right now .

[0072] Fragment Relationship Matrix Using first-order similarity to characterize fragment relationships is insufficient to form structural boundary cues that "aggregate similar groups and separate dissimilar groups." Therefore, in practical applications, relation propagation and higher-order consistency can be introduced to modify the fragment relationship matrix. The optimization process is as follows:

[0073] Based on the fragment relationship matrix Extracting the second-order relation matrix Then calculate the optimized fragment relationship matrix. In this embodiment, a thinned network composed of convolutions and nonlinearities is used to extract the second-order relation matrix. This mechanism can concentrate boundary-related structural cues, improve the separability of adjacent segments, and thus alleviate boundary jitter and drift.

[0074] The localization prediction module is used to predict based on weighted temporal features. Predict the start and end times of the target segment.

[0075] S103: Training the video temporal localization model:

[0076] The video temporal localization model is trained using a training sample set to obtain a well-trained video temporal localization model.

[0077] To address the issues of sparse supervision and lack of local relation constraints, this invention employs a loss function based on a prior semantic relation alignment strategy of a multimodal large model. This involves introducing two types of text priors generated by a multimodal large model during the training phase: fragment-level fine-grained descriptions and event-level semantic analysis and their time ranges, which are used for event semantic temporal alignment and local relation consistency alignment, respectively. Figure 4 This is a schematic diagram of the prior semantic relation alignment strategy for multimodal large models in this invention. For example... Figure 4 As shown, the specific method of the prior semantic relation alignment strategy for multimodal large models is as follows:

[0078] A multimodal large model is set up according to actual needs, and the video is then processed using the multimodal large model. Divided into A semantically coherent set of events is obtained. ,in Indicates the first A textual description of the event. Indicates the first The time range of each event. A text encoder from a video temporal localization model is used to describe the event text. Encoding as event semantic features This constitutes a set of semantic features of the event. .

[0079] For the temporal context features obtained by the context reasoning module in the video temporal localization model Perform interval pooling within the event time range to obtain the first... Contextual features of an event :

[0080] ,

[0081] in, Represents a linear mapping;

[0082] The event semantic temporal alignment loss is calculated using the following formula. :

[0083] ,

[0084] in, This indicates the calculation of feature similarity; in this embodiment, cosine similarity is used.

[0085] This loss function enables the model to learn "which time period corresponds to which type of event semantics", improving the separability across time from the perspective of temporal semantics. It provides more reliable contextual semantic support for the semantic guidance enhancement module, making the fused context more consistent and separable in terms of event semantics.

[0086] At the same time, multimodal large models provide video Each video segment generates descriptive text, which is then encoded into a segment semantic vector using a text encoder in a video temporal localization model. Based on this, a text relation matrix is ​​constructed. ,in Indicates the first semantic vectors of fragments and the semantic vectors of fragments The similarity between them.

[0087] Fragment Relationship Matrix Text Relationship Matrix To apply consistency constraints, the loss for consistent alignment of local relations is calculated using the following formula. :

[0088] ,

[0089] in, This indicates the search for the L2 norm;

[0090] This loss operates directly on the relational structure rather than on single-point embeddings, enabling the model to obtain structural supervision that similar segments should be close together and irrelevant segments should be separated, thereby improving local discrimination ability and boundary stability, and complementing relational attention: the former provides relational supervision, while the latter provides relational propagation and higher-order refinement.

[0091] In summary, the loss function in this invention The loss items include the location of the main loss. Significant loss Event semantic temporal alignment loss Alignment loss consistent with local relations The calculation formula is as follows:

[0092] ,

[0093] in, , , These represent the weights of the corresponding loss terms.

[0094] When the alignment interaction module adopts the decoupled alignment interaction proposed in this embodiment, codebook constraint loss can also be introduced to avoid auxiliary token degradation, allowing it to continuously absorb background disturbances, thereby improving alignment robustness. Therefore, the loss function The loss term can also include codebook stability loss. At this time, the loss function The calculation formula is as follows:

[0095] ,

[0096] in, , , , These represent the weights of the corresponding loss terms.

[0097] Locating the main loss Significant loss and codebook stable loss It is a commonly used loss in video temporal localization, and its calculation process will not be elaborated here.

[0098] S104: Video timing positioning:

[0099] For videos requiring time-series localization, they are divided into... The system takes a video segment and inputs the video and the query to be located into a trained video temporal localization model to obtain the start and end times of the target segment.

[0100] To better illustrate the technical effects of this invention, specific examples are used to experimentally verify it. This embodiment is evaluated on the publicly available video temporal localization benchmark dataset QVHighlights. This dataset covers various scenarios such as sports, music, dance, and film, and includes challenges such as complex backgrounds, semantic overlap, and blurred event boundaries, making it suitable for testing cross-modal alignment stability and boundary regression accuracy. In this embodiment, eight existing methods are used as comparison methods to verify this invention, namely:

[0101] XML, see the document "Anne Hendricks L, Wang O, Shechtman E, et al. Localizing moments in video with natural language. Proceedings of the IEEE international conference on computer vision. 2017: 5803-5812.";

[0102] M-DETR, see the literature "Lei J, Berg TL, Bansal M. Detecting moments and highlights in videos via natural language queries. Advances in NeuralInformation Processing Systems, 2021, 34: 11846-11858."

[0103] UMT, see the literature "Liu Y, Li S, Wu Y, et al. Umt: Unified multi-modaltransformers for joint video moment retrieval and highlight detection.Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition. 2022: 3042-3051."

[0104] QD-DETR, see the literature "Moon W J, Hyun S, Park S U, et al. Query-dependentvideo representation for moment retrieval and highlight detection.Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition. 2023: 23023-23033."

[0105] UniVTG, see the literature "Lin K Q, Zhang P, Chen J, et al. Univtg: Towardsunified video-language temporal grounding. Proceedings of the IEEE / CVFInternational Conference on Computer Vision. 2023: 2794-2804."

[0106] MESM, see the literature "Liu Z, Li J, Xie H, et al. Towards balanced alignment:Modal-enhanced semantic modeling for video moment retrieval. Proceedings ofthe AAAI Conference on Artificial Intelligence. 2024, 38(4): 3855-3863."

[0107] UVCOM, see the document "Xiao Y, Luo Z, Liu Y, et al. Bridging the gap: Aunified video comprehension framework for moment retrieval and highlightdetection. Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition. 2024: 18709-18719."

[0108] LLMEPET, see the document "Jiang Y, Zhang W, Zhang X, et al. Prior knowledge integration via llm encoding and pseudo event regulation for video momentretrieval. Proceedings of the 32nd ACM International Conference on Multimedia.2024: 7249-7258."

[0109] This embodiment follows a predetermined dataset partitioning, performing parameter learning on the training set and evaluating and comparing models on the validation set. The evaluation metrics used are the domain-common Recall and mAP metrics, which represent the hit rate of statistical prediction results at a specified IoU threshold and the overall retrieval quality, respectively, reflecting both localization accuracy and robustness. To reduce the impact of differences in feature extraction, the experiment uses common visual and textual features consistent with the comparison method as input representations. Table 1 is a performance comparison table of the present invention and the comparison method in this embodiment.

[0110] method R1@0.5 R1@0.7 mAP@0.5 mAP@0.75 mAP Avg. XML 41.83 30.35 44.63 31.73 32.14 M-DETR 52.89 33.02 54.82 29.4 30.73 UMT 56.23 41.18 53.83 37.01 36.12 QD-DETR 62.4 44.98 62.52 39.88 39.86 UniVTG 58.86 40.86 57.6 35.59 35.47 MESM 62.78 45.2 62.64 41.45 40.68 UVCOM 63.55 47.47 63.37 42.67 43.18 LLMEPET 66.73 49.94 65.76 43.91 44.05 This invention 68.92 55.17 69.39 53.85 52.09

[0111] Table 1

[0112] As shown in Table 1, the present invention achieves better performance on multiple metrics of the QVHighlights validation set, especially with significant improvements in higher IoU thresholds and mAP metrics, indicating that event semantic supervision and local relation supervision can effectively enhance the concentration of boundary clues and local separability. At the same time, the decoupled alignment interaction of the context-aware codebook reduces query-irrelevant background interference, making the cross-modal alignment process more stable and comprehensively improving the localization effect.

[0113] Although the illustrative specific embodiments of the present invention have been described above to enable those skilled in the art to understand the invention, it should be understood that the invention is not limited to the scope of the specific embodiments. For those skilled in the art, various changes are obvious as long as they are within the spirit and scope of the invention as defined and determined by the appended claims, and all inventions utilizing the concept of the present invention are protected.

Claims

1. A video temporal localization method based on prior semantic relation alignment using a multimodal large model, characterized in that, Includes the following steps: S1: Obtain the training sample set according to actual needs. The input of each training sample includes video. and natural language query Video recording The number of video clips contained therein is Natural Language Query The number of words contained in it is The label is the start and end time of the target segment. , , These represent the start and end times of the target segment, respectively. S2: Construct a video temporal localization model, including a text encoder, a video encoder, an alignment and interaction module, a contextual reasoning module, a semantic guidance enhancement module, a temporal feature extraction module, a relationship attention module, and a localization prediction module, wherein: Text encoders are used for natural language queries Text features were extracted , Indicates the first Features of each word , Represent the feature dimensions, then use the text features Send to the alignment interaction module; Video encoders are used to process video. Video features were extracted. , Indicates the first Features of a video clip Then, video features Send to the alignment interaction module and the context reasoning module; The alignment interaction module is used to align text features and video features Perform alignment interaction to obtain interactive video features. And send it to the semantic guidance enhancement module; The context reasoning module is used for video features Perform temporal context encoding to obtain temporal context features. And send it to the semantic guidance enhancement module; The semantic guidance enhancement module is used to integrate temporal context features. and interactive video features By fusing, enhanced features are obtained. And send it to the time series feature extraction module and the relationship attention module; The temporal feature extraction module is used to enhance features. Temporal feature extraction is performed to obtain temporal features. And send it to the relationship attention module; The relational attention module is used to enhance features. Generate fragment relationship attention matrix Then, the temporal features Attention-weighted features are obtained. And send it to the localization prediction module. The segment relationship attention matrix is ​​included. The generation method is as follows: Based on enhanced features Generate fragment relation matrix ,in Indicates enhanced features The Middle Enhancement features of individual segments With the Enhancement features of individual segments Similarity between them For the fragment relation matrix Normalization yields the fragment relation attention matrix. ; The localization prediction module is used to predict based on weighted temporal features. Predict the start and end times of the target segment; S3: Train the video temporal localization model using the training sample set to obtain the trained video temporal localization model. During the training process, the loss function is set based on the prior semantic relationship alignment strategy of the multimodal large model. Its loss items include the location main loss. Significant loss Event semantic temporal alignment loss Alignment loss consistent with local relations The calculation formula is as follows: ， in, , , These represent the weights of the corresponding loss terms, and the event semantic temporal alignment loss. The calculation method is as follows: A multimodal large model is set up according to actual needs, and the video is then processed using the multimodal large model. Divided into A semantically coherent set of events is obtained. ,in Indicates the first A textual description of the event. Indicates the first The time range of each event; the text encoder in the video temporal localization model is used to describe the event text. Encoding as event semantic features This constitutes a set of semantic features of the event. ; For the temporal context features obtained by the context reasoning module in the video temporal localization model Perform interval pooling within the event time range to obtain the first... Contextual features of an event : ， in, Represents a linear mapping; The event semantic temporal alignment loss is calculated using the following formula. : ， in, This indicates the calculation of feature similarity; Local relational alignment loss The calculation method is as follows: Multimodal large model for video Each video segment generates descriptive text, which is then encoded into a segment semantic vector using a text encoder in a video temporal localization model. Based on this, a text relation matrix is ​​constructed. ,in Indicates the first semantic vectors of fragments and the semantic vectors of fragments The similarity between them is calculated using the following formula to determine the local relation consistency alignment loss. : ， in, This indicates the search for the L2 norm; S4: For videos requiring temporal positioning, divide them into... The system takes a video segment and inputs the video and the query to be located into a trained video temporal localization model to obtain the start and end times of the target segment.

2. The video temporal localization method according to claim 1, characterized in that, The alignment interaction module includes a first linear layer, a first pooling layer, a second linear layer, a second pooling layer, a background context calculation module, an auxiliary token generation module, and a cross-modal attention interaction module, wherein: The first linear layer is used for text features. Perform a linear transformation to obtain text features Send to the first pooling layer; The first pooling layer is used to process text features. Pooling is performed on the word sequence to obtain text pooling features. And send it to the background context calculation module; The second linear layer is used to process video features. Perform a linear transformation to obtain video features Send to the second pooling layer; The second pooling layer is used to process video features. Pooling is performed along the time dimension to obtain video pooling features. And send it to the background context calculation module; The background context calculation module is used to calculate background context features. And send it to the auxiliary token generation module; The auxiliary token generation module is used to store the codebook. , Indicates the first Each code character , Indicate the number of codewords, and then receive the background context features. Then, the one with the highest similarity was found from it. Each codeword serves as an auxiliary token. , constitute auxiliary token features Then, the auxiliary token feature Send to the cross-modal attention interaction module; The cross-modal attention interaction module is used for text features Video features and auxiliary token features Perform cross-modal attention interaction to obtain interactive video features. .

3. The video temporal localization method according to claim 2, characterized in that, The specific methods for cross-modal attention interaction in the cross-modal attention interaction module are as follows: Text features Video features and auxiliary token features Features obtained by splicing Then, a self-attention mechanism is used to obtain features. Self-attention characteristics Extract the features corresponding to the video features from it. As a feature of interactive video .

4. The video temporal positioning method according to claim 2, characterized in that, The loss function The loss items also include codebook stability loss. loss function The calculation formula is as follows: ， in, , , , These represent the weights of the corresponding loss terms.

5. The video temporal localization method according to claim 1, characterized in that, The semantic guidance enhancement module adopts a two-level fusion approach, as follows: ， ， in, This represents the cross-attention mechanism. These represent the query matrix, key matrix, and value matrix in the cross-attention mechanism, respectively.

6. The video temporal localization method according to claim 1, characterized in that, The temporal feature extraction module includes a Transformer encoder and a multilayer perceptron, wherein: Transformer encoders are used to fuse features. Perform temporal encoding to obtain temporal features and send them to a multilayer perceptron; A multilayer perceptron is used to process the received temporal features to obtain temporal characteristics. .

7. The video temporal localization method according to claim 1, characterized in that, The fragment relationship matrix The following methods were used for optimization: Based on the fragment relationship matrix Extracting the second-order relation matrix Then calculate the optimized fragment relationship matrix. .

8. The video temporal localization method according to claim 1, characterized in that, The attention relationship attention matrix .

Images