Efficient video time sequence positioning system based on time sequence feature aggregation
By designing a video timing positioning system based on timing feature aggregation, using multimodal feature extraction and fusion of 2D video features and text features, combined with learnable timestamp embedding and attention mechanisms, the calculation cost in the existing technology is solved. The problem of insufficient utilization of high and time dimension information has achieved efficient and accurate video timing positioning.
Patent Information
- Application Number
- CN202411329756.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-24
- Publication Date
- 2025-02-18
AI Technical Summary
Existing video timing positioning methods rely on 3D video features with high computational cost, or insufficient extraction and utilization of video time dimension information when using 2D video features, limiting the efficiency and performance of the system.
An efficient video timing positioning system based on timing feature aggregation is designed. The 2D video features and text features are extracted through a multi-modal feature extractor, combined with the query-guided timing feature aggregation module, a multi-modal encoder and a prediction decoder, to realize Video timing positioning. The system uses cross-attention mechanisms and deformable attention mechanisms for feature fusion and decouples video frame intervals and predicted intervals through learnable timestamp embeddings.
It achieves a significant improvement in the accuracy and efficiency of video timing positioning with less computational costs, and achieves leading results on the ActivityNet Captions and Charades-STA datasets.
Smart Images
Figure CN119478757A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of video processing, and in particular relates to an efficient video timing positioning system based on timing feature aggregation. Background Art
[0002] In recent years, video temporal localization has become a very worthy problem in multimodal learning. Videos usually contain rich information and are a common form of media, so how to make computers understand video language has a high research value. Video temporal localization focuses on cross-text and video retrieval. Its goal is to locate the most relevant video clips in the complete video based on a given text query. The results are usually expressed in terms of the start and end times of the clips.
[0003] Video timing positioning methods can be roughly divided into two categories, namely, methods based on timing proposals and methods without timing proposals. Methods based on timing proposals usually first determine multiple candidate segments, and then select the one with the highest similarity to the query text as the prediction result. Since the pre-delineated candidate segments are not particularly accurate, the boundaries of the candidate segments will be fine-tuned when predicting the results later. Methods without timing proposals do not need to generate many candidate segments in advance, simplifying the entire prediction process. This type of method achieves video timing positioning by directly predicting the start and end times of the target video segment.
[0004] However, the above methods usually rely on 3D video features. Such methods usually have high computational costs, which limits the development of video timing positioning systems in terms of efficiency. At present, some researchers have also begun to use 2D video features for video timing positioning. Although these methods significantly reduce computational costs, they are insufficient in extracting and utilizing information in the temporal dimension of the video, which limits their performance. Summary of the invention
[0005] The object of the present invention is to provide a high-efficiency video timing positioning system based on timing feature aggregation, which has a balanced accuracy and efficiency and excellent video timing positioning effect.
[0006] The efficient video temporal positioning system based on temporal feature fusion provided by the present invention comprises a multimodal feature extractor, a query-guided temporal feature aggregation module, a multimodal encoder and a predictive decoder; wherein:
[0007] The multimodal feature extractor is used to extract features of input video and text, and the extracted video features are 2D features, which can reduce the amount of calculation;
[0008] The query-guided temporal feature aggregation module extracts the temporal relationship information in the video at multiple time scales through convolution kernels of different sizes under the guidance of text queries after the multimodal feature extractor extracts the video features, thereby obtaining temporal enhanced video features to improve the problem of missing temporal information in the video 2D features without introducing too many parameters.
[0009] The multimodal encoder is used to fuse the video features and the text features after the temporal feature aggregation module obtains the temporal enhanced video features; specifically, the cross attention mechanism is used to further align the two modalities of text and video, and then the deformable attention is used to fuse the features of the two modalities of text and video while saving the amount of calculation;
[0010] The prediction decoder is used to complete the prediction of the final video timing positioning result after the multimodal encoder obtains the fusion features; specifically, it uses learnable timestamp embedding to decouple the prediction interval and the interval of the input video frame, and finally outputs the start and end time of the video segment to be positioned, thereby achieving more accurate video timing positioning with fewer video frame inputs.
[0011] Further:
[0012] The multimodal feature extractor consists of two modules: a video feature extractor and a text feature extractor. The video feature extractor is used to extract 2D features of the video, which significantly reduces the amount of calculation. Specifically, the video frames of the input video are first uniformly sampled, and then a 2D ResNet model is used to extract features from each sampled video frame. The features after the ResNet pooling layer are used as the final video features extracted by the module, which can be expressed as:
[0013]
[0014] Where T is the number of video frames, d v is the dimension of the video feature. In addition, the video feature extractor is trainable, allowing the video feature extractor parameters to be updated during the model training process, thereby achieving better representation learning.
[0015] For the text feature extractor, GloVe is used to obtain the representation embedding of the text query, which is the final text feature of this module and can be expressed as:
[0016]
[0017] Where N is the number of words in the text query, and d q is the dimension of text features.
[0018] After extracting video and text features, they are projected to a unified feature dimension d using two independent fully connected layers respectively.
[0019] The query-guided temporal feature aggregation module extracts the temporal relationship information in the video after the multimodal feature extractor extracts the video features, so as to improve the problem of missing temporal information in the video 2D features, without introducing too many parameters. The module first uses convolution kernels of different sizes to capture the temporal relationship between video frames of different time scales. Specifically, the module uses convolution operations with a convolution kernel size k of {1,3,5,7}. At the same time, in order to further reduce the amount of calculation, before performing the convolution operation on the feature, the channel dimension of the video feature V is first split to obtain a set of different features corresponding to convolution operations of different convolution kernel sizes, and then input them into different aggregation branches respectively. In this operation, the input feature of the aggregation branch corresponding to the convolution kernel size k is represented as V k . Considering that videos usually contain a lot of fine-grained time-related details (such as actions and events, etc.), which may be ignored in the feature extraction process of 2D ResNet, this module uses text queries to help extract and recover temporal information in videos. Specifically, the text query is regarded as a guide, and cross attention is used to further enhance the features extracted using convolution at different time scales. When using text queries as a guide, an MLP is used to map the dimension of text embedding to the dimension V of video features in each aggregation branch. k Thus, we get the temporally enhanced video feature V ′ :
[0020] V k ′ = CrossAttn(Conv k (V k ),MLP k (Q))#(3)
[0021] V ′ =Concat k∈{1,3,5,7} (V k ′ )#(4)
[0022] Among them, CrossAttn represents the cross attention operation, Concat(·) represents the concatenation operation along the channel dimension, and Conv k (·) represents a one-dimensional convolution operation corresponding to a convolution kernel size of k. MLP k represents the MLP network layer in the aggregation branch k. Afterwards, a cross-attention operation can be used to further fuse the video features V ′ And text feature Q, get the final feature V QTFA :
[0023] V QTFA =CrossAttn(V ′ ,Q)#(5)
[0024] Among them, CrossAttn represents the cross attention operation, V ′ As the query in the operation, Q is used as the key and value in the operation. QTFA Finally, the features are input into the subsequent multimodal encoder.
[0025] The multimodal encoder is used to fuse the video features and text features after the temporal feature aggregation module obtains the temporal enhanced video features. e The network layers further encode the video features and text features respectively, and realize the fusion of the two features. When further encoding the video features and text features respectively, the deformable attention mechanism is adapted to the multimodal coding. This mechanism uses reference points when calculating the attention mechanism, and can adaptively select features related to the input modality based on the reference points. Therefore, it is achieved to enhance the features that are more relevant to the other modality while saving computational effort. In order to better utilize multimodal input, the encoder fuses video and text features to predict one-dimensional reference points instead of using only the features of a single modality. The following will take video features as an example to give a detailed and specific introduction to the operations and steps of this part. For video features, a cross-attention mechanism is used to enhance the model's perception of text queries when selecting reference points, which can help the model learn more adaptive feature representations. This operation converts the feature V obtained in the query-guided temporal feature aggregation module into a cross-attention mechanism. QTFA As input, given a text query feature p corresponding to index q q In the case of a one-dimensional reference point p in the time dimension, the deformable attention operation in the video time dimension can be expressed as:
[0026]
[0027] Where m is the index of the head structure in the attention mechanism, k is the index of the key in the attention mechanism, and K is the number of keys in the sampled attention mechanism. mqk and A mqk They represent the sampling offset and attention weight of the k-th sampling point in the m-th attention head in the deformable attention, respectively.
[0028] Similarly, deformable attention is also used to encode text features, which allows the model to learn more appropriate text features based on the context information of the video. After encoding the video features and text features separately, the encoded features of the two are concatenated and feature fusion is performed using MLP to obtain the final fused feature F. The above process can be expressed as:
[0029] F = MLP(Concat.Encoder(V QTFA ),Encoder(Q) / )#(7)
[0030] Here, Encoder(·) represents the Transformer encoder using the deformable attention mechanism.
[0031] The prediction decoder is used to complete the prediction of the final video temporal positioning result after the multimodal encoder obtains the fusion feature. The module consists of a learnable timestamp embedding and a prediction module. When predicting, a common span-based method is used to predict the start and end of the target time segment. This method usually calculates the probability of each input video frame as the start and end, and usually requires a large number of video frames as input. However, inputting a large number of video frames often leads to an increase in computational cost. To this end, it is proposed to decouple the input video frame interval and the prediction interval of the target segment when predicting the probability of the start and end, so as to achieve a better balance between accuracy and efficiency. Specifically, the learnable timestamp embedding is used as input to replace the previous video frame input into the prediction module, and then the probability of different positions in the video sequence as the start or end is predicted. This allows the model to use relatively few video frames as input, significantly reducing the computational cost. At the same time, a large number of timestamp embeddings can be used as inputs of the prediction module, thereby obtaining more intensive boundary predictions about the target time segment, and achieving higher prediction accuracy when fewer video frames are input.
[0032] The predictive encoder takes the timestamp embedding E and the fused feature F as input and uses L d The standard Transformer decoder realizes the interaction between timestamp embedding and fusion features. In this process, sinusoidal position encoding PE is also added to obtain richer position information. The process can be expressed as:
[0033]
[0034] Among them, Decoder(·) represents the standard Transformer decoder, the timestamp embedding E is used as the query of the cross attention operation, and the fused feature F is used as the key and value.
[0035] When predicting the start and end of the target time segment, a common span-based approach is used. Input to the predictor, and get the probability of using the position of each timestamp embedding as the start and end boundary of the target time. The predictor consists of a GRU and two network layers. For each position i, calculate the true value and predicted output [t s ,t i The intersection and union ratio between ] i , and use it as a supervisory signal for the end position to train the model. The loss function is defined as:
[0036]
[0037] After that, the normalized starting position corresponding to position i is calculated And use it as the supervision signal of the starting position. The loss function can be defined as:
[0038]
[0039] Finally, the overall loss function used for model training is defined as:
[0040]
[0041] In the prediction stage of the model, the probability of the predicted start and end times for each location can be calculated, and the location with the highest probability is selected as the final prediction result.
[0042] The video timing positioning system based on 2D video features of the present invention has a workflow that can be summarized as follows:
[0043] (i) First, a multimodal feature extractor is used to extract features from the input text query and video;
[0044] (ii) The extracted features are input into a query-guided temporal feature aggregation module to explicitly extract information related to the temporal dimension of the video and obtain temporally enhanced video features.
[0045] (iii) Use a multimodal encoder to achieve alignment and feature fusion of video and text modalities.
[0046] (iv) The predictive decoder predicts the start and end times of the target video segment based on the fused features and the input timestamp embedding.
[0047] Advantages of the present invention include:
[0048] (1) The video temporal localization system based on 2D video features of the present invention can achieve explicit extraction of temporal dimension information from the 2D features of the video frame through the proposed query-guided temporal feature aggregation module, thereby alleviating the problem of lack of temporal context information in 2D video features.
[0049] (2) Adapt the deformable attention mechanism to multimodal encoding to extract features that are more relevant to the modality while reducing the amount of feature encoding computation.
[0050] (3) Design a learnable timestamp embedding to decouple the input video frame interval and the prediction interval of the target segment to achieve a better balance between accuracy and efficiency.
[0051] (4) The present invention achieves leading video temporal localization effects on the public datasets ActivityNet Captions and Charades-STA. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 It is a complete framework diagram of the model in the present invention.
[0053] Figure 2 It is a structural diagram of the query-guided temporal feature aggregation module in the present invention.
[0054] Figure 3 It is a structural diagram of the multimodal encoder in the present invention.
[0055] Figure 4 It is a structural diagram of the prediction decoder in the present invention.
[0056] Figure 5 It is the time series positioning result graph generated by the model after the time series feature aggregation module. DETAILED DESCRIPTION
[0057] Hereinafter, the specific details of the present invention will be described in conjunction with the accompanying drawings. Figure 1 As shown, the efficient video temporal positioning system based on temporal feature aggregation of the present invention includes a multimodal feature extractor, a query-guided temporal feature aggregation module, a multimodal encoder and a prediction decoder, and its working process is as follows:
[0058] 1. The multimodal feature extractor of the present invention is composed of two modules: a video feature extractor and a text feature extractor. The video feature extractor can extract 2D features of the video, significantly reducing the amount of calculation. Specifically, the video frames of the input video are first uniformly sampled, and then a 2D ResNet model is used to extract features from each sampled video frame, and the features after the ResNet pooling layer are used as the final video features extracted by the module, which can be expressed as:
[0059]
[0060] Where T is the number of video frames, d v is the dimension of the video feature. In addition, the video feature extractor is trainable, allowing the video feature extractor parameters to be updated during the model training process, thereby achieving better representation learning.
[0061] For the text feature extractor, GloVe is used to obtain the representation embedding of the text query, which is the final text feature of this module and can be expressed as:
[0062]
[0063] Where N is the number of words in the text query, and d q is the dimension of text features. In addition, after extracting video and text features, they are projected to a unified feature dimension d using two independent fully connected layers.
[0064] Second, after extracting the features of the video and text, the features of both are input into the query-guided temporal feature aggregation module of the present invention, and the temporal relationship information in the video is extracted through additional temporal feature aggregation. Figure 2 As shown in Figure 1, this module uses convolution operations with kernel size k of {1, 3, 5, 7} to capture the temporal relationship between video frames at different time scales. Before performing convolution operations on features, the channel dimension of the video feature V is split and then input into different aggregation branches respectively. In this operation, the input feature of the aggregation branch corresponding to the convolution kernel size k is represented as V k This module considers the text query as a guide and uses cross-attention to further enhance the features extracted using convolutions at different time scales. When using the text query as a guide, an MLP is used to map the dimensions of the text embedding to the dimensions V of the video features in each aggregation branch. k , and obtain the temporally enhanced video feature V ′ :
[0065] V k ′ = CrossAttn(Conv k (V k ),MLP k (Q))#(3)
[0066] V ′ =Concat k∈{1,3,5,7} (V k ′ )#(4)
[0067] Among them, CrossAttn represents the cross attention operation, Concat(·) represents the concatenation operation along the channel dimension, and Conv k (·) represents a one-dimensional convolution operation corresponding to a convolution kernel size of k, MLP k represents the MLP network layer in the aggregation branch k. Afterwards, a cross-attention operation can be used to further fuse the video features V ′ And text feature Q, get the final feature V QTFA :
[0068] V QTFA =CrossAttn(V ′ ,Q)#(5)
[0069] Among them, CrossAttn represents the cross attention operation, V ′ As the query in the operation, Q is used as the key and value in the operation. QTFA Finally, the features are input into the subsequent multimodal encoder.
[0070] 3. After obtaining the temporally enhanced video features, a multimodal encoder is used to fuse the video features with the features of the text query. Figure 3 As shown, the encoder uses L e The network layer further encodes the video features and text features respectively, and realizes the fusion of the two features. When further encoding the video features and text features respectively, the deformable attention mechanism is adapted to the multimodal encoding, in which the video and text features are fused when predicting the one-dimensional reference point. The following specific implementation method will take the video feature as an example to introduce the operation and steps of this part in detail. For the video feature, the cross attention mechanism is used to integrate the feature V obtained in the query-guided temporal feature aggregation module QTFA As input, given a text query feature p corresponding to index q q In the case of a one-dimensional reference point p in the time dimension, the deformable attention operation in the video time dimension can be expressed as:
[0071]
[0072] Where m is the index of the head structure in the attention mechanism, k is the index of the key in the attention mechanism, and K is the number of keys in the sampled attention mechanism. mqk and A mqk They represent the sampling offset and attention weight of the k-th sampling point in the m-th attention head in the deformable attention, respectively.
[0073] For text features, deformable attention is also used to encode them, allowing the model to learn more appropriate text features based on the context information of the video. After encoding the video features and text features separately, the encoded features of the two are concatenated and MLP is used for feature fusion to obtain the final fused feature F. The above process can be expressed as:
[0074] F = MLP(Concat.Encoder(V QTFA ),Encoder(Q) / )#(7)
[0075] Here, Encoder(·) represents the Transformer encoder using the deformable attention mechanism.
[0076] Fourth, after completing the multimodal feature encoding, use the prediction decoder to complete the prediction of the final target interval. Figure 4 As shown in Figure 1, the module consists of a learnable timestamp embedding and a prediction module. When predicting the start and end probabilities, the module decouples the input video frame interval and the prediction interval of the target segment, and uses the learnable timestamp embedding as input instead of the previous video frame input to the prediction module to predict the probability of different positions in the video sequence being the start or end. The prediction encoder takes the timestamp embedding E and the fused feature F as input and uses L d The standard Transformer decoder of the layer realizes the interaction of timestamp embedding and fusion features. In this process, sinusoidal position encoding PE is also added, and the process can be expressed as:
[0077]
[0078] Among them, Decoder(·) represents the standard Transformer decoder, the timestamp embedding E is used as the query of the cross attention operation, and the fused feature F is used as the key and value.
[0079] When predicting the start and end of the target time segment, a common span-based approach is used. Input into the predictor to obtain the probability of using the position embedded in each timestamp as the start and end boundary of the target time. The predictor consists of a GRU and two network layers. For each position i, the true value and predicted output [t s ,t i The intersection and union ratio between ] i , and use it as a supervisory signal for the end position to train the model. The loss function is defined as:
[0080]
[0081] After that, the normalized starting position corresponding to position i is calculated And use it as the supervision signal of the starting position. The loss function can be defined as:
[0082]
[0083] Finally, the overall loss function used for model training is defined as In the prediction stage of the model, the probability of the predicted start and end times for each location can be calculated, and the location with the highest probability is selected as the final prediction result.
[0084] In summary, the present invention designs an efficient video timing positioning system based on timing feature aggregation for the problem of video timing positioning. The proposed query-guided timing feature aggregation module can further extract time information from 2D video features and enhance video features. In addition, the deformable attention mechanism and learnable timestamp embedding can further reduce the computational cost. Ultimately, the present invention can achieve a better balance between the accuracy and efficiency of video timing positioning.
[0085] During the model training process, ActivityNet Captions and Charades-STA are used as public datasets for video temporal positioning, and their respective training sets are used for training. For the initialization of model parameters, the present invention uses ResNet-50 pre-trained on the ImageNet dataset to initialize the 2D video feature extractor, and uses GloVe to initialize the text feature extractor. When inputting the video, the video frames of the video are uniformly sampled to 64 frames, that is, T=64. For the resolution of the input video frames, it is uniformly adjusted to a size of 112×112. In the multimodal feature extractor, the unified feature dimension projected by the fully connected layer is set to 256, that is, d=256. In the multimodal encoder, the number of layers of the Transformer encoder used is set to 6 layers, that is, L e =6. In the prediction decoder, the number of layers of the Transformer decoder used is set to 3, i.e., L d = 3. In the prediction decoder module, the number of learnable timestamp embeddings is set to 128. When training the models on the above two datasets, the AdamW optimizer was used for a total of 50 rounds, and the initial learning rate during model training was set to 1×10 -4 , the batch size is 32, and the weight decay rate is set to 1×10 -4 .
[0086] The system was verified on the test sets of ActivityNet Captions and Charades-STA. The system first verified the performance and compared the accuracy of video timing positioning with multiple advanced video timing positioning methods. The specific comparison results are shown in Table 1. The above-mentioned video timing positioning methods can be divided into methods based on 2D video features and methods based on 3D video features according to the different video features used. They are marked in the "Type" column of the table, indicating "2D" and "3D" respectively. It can be seen from Table 1 that the method of the present invention performs best among all the current methods based on 2D video features, and can exceed most methods based on 3D video features. For example, on the ActivityNet Captions dataset, the system is 3.25 and 2.15 higher than TVP at IoU=0.5 and IoU=0.7, respectively.
[0087] In order to prove the high efficiency of this system, the efficiency of the model is also compared with other methods. The specific comparison results are shown in Table 2. When calculating the model parameters and floating-point operations, the feature extraction part of the text in all models is excluded because it is not trainable in most methods. The number of floating-point operations in the results is calculated when the batch size is set to 1. It can be seen from Table 2 that the method of this system has the lowest model parameters and the least floating-point operations. For example, compared with the influential classic method 2D-TAN, the model parameters of this system are about 1 / 3 of that method, and the number of floating-point operations is reduced to about 1 / 150. Compared with the recently proposed 2D video feature-based method TVP, the model of this system saves about half of the parameters and floating-point operations, while achieving higher accuracy.
[0088] Figure 5 The complete model in this system and the temporal localization results generated by the model after removing the query-guided temporal feature aggregation module are visualized in Figure 2 to further illustrate the effectiveness of the modules in this system. Figure 5 As shown in the figure, the text query contains actions such as "jumped on" and "began balancing", which requires the model to understand the relationship between video frames. The query-guided temporal feature aggregation module in this system can capture the temporal relationship between video frames at different time scales, further extract the temporal information in the video, and then more accurately predict the boundary of the target moment.
[0089] Table 1
[0090]
[0091] Table 2
[0092]
Claims
1. An efficient video temporal positioning system based on temporal feature fusion, characterized in that: It includes a multimodal feature extractor, a query-guided temporal feature aggregation module, a multimodal encoder and a predictive decoder; wherein: The multimodal feature extractor is used to extract features of input video and text, and the extracted video features are 2D features, which can reduce the amount of calculation; The query-guided temporal feature aggregation module extracts the temporal relationship information in the video at multiple different time scales through convolution kernels of different sizes under the guidance of text queries after the multimodal feature extractor extracts the video features, and obtains temporal enhanced video features to improve the problem of missing temporal information in the video 2D features; The multimodal encoder is used to fuse the video features and the text features after the temporal feature aggregation module obtains the temporal enhanced video features; specifically, the cross attention mechanism is used to further align the two modalities of text and video, and then the deformable attention is used to fuse the features of the two modalities of text and video while saving the amount of calculation; The prediction decoder is used to complete the prediction of the final video timing positioning result after the multimodal encoder obtains the fusion features; specifically, it uses learnable timestamp embedding to decouple the prediction interval and the interval of the input video frame, and finally outputs the start and end time of the video segment to be positioned, thereby achieving more accurate video timing positioning with fewer video frame inputs.
2. The efficient video timing positioning system according to claim 1, characterized in that: The multimodal feature extractor is composed of two modules: a video feature extractor and a text feature extractor; wherein: The video feature extractor is used to extract 2D features of the video to reduce the amount of calculation; specifically, the video frames of the input video are first uniformly sampled, and then the 2D ResNet model is used to extract features from each sampled video frame, and the features after the ResNet pooling layer are used as the final video features extracted by the module, which is expressed as: Where T is the number of video frames, d v is the dimension of video features; The text feature extractor uses GloVe to obtain the representation embedding of the text query as the final text feature of the module, expressed as: Where N is the number of words in the text query, and d q is the dimension of text features; After extracting video and text features, they are projected to a unified feature dimension d using two independent fully connected layers respectively.
3. The efficient video timing positioning system according to claim 2, characterized in that: In the query-guided temporal feature aggregation module, convolution kernels of different sizes are first used to capture the temporal relationship between video frames of different time scales; before performing convolution operations on the features, the channel dimension of the video feature V is first split to obtain a set of different features corresponding to convolution operations of different convolution kernel sizes, and then they are input into different aggregation branches respectively; in the operation, the input feature of the aggregation branch corresponding to the convolution kernel size k is represented as V k ; and use text queries to help extract and recover temporal information in videos; specifically, we consider text queries as guidance and use cross-attention to further enhance the features extracted using convolutions at different time scales; when using text queries as guidance, we use an MLP to map the dimension of text embedding to the dimension V of video features in each aggregation branch k ; Thus, the video feature V after temporal enhancement is obtained ′ : V k ′ =CrossAttn(Conv k (V k ),MLP k (Q))#(3) In ′ =Concat k∈{1,3,5,7} (In k ′ )#(4) Among them, CrossAttn represents the cross attention operation, Concat(·) represents the concatenation operation along the channel dimension, and Conv k (·) represents a one-dimensional convolution operation corresponding to a convolution kernel size of k; MLP k represents the MLP network layer in the aggregation branch k; after that, a cross-attention operation is used to further fuse the video features V ′ And text feature Q, get the final feature V QTFA : V QTFA =CrossAttn(V ′ ,Q)#(5) Among them, CrossAttn represents the cross attention operation, V ′ As the query in the operation, Q serves as the key and value in the operation.
4. The efficient video timing positioning system according to claim 3, characterized in that: The multimodal encoder uses L e The network layers encode the video features and text features respectively and realize the feature fusion of the two. Among them, the deformable attention mechanism is adapted to the multimodal coding. When calculating the attention mechanism, the reference point is used, and the features related to the input modality are adaptively selected according to the reference point. The encoder fuses the video features and the text features to predict the one-dimensional reference point. For video features, a cross-attention mechanism is used to enhance the model’s perception of text queries when selecting reference points, thereby helping the model learn more adaptive feature representations; this operation adds the feature V obtained in the query-guided temporal feature aggregation module to the QTFA As input, given a text query feature p corresponding to index q q In the case of a one-dimensional reference point p in the time dimension, the deformable attention operation in the video time dimension is expressed as: Where m represents the index of the head structure in the attention mechanism, k represents the index of the key in the attention mechanism, and K is the number of keys in the sampled attention mechanism; ΔP mqk and A mqk They represent the sampling offset and attention weight of the k-th sampling point in the m-th attention head in the deformable attention. For text features, deformable attention is also used to encode text features, and more appropriate text features are learned based on the context information of the video; After encoding the video features and text features respectively, the encoded features of the two are concatenated and MLP is used for feature fusion to obtain the final fused feature F. The above process is expressed as: F=MLP(Concat.Encoder(V QTFA ),Encoder(Q) / )#(7) Here, Encoder(·) represents the Transformer encoder using the deformable attention mechanism.
5. The efficient video timing positioning system according to claim 4, characterized in that: The prediction decoder is composed of a learnable timestamp embedding and a prediction module. When predicting, a span-based method is used to predict the start and end of a target time segment, and when predicting the probability of the start and end, the input video frame interval and the prediction interval of the target segment are decoupled to achieve a better balance between accuracy and efficiency. Specifically, the learnable timestamp is embedded as an input to the prediction module to predict the probability of different positions in the video sequence being the start or end. At the same time, a large number of timestamp embeddings are used as input to the prediction module to obtain a more intensive boundary prediction of the target time segment, thereby achieving a higher prediction accuracy with fewer video frames input.
6. The efficient video timing positioning system according to claim 5, characterized in that: The predictive encoder takes the timestamp embedding E and the fused feature F as input and uses L d The standard Transformer decoder of the layer realizes the interaction between timestamp embedding and fusion features; in this process, sinusoidal position encoding PE is also added to obtain richer position information. The process is expressed as: Where Decoder(·) represents a standard Transformer decoder, the timestamp embedding E is used as the query of the cross attention operation, and the fused feature F is used as the key and value; When predicting the start and end of the target time segment, a span-based approach is used; the output of the decoder in the previous step is Input into the predictor to obtain the probability of using the position embedded in each timestamp as the start and end boundary of the target time; the predictor consists of a GRU and two network layers; for each position i, the true value and predicted output [t s ,t i The intersection and union ratio between ] i , and use it as a supervisory signal for the end position to train the model; the loss function is defined as: After that, the normalized starting position corresponding to position i is calculated And use it as the supervision signal of the starting position; the loss function can be defined as: Finally, the overall loss function used for model training is defined as: In the prediction stage of the model, the probability of the predicted start and end times for each location is calculated, and the location with the highest probability is selected as the final prediction result.