End-to-end video time sequence positioning system based on memory enhanced Transform

Abstract

The invention belongs to the technical field of video processing, and particularly relates to an end-to-end video time sequence positioning system based on a memory enhancement Transform. The system comprises a video and text feature coding module which is composed of a video feature coding module and a text feature coding module which are respectively used for extracting features of an input video and a text; according to the memory-enhanced Transform, a memory mechanism is introduced into an existing Transform encoder, feature coding of a video is enhanced, and the memory-enhanced Transform is used for video time sequence positioning; and the multi-granularity prediction module is used for performing multi-granularity prediction on a final video time sequence positioning result at three levels of a foreground background, a target moment interval and a video clip. According to the method, the video is processed in a fragment-by-fragment analysis mode, and the time context information is cached in the memory, so that the overall calculation overhead can be reduced, and the accuracy of video time sequence positioning is improved; and a leading video time sequence positioning effect is obtained on the public data set.

Description

Technical Field

[0001] The present invention belongs to the technical field of video processing, and in particular relates to an end-to-end video timing positioning system based on a memory-enhanced Transformer. Background Art

[0002] In recent years, video temporal localization has become an important research topic in multimodal learning. Video temporal localization aims to achieve cross-text and video retrieval, and its goal is to locate the most relevant segments in the complete video based on a given text query. The results are usually represented by the start and end time of the segment.

[0003] Current video temporal localization can generally be divided into two categories: methods based on temporal proposals and methods without temporal proposals. Methods based on temporal proposals generally generate a large number of temporal proposals of different sizes and positions, and then calculate the matching degree between the text query and each temporal proposal. Methods without temporal proposals do not rely on pre-generated temporal proposals. Such methods can be further divided into two categories: regression-based methods and span-based methods. Regression-based methods directly regress the boundaries of the target interval, while span-based methods predict the probability of each video frame being the start or end moment.

[0004] However, among the existing methods, no matter which of the above categories they belong to, most of them work in a two-stage manner, and their video encoders are only used for offline video feature extraction, and parameters cannot be updated during model training. This two-stage method has the following disadvantages. First, this type of method uses a pre-trained video feature extractor, whose parameters are fixed during the training process, so when directly used in the video timing positioning task, it cannot extract the most suitable video features for the task. Secondly, the computational resource consumption of its video feature extractor is generally large, and feature extraction is often completed offline, so it is difficult to use it in online video timing positioning.

[0005] To solve this problem, the present invention proposes an efficient end-to-end video temporal localization system based on memory-enhanced Transformer. Specifically, each video is evenly divided into non-overlapping video segments, and then the system processes the video in a segment-by-segment analysis manner, ultimately achieving higher localization performance while also achieving higher efficiency. Summary of the invention

[0006] The purpose of the present invention is to provide an end-to-end video temporal positioning system based on memory-enhanced Transformer with good positioning performance and high efficiency.

[0007] Video temporal positioning is to find the video clips in the video that are most relevant to the text query given a text query and a complete video. Since video processing requires a very high computational cost, a pre-trained video encoder is usually used to extract video features in advance. The parameters of the video encoder are fixed during model training, which often limits the performance of the model's video temporal positioning. To solve this problem, the present invention proposes an efficient end-to-end video temporal positioning system based on a memory-enhanced Transformer. Each video is evenly divided into non-overlapping video clips, and then the system processes the video in a segment-by-segment analysis manner, ultimately achieving higher positioning performance while also achieving higher efficiency.

[0008] The end-to-end video temporal positioning system based on memory-enhanced Transformer provided by the present invention comprises a video and text feature encoding module, a memory-enhanced Transformer, and a multi-granularity prediction module; wherein:

[0009] The video and text feature encoding module is composed of a video feature encoding module and a text feature encoding module, which are used to extract features of input video and text respectively;

[0010] The memory-enhanced Transformer introduces a memory mechanism into the traditional Transformer encoder to enhance the feature encoding of the video and use it for video temporal positioning;

[0011] The multi-granularity prediction module performs multi-granularity prediction on the final video timing positioning result at three levels: foreground background, target time interval and video clip.

[0012] Further:

[0013] The video and text feature encoding module is composed of a video feature encoding module and a text feature encoding module, wherein:

[0014] The video feature encoding module uses MoViNet-A2[1] to extract video features, which can achieve high computational efficiency while extracting better video features. Specifically, given a video clip MoViNet-A2 is used to extract features from the video clip. MoViNet-A2 is a video feature extractor that does not downsample in the temporal dimension of the video. Therefore, after encoding the video, it can still maintain full resolution in the temporal dimension, making it more suitable for time-sensitive tasks such as video timing positioning. The features after video encoding can be expressed as where d v is the feature dimension. In addition, this module also adds vAdd video modal type embed on Then a linear layer is used to map the video features to a uniform dimension d, and the final features can be expressed as

[0015] The text feature encoding module is used to extract features from the input text query. This module uses GloVe embedding, and the features of the input text can be expressed as Where K is the length of the input text, d t is the dimension of text feature encoding. After that, a [CLS] classification tag is spliced ​​to the beginning of the sentence as the overall embedding feature of the sentence. Similar to the video feature encoding module, this module t Add text modal type embed on To increase the context information in the text feature encoding, position encoding is also added Then a linear layer is used to map the video features to a uniform dimension d, and the final features can be expressed as

[0016] The memory-enhanced Transformer introduces a memory mechanism into the standard Transformer encoder to enhance feature representation and is used for video temporal positioning. The module consists of L layers of Transformer network layers, each of which contains an attention mechanism and a multi-layer perceptron. Memory enhancement is that the input of each layer of Transformer contains memory information. Specifically, the input of each layer of Transformer consists of cached memory information m (l-1) , Video Features and text features concatenated, where l is the index of the network layer, and Each layer m (l) The input memory information is cached in the previous round. These cached memories are stored in a first-in-first-out queue and are constantly updated. Each layer of Transformer can be calculated as follows:

[0017]

[0018] The key vector (Key) and the value vector (Value) are predefined disease categories. As the query vector (Query), [·°·] represents the concatenation operation along the input sequence dimension, MHA represents the multi-head attention operation, and its first input is the query vector, and the second input is the key vector and value vector. MLP represents multi-layer perceptron, and LN represents layer normalization. In addition, position information is added to the input of the Transformer layer. Specifically, the position information is encoded using relative position, and a learnable bias parameter is added to each input of all head structures. Compared with other more complex position encodings, this method can achieve better results in video temporal positioning, while also simplifying the design. Since the position information has been added when extracting text features, this relative position encoding is only used on the input video features.

[0019] The memory information in this module draws on the memory mechanism of Transformer in natural language processing and adapts it to the multimodal task of video temporal localization. When updating the memory information in the queue, the video clip features after each iteration are placed at the end of the memory queue. The cache memory m of each Transformer layer (l) Set a maximum length In addition, the module also designs a cache memory compression mechanism. When there is an older memory that needs to be deleted, it will be compressed instead of being deleted directly, and the compressed memory will be saved in an additional compressed memory queue. The above compressed memory is represented by cm (l) , use a one-bit convolution in the time dimension to reduce the length of the cache memory when compressing the memory, and record the compression rate as c. When the length of the compressed memory queue reaches the maximum length After that, the oldest compressed memory will be deleted. When using the segment-by-segment analysis method to process the complete video, the memory input to the Transformer layer is the memory queue m (l) and compressed memory queue cm (l) The memory after splicing. To make the description more concise, cm is omitted in the above formula (l) .

[0020] The multi-granularity prediction module predicts the final video timing positioning results at different granularities. First, a timing proposal is generated with the position of the video frame as the center. Then, the boundary of the timing proposal is regressed through a two-layer convolutional network. The process is expressed as:

[0021]

[0022] in, Represents the features of the last layer in the memory-enhanced Transformer, Represents the distance from the center point of the prediction time interval to its start and end positions, Conv bdrepresents the convolution operation used to generate the boundary. Therefore, the above temporal proposal centered on the position of the video frame can be expressed as in and Indicates the start and end time respectively.

[0023] Afterwards, the confidence of each time series proposal generated above is predicted at three different granularities, which is to achieve the multi-granularity prediction of this part.

[0024] The first granularity is the confidence prediction of the foreground and background, indicating whether the current frame is in the target time interval. The prediction process can be expressed as:

[0025]

[0026] Among them, the component g i Represents the confidence score of the i-th video frame as a foreground. Conv fg Represents the convolution operation used for foreground background confidence prediction.

[0027] The second granularity is the prediction of the intersection-over-union ratio. Although the confidence score of the foreground can distinguish the foreground from the background, the prediction of the confidence scores of all video frames in the foreground is not detailed enough. Therefore, the present invention uses the intersection-over-union ratio as a more fine-grained supervision. Specifically, the convolutional layer is first used to predict the intersection-over-union ratio of the video frame corresponding to the prediction result:

[0028]

[0029] Among them, the component u i Conv represents the intersection-over-union ratio of the predicted temporal proposal centered on the i-th video frame and the true target time interval. iou Represents a convolution operation that predicts the Intersection-Over-Union (IoU) ratio.

[0030] The third granularity is the prediction of the matching degree of the video clip. This part uses the [CLS] tag of the last Transformer layer as the overall feature representation of the entire video clip and inputs it into a multi-layer perceptron to predict the matching degree m of the video clip. clip The confidence is shared among the temporal proposals of each video frame in a video clip.

[0031] When training the model, the video is processed in a segment-by-segment analysis manner, so the loss function is calculated at the video segment level (consisting of T video frames). The first loss function of the model training is used to regress the boundary of the predicted target time interval, and its definition is as follows:

[0032]

[0033] Among them, b = (s, e) is the true value of the target time interval. In order to make the model training more stable, this loss function is only used on the video frames belonging to the foreground.

[0034] The second loss function of the model training is used to optimize the prediction of foreground and background. This loss function uses the cross entropy loss function, which is defined as follows:

[0035]

[0036] Among them, BCE represents binary cross entropy, is the indicator function. After that, the true value of the intersection-over-intersection ratio is used to supervise the intersection-over-intersection ratio predicted by the model. Since the value range of the intersection-over-intersection ratio is between 0 and 1, cross entropy can be used to optimize it:

[0037]

[0038] Among them, SG represents the stop gradient operation, which can calculate the intersection-and-union ratio as the training label of the above loss function based on the boundary prediction, while updating the parameters of the boundary prediction branch without affecting the update of the parameters of the boundary prediction branch.

[0039] For the prediction of the matching degree of the video clip, the present invention also designs a loss function for optimization. The loss function uses the proportion of the foreground video frames in a video clip to the total number as a supervisory signal, which is defined as follows:

[0040]

[0041] Finally, the overall loss function used to train the model is:

[0042]

[0043] During the model inference process, all video clips are processed in sequence, and finally NT time series proposals can be generated. The confidence of each time series proposal can be defined as f (i) =g i +u i +m clip Finally, among all NT time series proposals, the time series proposal with the highest confidence score is used as the final predicted video time series positioning result of the system.

[0044] The efficient end-to-end video temporal positioning system based on memory-enhanced Transformer of the present invention has the following working process:

[0045] (i) First, feature extraction is performed on the input video and text query respectively, where the video feature extractor uses a lightweight model, which does not cause information loss caused by downsampling in the temporal dimension of the video, and allows the entire model to be trained end-to-end;

[0046] (ii) Based on the extracted features of the video and text query, the memory-enhanced Transformer is used to further encode and fuse them;

[0047] (iii) Based on the features obtained by the memory-enhanced Transformer, a multi-granularity prediction method is used to achieve more accurate video timing positioning.

[0048] Advantages of the present invention include:

[0049] Firstly, the present invention proposes for the first time a system for realizing efficient video timing positioning in a divide-and-conquer manner, wherein the system processes the video in a segment-by-segment analysis manner and can cache the temporal context information in the memory.

[0050] Secondly, the present invention designs a novel multi-granularity prediction method, which can output prediction results at the video frame level and the video segment level, and can further improve the accuracy of video timing positioning.

[0051] Finally, the present invention achieves leading video temporal localization effects on the public datasets ActivityNet Captions, Charades-STA and TACoS datasets. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a framework diagram of the end-to-end video timing positioning system of the present invention.

[0053] Figure 2 This is a flow chart of video segment-by-segment processing in the present invention. DETAILED DESCRIPTION

[0054] The specific details of the present invention are described below in conjunction with the accompanying drawings.

[0055] like Figure 1 As shown, the memory-enhanced Transformer-based efficient end-to-end video temporal positioning system of the present invention includes a video and text feature encoding module, a memory-enhanced Transformer and a multi-granularity prediction module. The workflow of the system is:

[0056] 1. The video feature encoding module uses MoViNet-A2 to extract video features, which can achieve high computational efficiency while extracting better video features. Given a video clip MoViNet-A2 is used to extract features from the video clip. The features after video encoding can be expressed as where d v is the feature dimension. In addition, this module also adds vAdd video modal type embed on Then a linear layer is used to map the video features to a uniform dimension d, and the final features can be expressed as

[0057] The text feature encoding module realizes feature extraction of input text query. This module uses GloVe embedding, and the features of the input text can be expressed as Where K is the length of the input text, d t is the dimension of text feature encoding. After that, a [CLS] classification marker is concatenated to the beginning of the sentence as the overall embedding feature of the sentence. Similar to the video feature encoding module, this module adds the text modality type embedding t to the text feature. t To increase the contextual information in the text features, position encoding is also added Then a linear layer is used to map the video features to a uniform dimension d, and the final features can be expressed as

[0058]

[0059] Second, after extracting the features of the video and text, the memory-enhanced Transformer is used to fuse the features. This module consists of L layers of Transformer networks, each of which contains an attention mechanism and a multi-layer perceptron MLP. Specifically, the input of each layer of Transformer consists of cached memory information m (l-1) , Video Features and text features concatenated, where l is the index of the network layer, and Each layer m (l) The input memory information is cached in the previous round. These cached memories are stored in a first-in-first-out queue and are constantly updated. The calculation process of each layer of Transformer is as follows:

[0060]

[0061]

[0062] Among them, [·°·] represents the concatenation operation along the input sequence dimension, MHA represents the multi-head attention operation, whose first input is the query and the second input is the key and value. In addition, position information is added to the video feature input of the Transformer layer. Specifically, the position information is encoded using relative position, and a learnable bias parameter is added to each input of all head structures.

[0063] When updating the memory information in the queue, the video clip features after each iteration are placed at the end of the memory queue. This module also designs a cache memory compression mechanism. When older memories need to be deleted, they will be compressed and saved in an additional compressed memory queue. The cache memory m of each Transformer layer (l) Set a maximum length When the length of the cache memory queue reaches the maximum length and there is still new memory to be stored, the oldest cache memory will be compressed. The compressed memory is represented by cm (l) , use a one-bit convolution in the time dimension to reduce the length of the cache memory when compressing the memory, and record the compression rate as c. When the length of the compressed memory queue reaches the maximum length After that, the oldest compressed memory will be deleted. When the system uses a segment-by-segment analysis method to process the complete video, the memory input to the Transformer layer is the memory queue m (l) and compressed memory queue cm (l) The memory after splicing. To make the description more concise, cm is omitted in the above formula (l) .

[0064] 3. After completing feature fusion, use the multi-granularity prediction module to predict the final video timing positioning results at different granularities. Figure 1 As shown, the module includes a positioning module and a multi-layer perceptron module, wherein the positioning module is used for predicting boundary regression, foreground confidence score and intersection-over-union ratio, and the multi-layer perceptron module is used to predict the matching score at the fragment level.

[0065] First, a temporal proposal is generated with the position of the video frame as the center. Then, the boundary of the temporal proposal is regressed through a two-layer convolutional network. The process is expressed as:

[0066]

[0067] in, Represents the features of the last layer in the memory-enhanced Transformer, represents the distance from the center point of the predicted time interval to its start and end positions. Therefore, the above time series proposal centered on the position of the video frame can be expressed as in and Indicates the start and end time respectively.

[0068] Afterwards, the confidence of each time series proposal generated above is predicted at three different granularities. The first granularity is the confidence prediction of the foreground background, indicating whether the current frame is in the target time interval. The prediction process can be expressed as:

[0069]

[0070] Among them, g i represents the confidence score of the i-th video frame as the foreground.

[0071] The second granularity is the prediction of the intersection-over-union ratio. First, use the convolutional layer to predict the intersection-over-union ratio of the video frame corresponding to the prediction result:

[0072]

[0073] Among them, u i It represents the intersection-over-union ratio of the predicted temporal proposal centered at the i-th video frame and the true target time interval.

[0074] The third granularity is the prediction of the matching degree of the video segment. This part uses the [CLS] tag of the last Transformer layer as the overall feature representation of the entire video segment and inputs it into a multi-layer perceptron to predict the matching degree m of the video segment. clip The confidence is shared among the temporal proposals of each video frame in a video clip.

[0075] The model is trained by processing videos in a segment-by-segment analysis manner (e.g. Figure 2 As shown in Figure 2, the loss function is calculated at the video segment level (consisting of T video frames). The first loss function of the model training is used to regress the boundary of the predicted target time interval, which is defined as follows:

[0076]

[0077] Among them, b = (s, e) is the true value of the target time interval. In order to make the model training more stable, this loss function is only used on the video frames belonging to the foreground.

[0078] The second loss function of the model training is used to optimize the prediction of foreground and background. This loss function uses the cross entropy loss function, which is defined as follows:

[0079]

[0080] Among them, BCE represents binary cross entropy, is the indicator function. After that, the true value of the intersection-over-intersection ratio is used to supervise the intersection-over-intersection ratio predicted by the model. Since the value range of the intersection-over-intersection ratio is between 0 and 1, cross entropy can be used to optimize it:

[0081]

[0082] Among them, SG represents the stop gradient operation, which can calculate the intersection-and-union ratio based on the boundary prediction and use it as the training label of the loss function. While updating the parameters of the intersection-and-union ratio prediction branch, it does not affect the update of the parameters of the boundary prediction branch.

[0083] For the prediction of the matching degree of video clips, the present invention also designs a loss function for optimization. The loss function uses the proportion of foreground video frames in a video clip to the total number as a supervisory signal, which is defined as follows:

[0084]

[0085] Finally, the overall loss function used to train the model is:

[0086]

[0087] During the model inference process, all video clips are processed in sequence, and finally NT time series proposals can be generated. The confidence of each time series proposal can be defined as f (i) =g i +u i +m clip Finally, among all NT time series proposals, the time series proposal with the highest confidence score is used as the final predicted video time series positioning result of the system.

[0088] In summary, the present invention is aimed at the problem of video timing positioning, proposes a new segment-by-segment analysis method to process videos, and realizes end-to-end trained video timing positioning. This system uses MoViNet for video encoding, and the extracted features avoid the loss caused by downsampling in the video time dimension, and can extract video features more effectively. The memory-enhanced Transformer uses memory to enhance the Transformer encoder, which can enhance its ability to process long videos and reduce the overall computational complexity. The system also designs a multi-granularity prediction module to further improve the positioning accuracy with almost no increase in computational complexity. Ultimately, the present invention improves overall efficiency while ensuring high video timing positioning performance.

[0089] During the training process of the model, ActivityNet Captions, Charades-STA and TACoS are used as public data sets for video temporal positioning, and their respective training sets are used during training. For the video feature encoding module, the present invention uses MoViNet-A2 pre-trained on the Kinetics data set as the initialization of the model parameters. For video data enhancement, common data enhancement methods such as multi-scale cropping and random horizontal flipping are adopted. For the input videos on the above three data sets, the video frames of the video are uniformly sampled to 64, 96 and 160 frames respectively. In the video and text feature encoding modules, for the Charades-STA data set, the mapped unified dimension d is set to 384, and for the ActivityNet Captions and TACoS data sets, the mapped unified dimension d is set to 512. For the memory-enhanced Transformer module, the number of layers L is set to 6. In this module, the length of the cache memory is adjusted for different data sets. For the ActivityNetCaptions and Charades-STA data sets, the maximum length of the cache memory Set to 16, the maximum length of compressed memory Set to 4. For the TACoS dataset, the maximum length of the cache memory Set to 32, the maximum length of compressed memory Set to 8. When training the model on the above three datasets, the AdamW optimizer is used. The initial learning rate during model training is set to 4×10 -5 , the weight decay rate is set to 1×10 -4 .

[0090] In order to verify the effectiveness of this system, tests were conducted on the above three datasets and compared with many recent state-of-the-art methods. Table 1 shows the performance comparison on the three datasets. On the Charades-STA dataset, this system outperforms other methods in terms of R@1, IoU=0.7 and achieves comparable results on other metrics. This shows that this system is able to generate higher quality prediction results. On the ActivityNet Captions dataset, this system also achieves the best performance in terms of R@1, IoU=0.7. On the TACoS dataset, this system surpasses all other methods, demonstrating the effectiveness of this system in processing long videos such as the TACoS dataset.

[0091] In order to verify the efficiency of this system, it is compared with several existing classic methods on the ActivityNet Captions dataset. During the test, the number of model parameters, floating-point operations, and queries per second are used as indicators to measure efficiency. When testing the number of queries per second, the batch size is set to 1 and the NVIDIA RTX 6000 GPU is used for testing. The results are shown in Table 2. This system can achieve high performance while also having great advantages in efficiency. When the "Whether to include the video encoder" in the table indicates the calculation efficiency index, the model includes the video encoder. If included, its evaluation includes the sum of the calculation amount / running time of the video encoder and the calculation amount / running time of the video positioning model. From the results in the table, it can be seen that the calculation amount of other existing methods will increase sharply after considering the feature extraction stage. In the end-to-end scenario that is closer to actual application, this system is significantly better than other methods in terms of calculation and reasoning speed. Compared with the MMN method, the number of floating-point operations of this system is 1 / 900 of that of this method, and the actual running speed is 11 times faster.

[0092] References

[0093] [1]Kondratyuk D, Yuan L, Li Y, et al. Movinets: Mobile video networks for efficient video recognition [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2021: 16020-16030.

[0094] Table 1

[0095]

[0096] Table 2

[0097]

Claims

1. An end-to-end video temporal positioning system based on memory-enhanced Transformer, characterized in that: It includes video and text feature encoding modules, memory-enhanced Transformer, and multi-granularity prediction modules; among them: The video and text feature encoding module is composed of a video feature encoding module and a text feature encoding module, which are used to extract features of input video and text respectively; The memory-enhanced Transformer introduces a memory mechanism into the existing Transformer encoder to enhance the feature encoding of the video and use it for video temporal positioning; The multi-granularity prediction module performs multi-granularity prediction on the final video timing positioning result at three levels: foreground background, target time interval and video clip.

2. The end-to-end video timing positioning system according to claim 1, characterized in that: In the video and text feature encoding module: The video feature encoding module uses MoViNet-A2 to extract video features; specifically, given a video clip MoViNet-A2 is used to extract features from the video clip; the features after video encoding are expressed as where d v is the characteristic dimension; In addition, in the video feature f v Add video modal type embed on Then a linear layer is used to map the video features to a uniform dimension d, and the final feature representation is The text feature encoding module is used to extract features from the input text query; this module uses GloVe embedding, and the feature representation of the input text is Where K is the length of the input text, d t is the dimension of text feature encoding; then, a [CLS] classification marker is spliced ​​to the beginning of the sentence as the overall embedding feature of the sentence; Similar to the video feature encoding module, this module is based on the text feature f t Add text modal type embed on To increase the context information in the text feature encoding, position encoding is also added Then a linear layer is used to map the video features to a uniform dimension d, and the final feature representation is 3. The end-to-end video timing positioning system according to claim 2, characterized in that: The memory-enhanced Transformer is composed of L layers of Transformer network layers, each of which includes an attention mechanism and a multi-layer perceptron. The memory enhancement is that the input of each layer of Transformer contains memory information. Specifically, the input of each layer of Transformer consists of cached memory information m (l-1) , Video Features and text features concatenated, where l is the index of the network layer, and Each layer m (l) The input memory information is cached in the previous round. These cached memories are stored in a first-in-first-out queue and are constantly updated. Each layer of Transformer is calculated as follows: in, represents the concatenation operation along the input sequence dimension, MHA represents the multi-head attention operation, whose first input is the query vector (Query), and the second input is the key vector (Key) and the value vector (Value); MLP represents the multi-layer perceptron, and LN represents the layer normalization; in addition, the position information is added to the input of the Transformer layer; specifically, the position information is encoded using relative position, and a learnable bias parameter is added to each input of all head structures; The memory information in this module draws on the memory mechanism of Transformer in natural language processing and adapts it to the multimodal task of video temporal localization. When updating the memory information in the queue, the video clip features after each iteration are placed at the end of the memory queue. The cache memory m of each Transformer layer (l) Set a maximum length In addition, a cache memory compression mechanism is designed. When an older memory needs to be deleted, it is compressed and stored in an additional compressed memory queue. The compressed memory is represented by cm (l) , use a one-bit convolution in the time dimension to reduce the length of the cache memory when compressing the memory, and record the compression rate as c; when the length of the compressed memory queue reaches the maximum length After that, the oldest compressed memory is deleted; when using the segment-by-segment analysis method to process the complete video, the memory input to the Transformer layer is the memory queue m (l) and compressed memory queue cm (l) Memories after splicing.

4. The end-to-end video timing positioning system according to claim 3, characterized in that: The multi-granularity prediction module predicts the final video timing positioning result at different granularities, specifically including: First, a temporal proposal is generated with the position of the video frame as the center; then, the boundary of the temporal proposal is regressed through a two-layer convolutional network. The process is expressed as: in, Represents the features of the last layer in the memory-enhanced Transformer, Represents the distance from the center point of the prediction time interval to its start and end positions, Conv bd represents the convolution operation used to generate the boundary; thus, the temporal proposal centered at the position of the video frame is represented as in, and Respectively indicate the start and end time; Then, for each of the generated time series proposals, the confidence is predicted at three different granularities: The first granularity is the confidence prediction of the foreground and background, indicating whether the current frame is in the target time interval; its prediction process is expressed as: Among them, g i Indicates the confidence score of the i-th video frame as the foreground; Conv fg Represents the convolution operation used for foreground background confidence prediction; The second granularity is the prediction of the intersection-over-union ratio. Specifically, the intersection-over-union ratio is used for more fine-grained supervision. First, the convolutional layer is used to predict the intersection-over-union ratio of the video frame corresponding to the prediction result: Among them, u i Conv represents the intersection-over-union ratio of the predicted temporal proposal centered on the i-th video frame and the true target time interval; iou represents the convolution operation of predicting the intersection-over-union ratio; The third granularity is to predict the matching degree of the video clip. Specifically, the [CLS] tag of the last Transformer layer is used as the overall feature representation of the entire video clip, and is input into a multi-layer perceptron to predict the matching degree m of the video clip. clip ; This confidence is shared among the temporal proposals of each video frame in a video clip.

5. The end-to-end video timing positioning system according to claim 4, characterized in that: During model training, the video is processed in a segment-by-segment analysis manner, so the loss function is calculated at the video segment level; the first loss function of model training is used to regress the boundary of the predicted target time interval, and its definition is as follows: Where b = (s, e) is the true value of the target time interval; to make the model training more stable, this loss function is only used on the video frames belonging to the foreground; The second loss function of the model training is used to optimize the prediction of foreground and background; this loss function adopts the cross entropy loss function, which is defined as follows: Among them, BCE represents binary cross entropy, is the indicator function, T is the number of video frames in the video clip; then, the true value of the intersection-over-union ratio is used to supervise the intersection-over-union ratio predicted by the model; since the value range of the intersection-over-union ratio is between 0 and 1, cross entropy is used to optimize it: Among them, SG represents the stop gradient operation, which calculates the intersection-over-union ratio based on the boundary prediction as the training label of the above loss function; For the prediction of the matching degree of video clips, a loss function is also designed for optimization; the loss function uses the proportion of foreground video frames in a video clip to the total number as the supervisory signal, which is defined as follows: Finally, the overall loss function used to train the model is: During the model inference process, all video clips are processed in sequence, and finally NT time series proposals are generated; the confidence of each time series proposal is defined as f (i) =g i +u i +m clip ,Finally, among all NT time sequence proposals, the time sequence proposal with the highest confidence score is used as the final predicted video time sequence positioning result of the system.

6. The end-to-end video timing positioning system according to claim 5, characterized in that: The workflow is: (i) First, feature extraction is performed on the input video and text query respectively, where the video feature extractor uses a lightweight model without information loss caused by downsampling in the temporal dimension of the video, and the entire model is trained end-to-end; (ii) Based on the extracted features of the video and text query, the memory-enhanced Transformer is used to further encode and fuse them; (iii) Based on the features obtained by the memory-enhanced Transformer, a multi-granularity prediction method is used to achieve more accurate video timing positioning.

Images