A weakly supervised video temporal localization method and system

Abstract

This invention relates to a weakly supervised video temporal localization method and system. The method includes: extracting features from the video and query statements using pre-trained models; fusing information from the video and language modalities, predicting an end-to-end learnable Gaussian mask and treating it as a positive sample, considering the portion of the video suppressed by the Gaussian mask as a simple negative sample, and considering the entire video segment as a difficult negative sample; introducing a mask-based attention mechanism to collect contextual information from the video frames emphasized by the mask; using the mask-based attention mechanism in conjunction with a reconstruction loss function to reconstruct the original query statement; and designing an intra-video contrastive loss function to make the distinction between positive and negative samples more obvious. The end-to-end learnable Gaussian mask used in this invention allows for higher efficiency and quality in generating positive samples; simultaneously, the learning difficulty of the negative samples involved in this invention is greater, helping the model to more accurately complete the localization work in more complex scenarios.

Description

Technical Field

[0001] This invention relates to the field of video positioning, and specifically to a weakly supervised video temporal positioning method and system. Background Technology

[0002] Video temporal localization is an important and challenging task with potential applications in video surveillance, robotics, and other fields. Video temporal localization technology analyzes and understands video content and, based on a given query, locates the portion of the video most relevant to that query. This technology is mainly divided into two categories: fully supervised and weakly supervised. In a fully supervised environment, the training set provides the precise start and end positions of the video segment corresponding to a query; while in a weakly supervised environment, only the query and the corresponding entire video segment are provided in the training set.

[0003] The concept of a fully supervised environment was first proposed by Gao et al. in 2017 in “TALL: Temporal Activity Localization via Language Query”. Representative methods include: the 2D-TAN (2D Temporal Adjacent Networks) algorithm proposed by Zhang S. et al., which utilizes features from adjacent frames; the BPNet (Boundary Proposal Net) algorithm proposed by Xiao et al., which fuses video frame-level features and query features; the DORi (Discovering Object Relationship for Moment Localization of a Natural-Language Query in Video) algorithm proposed by Rodriguez-Opazo et al., which constructs a time-space relationship graph to discover relationships between people and objects; the MSA (Multi-stage Aggregated Transformer Network) algorithm proposed by Zhang M. et al., which attempts to utilize features from the beginning and end of segments corresponding to the query in the training set video, as well as features from frames in the middle of the video; and the DPIN (Dual Path Interaction Network) and SMIN (Multi-Level Interaction Network) algorithms proposed by Wang H. et al., which construct interaction modules at different levels to optimize the logical relationship between the query and video segments. However, these fully supervised algorithms all require a significant amount of time and effort to annotate the video, limiting their further application.

[0004] Tasks performed in weakly supervised environments can be categorized into algorithms based on reconstruction mechanisms and those using Multi-Instance Learning (MIL). A typical example of the former is the SCN (Semantic Completion Network) algorithm proposed by Lin et al. This algorithm assumes that a video segment with a high degree of matching to the query can reconstruct the entire sentence well. However, such reconstruction-based algorithms ignore information from unmatched videos and the query, which could be used for contrastive learning. Examples of the latter include the LCNet (Local Correspondence Network) algorithm proposed by Yang et al., the CRM (Cross-Sentence Relations Mining) algorithm proposed by Huang et al., and the TGA (Text-Guided Attention) algorithm proposed by Mithun et al. Given a specific query, these algorithms treat matching video-to-sentence pairs as positive samples and other video-to-sentence pairs as negative samples. They then train the model using a specially designed loss function to distinguish between positive and negative samples. However, for these MIL-based algorithms, the content of each video differs significantly, making the negative samples insufficiently challenging. This results in poor performance for these models in easily confused scenes within videos. Furthermore, algorithms like TGA and CRM use sliding windows to generate proposals. However, the proposals generated by these methods are irrelevant to the video content. During training, these algorithms generate a large number of redundant proposals and use computationally expensive Non-Maximum Suppression (NMS) as a post-processing method. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention proposes a weakly supervised video temporal localization method. On one hand, it fully combines the advantages of reconstruction mechanisms and the MIL method, treating all parts of the same video except those matching the query as simple negative samples, while introducing the entire video segment as a difficult negative sample. This increases the learning difficulty of negative samples and enhances the model's ability to correctly distinguish easily confused scenes. On the other hand, it introduces a learnable Gaussian mask to generate positive samples. These positive samples are closely related to the video content, and this approach saves the time and resources previously used for generating proposals using a sliding window.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A weakly supervised video temporal localization method for locating the video segment most relevant to a query statement includes the following steps:

[0008] (1) Data preprocessing: Features are extracted from the video and query statements using a pre-trained model;

[0009] (2) Mask generation: The information from the video and language modalities is fused to predict the Gaussian mask that can be learned end-to-end. The part of the video processed by the Gaussian mask is regarded as a positive sample, the part of the video suppressed by the Gaussian mask is regarded as a simple negative sample, and the whole video is regarded as a hard negative sample.

[0010] (3) Mask-based reconstruction: The contextual information of the video frames emphasized by the mask is collected using a mask-based attention mechanism; the query statement is reconstructed using the mask-based attention mechanism in conjunction with the reconstruction loss function, and it is assumed that the video segment that best matches the query statement can best reconstruct the query statement. The video segment with the smallest reconstruction loss is taken as the video temporal localization result.

[0011] Further, in step (1), the pre-trained CLIP (Contrastive Language-Image Pretraining) model is used to extract features from the training video, and the GloVe (Global Vectors) model is used to extract word embeddings from the query statement.

[0012] Furthermore, in step (2), the features of the video frames and words in the sentences obtained in step (1) are used to output features that fuse semantic and visual features using the Transformer model. The last layer of the fused features is extracted using a fully connected network, and the width and center of the Gaussian mask are output based on this.

[0013] Further, in step (2), based on the Gaussian mask obtained in the previous step, the part processed by the Gaussian mask is considered as a positive sample, the frame suppressed by the Gaussian mask is considered as a simple negative sample, and the entire video is considered as a difficult negative sample.

[0014] Furthermore, step (3) introduces a mask-based attention mechanism, adds weights to the attention map brought about by the mask, and collects the contextual information emphasized by the mask; a video-in-contrast loss function is designed, requiring the reconstruction results of positive samples, difficult negative samples, and simple negative samples to be from good to bad; a reconstruction loss is designed to reduce the cross-entropy loss between the reconstructed query statement and the original statement, and the reconstruction loss is used as an indicator to measure the semantic relevance between positive and negative samples and the query statement.

[0015] Furthermore, step (3) replaces the traditional attention mechanism in the Transformer with a newly designed mask-based attention model, while keeping other components unchanged. The newly designed mask-based attention model includes an encoder and a decoder that can take any mask as input and restrict the attention of the frames represented by the mask.

[0016] Furthermore, to measure the semantic relevance of positive and negative samples to the query statement, step (3) uses a mask-based reconstruction system to guess (reconstruct) the original query statement using information from frames emphasized by the mask. It is assumed that the frame with the best match to the query statement can best reconstruct the entire query statement. First, one-third of the words in the original query statement are randomly replaced with special symbols, with nouns, verbs, and adjectives being more likely to be replaced. Then, using the text features obtained in step (1) as input, a mask-based attention mechanism is used to obtain cross-modal semantic feature representations based on positive sample masks. Finally, a fully connected layer is used to output the probability distribution of the next word in the dictionary based on the positive sample mask.

[0017] Further, step (3) uses cross-entropy loss to calculate the cross-entropy loss of the above probability distribution and the true distribution. Since only positive samples and hard negative samples contain video segments related to the query, only the cross-entropy loss of positive samples and hard negative samples participates in the reconstruction loss.

[0018] Furthermore, in order to optimize the mask generation process and train the model to distinguish between positive and negative samples, the similarity between the query statement reconstructed from positive samples, difficult negative samples, and simple negative samples and the original statement should be from high to low. Therefore, in the mask generation process of step (2), an intra-video contrast loss similar to the margin ranking loss was designed.

[0019] A weakly supervised video temporal localization system employing the above method includes:

[0020] The data preprocessing module is used to extract features from the video and query statements using a pre-trained model, respectively.

[0021] The mask generation module is used to fuse the features extracted from the video and the query statement respectively, predict an end-to-end learnable Gaussian mask, treat the parts of the video processed by the Gaussian mask as positive samples, the parts of the video suppressed by the Gaussian mask as simple negative samples, and the entire video as hard negative samples.

[0022] The mask-based reconstruction module is used to collect contextual information of video frames emphasized by the mask using a mask-based attention mechanism. It also uses the mask-based attention mechanism, in conjunction with a reconstruction loss function, to reconstruct the query statement. It assumes that the video segment that best matches the query statement can best reconstruct the query statement, and takes the video segment with the smallest reconstruction loss as the video temporal localization result.

[0023] The reason this invention achieves the aforementioned effects is twofold: First, by introducing negative samples from within the video and the entire video segment, negative samples are divided into easy and difficult parts, making them more difficult for the model to learn, thereby enhancing the model's ability to distinguish easily confused scenes. Second, the Gaussian mask designed in this method is highly correlated with the video content, and its end-to-end learnability ensures that the process of generating positive samples is efficient and of high quality, improving the accuracy of localization. Attached Figure Description

[0024] Figure 1 This is a flowchart of the weakly supervised video time localization method of the present invention.

[0025] Figure 2 This is a schematic diagram of the mask-based attention mechanism structure of the present invention. Detailed Implementation

[0026] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0027] The present invention provides a weakly supervised video temporal localization method, the process of which is as follows: Figure 1 As shown, the specific steps include:

[0028] (1) Data preprocessing

[0029] Data preprocessing involves extracting features from the training video and the video to be predicted using a pre-trained CLIP model, and extracting word embeddings from the query statement using a GloVe model. Therefore, in this embodiment, the video can be represented as... A query statement can be represented as Where N is the number of video frames, D V This is the video feature dimension, where M is the number of words in the query statement, and D is... W It is the feature dimension of words.

[0030] (2) Mask generation

[0031] Information from both video and language modalities is fused to predict end-to-end learnable Gaussian masks. The input consists of video frame and word features obtained in step (1), and the output consists of positive sample masks and negative sample masks.

[0032] First, the Transformer model is used to output features that fuse semantic and visual information:

[0033]

[0034] Where E(·) is the encoder of the Transformer, D(·) is the decoder of the Transformer, and D H It is a hidden layer along the feature dimension, H = {h1, h2, ..., h N} is a fusion feature. Because h N Combining all frame and word features, the center and width of the Gaussian mask can be determined by h. N We obtain the following from the following formulas:

[0035]

[0036]

[0037] Where FC(·) represents a layer of a fully connected network. According to equations (2) and (3), the corresponding Gaussian mask m for the positive samples... p It can be obtained from the following formula:

[0038]

[0039] in, α is the weight of the i-th video frame in the Gaussian mask, while α is a hyperparameter that controls the variance of the Gaussian function.

[0040] Based on the Gaussian mask obtained from equation (4), the Gaussian mask m p The processed portion is considered a positive sample, while frames suppressed by a Gaussian mask are considered simple negative samples. e , represented as:

[0041]

[0042] Simple negative samples consist of frames from the same video that are unrelated to the query. Compared to positive samples, these frames have similar background and semantic information, making them highly susceptible to confusion. Therefore, training the model to distinguish between positive and simple negative samples can improve the model's ability to identify easily confused scenes.

[0043] Secondly, in most cases, the entire video can be considered a negative sample because it contains a large amount of redundant information unrelated to the query. Therefore, the entire video is considered a hard negative sample.

[0044]

[0045] Difficult negative samples consist of positive samples and a large number of video frames irrelevant to the query, making them more difficult for the model to distinguish. Training the model to differentiate between positive and complex negative samples helps the model locate more accurately and prevents the model from outputting overly long predictions that contain correct values.

[0046] Considering the characteristics of difficult negative samples, simple negative samples, and positive samples, the semantic relevance among them should meet the following requirements:

[0047] R(m p ,W)>R(m h ,W)>R(m e ,W) (7)

[0048] Here, R(·) is a function used to measure the relevance between the query statement W and the video segment m represented by the mask. Its specific expression will be explained in step (3).

[0049] (3) Mask-based reconstruction

[0050] like Figure 2 As shown, to ensure the differentiability of the reconstructed result with respect to the mask, this invention designs a mask-based attention mechanism. This invention replaces the traditional attention mechanism in Transformer with a newly designed mask-based attention model, while keeping other components unchanged. The newly designed mask-based attention model includes an encoder E. m (·) and decoder D m (·), they can take any mask as input and restrict the attention of the frame that the mask emphasizes.

[0051] Encoder E m (·) Set the mask m∈R N and video features As input, information is exchanged between frame features emphasized by the mask. First, a fully connected layer is used to project V onto the Query in the attention layer. Key Value The elements, including Query, Key, and Value, are defined the same as in a traditional Transformer. Next, Q is calculated. a and K a The similarity between them is used to obtain the attention map. To limit the attention to frames emphasized by mask m, m is multiplied by each row of A. After a softmax operation is performed sequentially on each row, the attention map is then summed with V. a Multiply, and output the aggregation context information:

[0052]

[0053] in, This indicates that m will be multiplied in each row of A. Decoder D m (·) Combine the mask m, the query feature W, and E m The output of (·) is used as input, and contextual information between the mask-emphasized frames and the features of each word is collected. D m The calculation of (·), except for Q a K is obtained by projecting query statement W. a and V a All are made by E m In addition to the output projection obtained by (·), and E m (·) are very similar.

[0054] Furthermore, one-third of the words in the original query statement are randomly replaced with special symbols, with nouns, verbs, and adjectives being more likely to be replaced. The textual features of the replaced query statement are then extracted using the method mentioned in step (1). As input, a mask-based attention mechanism is used to obtain a cross-modal semantic feature representation H based on positive sample masks. p :

[0055]

[0056] Next, for H p Using a fully connected layer, output the probability distribution P of the next word in the dictionary based on the positive sample mask. p .

[0057]

[0058] Where FC(·) is a fully connected layer, N w It's the size of the dictionary. This represents the first i characters of the reconstructed query statement. P is calculated using cross-entropy loss. p The difference between the true distribution and the actual distribution:

[0059]

[0060] Similarly, by taking m p Replace each with m e and m h , can be obtained separately and Since only the positive samples and the hard negative samples (the entire video segment) contain video segments relevant to the query, only the cross-entropy loss of the positive samples and the hard negative samples contributes to the reconstruction loss.

[0061]

[0062] To optimize the mask generation process and train the model's ability to distinguish between positive and negative samples, in step (2), the similarity between the reconstructed query statement and the original statement, obtained from positive samples, difficult negative samples, and simple negative samples, should be from high to low. Therefore, similar to the margin ranking loss, an intra-video contrast loss was designed.

[0063]

[0064] Here, β1 and β2 are both hyperparameters, and β1 < β2. The loss of positive samples must be at least β1 smaller than the loss of difficult negative samples and at least β2 smaller than the loss of simple negative samples.

[0065] During model training, regardless of whether the samples are positive or negative, the goal is to reconstruct the query from a video clip as accurately as possible. Used only during mask generation. Used only during the training of mask-based reconstruction. Specifically, first through... Update the reconstruction process while freezing the mask generation process. Then proceed through... Update the mask generation process while freezing the reconstruction process:

[0066]

[0067] Where θ1 is the parameter for mask generation, and θ2 is the parameter based on the mask reconstruction process. This represents the parameters for generating the final optimized mask. Let represent the parameters of the final optimized mask reconstruction. This design avoids consistently assigning low scores to predicted negative samples during the reconstruction process, thus preventing the accumulation of a trivial solution with errors in the early stages of training.

[0068] The reasoning process of this model is relatively simple. From equations (2) and (3), the center c and width w of the predicted Gaussian mask can be obtained. Therefore, the predicted temporal boundary (s,e) can be obtained:

[0069]

[0070] Since this model does not generate dense proposals using a sliding window, it abandons complex post-processing methods such as non-maximum suppression.

[0071] The experimental results below show that, compared with existing methods, the weakly supervised video temporal localization method of the present invention can achieve a higher localization accuracy.

[0072] This embodiment uses the ActivityNet Captions dataset. This dataset consists of 19,290 videos, with 37,417, 17,505, and 17,031 Moments of Interest (MoIs) on the training set, test set 1, and test set 2, respectively. The average length of the query is 14 words. The average lengths of the MoIs and the original videos are 36.2 seconds and 117.6 seconds, respectively. The videos in this dataset vary in video length, camera movement, perspective, and video quality, making localization challenging and thus contributing to demonstrating the effectiveness of our method.

[0073] Existing Method 1: The method proposed by Lin et al. in the paper "Weakly-Supervised Video Moment Retrieval via Semantic Completion Network".

[0074] Existing Method 2: The method proposed by Song et al. in the paper "Weakly-Supervised Multi-Level Attentional Reconstruction Network for Grounding Textual Queries in Videos".

[0075] Existing Method 3: The method proposed by Wang et al. in the paper "Weakly Supervised Temporal Adjacent Network for Language Grounding".

[0076] This invention: The method of this embodiment.

[0077] The experiment uses the Intersection over Union (IoU) metric to evaluate the accuracy of video localization. This metric measures the percentage of predicted video segments where the IoU with the correct result is greater than or equal to a threshold m. In the ActivityNet Captions dataset, m takes values ​​of 0.3 and 0.5.

[0078] As shown in Table 1, this invention achieves better video localization results. Existing methods suffer from lower localization results due to factors such as lower learning difficulty of negative samples, insufficient attention to negative sample information in videos, and generated proposals that are irrelevant to the video content and unlearnable. This invention constructs a learnable Gaussian mask as a positive sample, achieving a balance between efficiency and accuracy. Through comparative learning, it fully explores negative sample information within the same video and introduces more challenging negative samples, improving the model's ability to distinguish complex scenes and enhancing the accuracy of video localization.

[0079] Table 1. Comparative experimental results with existing methods (unit: %)

[0080] method IoU = 0.3 IoU = 0.5 Existing Method 1 47.23 29.22 Existing Method 2 47.01 29.95 Existing Method 3 52.45 30.01 This invention 55.68 33.33

[0081] Based on the same inventive concept, another embodiment of the present invention provides a weakly supervised video temporal localization system employing the above-described method, comprising:

[0082] The data preprocessing module is used to extract features from the video and query statements using a pre-trained model, respectively.

[0083] The mask generation module is used to fuse the features extracted from the video and the query statement respectively, predict an end-to-end learnable Gaussian mask, treat the parts of the video processed by the Gaussian mask as positive samples, the parts of the video suppressed by the Gaussian mask as simple negative samples, and the entire video as hard negative samples.

[0084] The mask-based reconstruction module is used to collect contextual information of video frames emphasized by the mask using a mask-based attention mechanism. It also uses the mask-based attention mechanism, in conjunction with a reconstruction loss function, to reconstruct the query statement. It assumes that the video segment that best matches the query statement can best reconstruct the query statement, and takes the video segment with the smallest reconstruction loss as the video temporal localization result.

[0085] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smartphone, etc.) including a memory and a processor, wherein the memory stores a computer program configured to be executed by the processor, and the computer program includes instructions for performing the steps of the method of the present invention.

[0086] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk), wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, it implements the various steps of the method of the present invention.

[0087] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A weakly supervised video temporal localization method, characterized in that, Includes the following steps: Features are extracted from the video and the query statement using pre-trained models respectively; After feature extraction, the video is represented as The query statement is represented as ,in, It is the video frame rate. It is the video feature dimension. It is the number of words in the query statement. It is the dimension of word features; The features extracted from the video and the query statement are fused together to predict an end-to-end learnable Gaussian mask. The parts of the video processed by the Gaussian mask are regarded as positive samples, the parts of the video suppressed by the Gaussian mask are regarded as simple negative samples, and the entire video is regarded as hard negative samples. Utilize a mask-based attention mechanism to collect contextual information of video frames emphasized by the mask; Using a mask-based attention mechanism in conjunction with a reconstruction loss function, the query statement is reconstructed. It is assumed that the video segment that best matches the query statement can best reconstruct the query statement, and the video segment with the smallest reconstruction loss is taken as the video temporal localization result. The mask-based attention mechanism replaces the traditional attention mechanism in Transformer with a mask-based attention model while keeping other components unchanged; the mask-based attention model includes an encoder. and decoder It takes an arbitrary mask as input and restricts the attention to the frames that the mask emphasizes; The encoder mask and video features As input, information is exchanged between frame features emphasized by the mask; first, a fully connected layer is used to... Query projected into attention Key Value On the elements; then, calculate and The similarity between them is used to obtain the attention map. To limit the mask Emphasis on frame attention, Take a ride Each row is processed sequentially using the Softmax operation, and the attention map is then summed. The aggregated context information of the output after multiplication is calculated using the following formula: in, express Will be taken Each line; The decoder mask Query features as well as The output is used as input, and contextual information between the mask-emphasized frames and the features of each word is collected.

2. The method as described in claim 1, characterized in that, The step of extracting features from the video and the query statement using pre-trained models includes: extracting features from the video using a pre-trained CLIP model, and extracting features from the query statement using a GloVe model.

3. The method as described in claim 1, characterized in that, The process of fusing features extracted from the video and query statements involves using a Transformer model to output fused features, extracting the final layer of these fused features using a fully connected network, and then outputting the width and center of a Gaussian mask based on this fused feature. The center, width, and expression of the Gaussian mask are calculated using the following formulas: in, This represents a layer in a fully connected network. It is the first Gaussian mask. The weights of each video frame, and It is a hyperparameter that controls the variance of the Gaussian function. Indicates fusion features, The center of the Gaussian mask, is the width of the Gaussian mask.

4. The method as described in claim 1, characterized in that, The simple negative sample and the difficult negative sample are calculated using the following formulas respectively: in, It is a simple negative sample. It is a difficult negative sample. It is a Gaussian mask for positive samples; the semantic relevance between the hard negative samples, the simple negative samples, the positive samples, and the query statement should meet the following requirements: in, It is used to measure query statements and video clips represented by masks A function of the correlation between them.

5. The method as described in claim 4, characterized in that, The reconstruction of the query statement includes: Randomly replace one-third of the words in the query statement with special characters, and extract the textual features of the query statement after the replacement. As input, a mask-based attention mechanism is used to obtain the cross-modal semantic feature representation based on positive sample masks, as follows: Then, output the probability distribution of the next word in the dictionary based on the positive sample mask. : in, It is a fully connected layer. It's the size of the dictionary. This represents the first i characters of the reconstructed query statement; The difference between the true and false distributions is calculated using the following formula for cross-entropy: Furthermore, by putting Replace them with and The cross-entropy loss of simple negative samples was obtained respectively. Cross-entropy loss for hard negative samples .

6. The method as described in claim 1, characterized in that, The reconstruction loss is calculated using the following formula: in, This represents the cross-entropy loss of positive samples. This represents the cross-entropy loss for difficult negative samples.

7. The method as described in claim 6, characterized in that, The intra-video contrast loss is designed such that the reconstruction results of positive samples, difficult negative samples, and simple negative samples are ranked from best to worst. The intra-video contrast loss is calculated using the following formula: in, This represents the cross-entropy loss of positive samples. The cross-entropy loss represents the loss from difficult negative samples. This represents the cross-entropy loss for simple negative samples. and All are hyperparameters, and satisfy... , The loss for positive samples must be at least less than the loss for difficult negative samples. The loss is at least smaller than that of simple negative samples. .

8. The method as described in claim 7, characterized in that, During model training, whether it is a positive sample or a negative sample, Used only during mask generation. Used only during the training of mask-based reconstruction; first through Update the reconstruction process while freezing the mask generation process; then... Update the mask generation process while freezing the reconstruction process; mask generation parameters Parameters based on the mask reconstruction process The following formulas are used to calculate the results: in, This indicates that the mask generation parameters are obtained through optimization. This indicates that the mask reconstruction parameters are obtained through optimization. Representing video features, This indicates the characteristics of the query statement.

9. A weakly supervised video temporal positioning system employing the method described in any one of claims 1 to 8, characterized in that, include: The data preprocessing module is used to extract features from the video and query statements using a pre-trained model, respectively. The mask generation module is used to fuse the features extracted from the video and the query statement respectively, predict an end-to-end learnable Gaussian mask, treat the parts of the video processed by the Gaussian mask as positive samples, the parts of the video suppressed by the Gaussian mask as simple negative samples, and the entire video as hard negative samples. The mask-based reconstruction module is used to collect contextual information of video frames emphasized by the mask using a mask-based attention mechanism. It also uses the mask-based attention mechanism, in conjunction with a reconstruction loss function, to reconstruct the query statement. The module assumes that the video segment that best matches the query statement can best reconstruct the query statement, and takes the video segment with the smallest reconstruction loss as the video temporal localization result.

Images