A point-level supervised video segment positioning method and system based on correspondence learning and application

Abstract

The application relates to a point-level supervised video segment positioning method and system based on correspondence learning and application, and belongs to the technical field of target video segment positioning. The method comprises the following steps: firstly, a pre-training model is used to extract features to obtain enhanced visual representation and enhanced text representation, and the enhanced visual representation and the enhanced text representation are processed to obtain multi-granularity text representation and multi-granularity visual representation; finally, the multi-granularity text representation is subjected to maximum pooling to obtain a representation integrating the whole sentence semantic information; in the training stage, different video segments are measured by using a Gaussian distribution combined with key frames; in the inference stage, the best matching key point of a video and a query is first recognized, and then the key point is expanded to obtain a predicted video segment most similar to the query; and the model is optimized by using model loss composed of a global alignment loss, a frame-segment correspondence loss and a segment-frame correspondence loss. The application solves the granularity and scale problems existing in video segment positioning under point supervision, and effectively improves the model performance.

Description

TECHNICAL FIELD

[0001] The present application relates to a point-level supervised video segment positioning method and system based on correspondence learning and application, belonging to the technical field of target video segment positioning (VML). BACKGROUND

[0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute the prior art.

[0003] The target video segment localization task is to return the corresponding segment of a given video according to the user's description information. This process mainly includes extracting video features, extracting text features, processing features, and locating video segments. This task has strong application value in real-world scenarios, such as: finding specific segments in surveillance videos to help the police solve cases quickly; helping to quickly review network videos to prevent the uploading of illegal videos from causing adverse effects; helping users quickly find interesting segments in long videos, etc. In view of this, many researchers are committed to studying how to effectively locate the target video segment. A key task to efficiently solve this task is to obtain a video segment localization model with excellent performance through efficient training. Specifically, the problem mainly includes two aspects: 1. How to process the dataset 2. How to use the dataset for efficient training so that the model can learn more knowledge and obtain a video segment localization model with good performance and generalization ability. In order to solve these problems, many methods have been proposed, which can be divided into three categories: full supervision, weak supervision, and point supervision. Full supervision means accurately marking the real boundaries of the video segment corresponding to the query statement. A large number of practices have proved that because full supervision can provide rich information, a model with relatively excellent performance can often be obtained through this way. However, the disadvantage of full supervision is that it requires manual marking of the real boundaries of the query statement corresponding to the video, which is too expensive. Therefore, the weak supervision method is proposed. Specifically, weak supervision does not label any video, so it greatly reduces the cost of obtaining the dataset. However, at the same time, because the dataset provides too little information, the performance of the model trained by weak supervision is often greatly different from that of the model trained by full supervision. In view of the shortcomings of the above two methods, the point supervision method is proposed. Specifically, the point supervision method will label one frame in the real boundary corresponding to the query statement. This is low-cost because we do not need to give the real boundary composed of accurate start and end markers, but only need to label any frame in the corresponding real boundary. At the same time, because of the additional labeling information, the point supervision model often shows more excellent performance than the weak supervision model. As mentioned above, point supervision achieves a good balance between model performance and labeling cost. Compared with full supervision annotation, it is more practical and flexible, and costs only 1 / 6 of the full supervision method; compared with weak supervision, it can obtain more excellent performance at a small increase in cost. Based on the above advantages, point supervision has gradually become a popular field in video segment localization, and more and more researchers are committed to exploring an efficient point supervision scheme.

[0004] Previous studies to enhance the understanding of video and text can be divided into two main methods: granularity modeling and scale modeling. Granularity modeling involves depicting the relationship between various features (such as frame-word, video segment-sentence) through the explicit boundaries existing in video-text information, promoting the differentiation of data at different entity granularity. In contrast, scale modeling comes from the complex semantic logic and inherent hierarchy within the text and visual field. This method acknowledges the implicit semantic scale in video and language interaction, which exists in an alternative modality space. However, the existing point-supervised target video segment localization model still has certain limitations:

[0005] 1) Fine-grained mismatch. Because of single-frame annotation, there may be words in the query sentence that do not correspond to the annotated frame. This is because the query sentence is a description of the segment, and the annotated frame is only an instant of this process, so it is likely that this instant only contains part of the information in the query sentence. This problem often hinders the determination of the key frame.

[0006] 2) Coarse-grained matching barrier. The incompleteness of interval annotation information hinders the correspondence between the segment and the query sentence at the coarse-grained level. Undoubtedly, this challenge constitutes a major obstacle to effective language and visual alignment.

[0007] 3) Implicit scale perception. Query sentences for the same video may differ in semantic scale. This problem is particularly evident in point-supervised methods. Because only single-frame annotation is used, resulting in a lack of effective context modeling constraints, these methods have difficulty capturing temporal relationships in videos. SUMMARY

[0008] In order to overcome the deficiencies of the prior art described above, the present application provides a point-supervised method based on low annotation cost, a target video segment localization method and system trained through explicit granularity and implicit scale correspondence learning, which improves the performance of the video segment localization model under point supervision.

[0009] The technical solutions of the present application are as follows:

[0010] The first aspect of the present application provides a point-level supervised video segment localization method based on correspondence learning, comprising:

[0011] (1) The unpruned video and the query text are first extracted by a pre-training model, and then through a fully connected layer, enhanced visual representation and enhanced text representation are obtained;

[0012] (2) The enhanced text representation and the enhanced visual representation obtained by the feature extraction module are further processed by a cross-modal multi-head attention mechanism, and the processed cross-modal text representation, cross-modal visual representation and cross-modal attention score obtained by the last layer of the cross-modal multi-head attention mechanism are input into a granularity corresponding alignment module for processing to obtain multi-granularity text representation and multi-granularity visual representation;

[0013] Finally, the multi-granularity text representation is maximally pooled to obtain a representation integrating the semantic information of the entire sentence;

[0014] (3) Segment positioning, including a training phase and an inference phase;

[0015] In the training phase, different video segments are measured using a Gaussian distribution combined with key frames;

[0016] In the inference phase, the best matching key point of the video and the query is first identified, and then the point is expanded to obtain a predicted video segment most similar to the query;

[0017] (4) The model is optimized using a model loss composed of a global alignment loss, a frame-segment correspondence loss and a segment-frame correspondence loss;

[0018] (5) The video to be positioned and the text are imported into the optimized model to obtain the positioned video segment.

[0019] Preferably, step (1) comprises the following steps: feature extraction; before multi-modal interaction, we extract visual and text features, and then learnable position embeddings are embedded into the extracted features, including:

[0020] Step 101, obtaining a visual representation

[0021] Step 1011, for an unpruned video V, we use a pre-trained CNN model to extract visual features;

[0022] Further preferably, the pre-trained CNN model is C3D or I3D, wherein the structure of the C3D model comprises 8 convolutional layers, 5 pooling layers and 2 fully connected layers, and finally a softmax output layer, each convolutional kernel has a size of 3x3x3 and a stride of 1, and the window size of all the pooling layers is 2x2x2 and the stride is also 2; the structure of the I3D model comprises a 3D convolutional layer with a kernel size of 7x7x7 and a stride of 2, followed by a 3D max-pooling layer with a kernel size of 3x3x3 and a stride of 2, then two convolutional layers with a stride of 1 and kernel sizes of 1x1x1 and 3x3x3, respectively, followed by a max-pooling layer with a kernel size of 3x3x3 and a stride of 2, and then a plurality of 3D Inception modules are stacked, which contain convolutional kernels of different sizes, including 1x1x1, 3x3x3 and 5x5x5, and finally a global average pooling layer is used to output to a softmax layer.

[0023] Step 1012, then the extracted features are dimensionally mapped by using a trainable fully connected (FC) layer to obtain the corresponding visual representation, denoted as V=v1, v2,..., vT. T wherein T represents the number of frames of the video that are sampled at intervals, and v i represents the i-th frame; in order to perceive the time of the video, a learnable position embedding (PE) is added to the model, that is, a corresponding learnable position vector is added to the visual feature of each time step to obtain an enhanced visual representation wherein d is the dimension of the visual representation.

[0024] Step 102, obtaining a text representation

[0025] Step 1021, for a given query Q, a pre-trained Glove model is used to extract text features; wherein the structure of the Glove model comprises constructing a word-word co-occurrence matrix from a large corpus, and defining a weighted loss function to minimize the difference between word vectors; the weighted loss function is represented as wherein X ij is the co-occurrence frequency of word i and word j, w i and are the word and context word vectors, respectively, b i and are bias terms, and a is an average weighting factor; f(X ij ) is a weighting function for controlling the sensitivity of the loss to different word frequencies; the model learns the low-dimensional representation of the word by minimizing the difference between the inner product of the word vector and the logarithm of the co-occurrence frequency;

[0026] Step 1022, adopt bidirectional gate recurrent unit Bi-GRU to obtain the potential sequence order of the original sentence, and obtain a new text representation Q={q1, q2,..., q L} where L represents the number of words in the query; similar to the visual representation, we obtain an enhanced text representation by adding the corresponding learnable position vector to the text representation

[0027] where the structure of the Bi-GRU model includes two layers of bidirectional GRU (Gated Recurrent Unit) layers, each layer contains 128 hidden units, followed by a fully connected layer, and finally classified by a softmax output layer.

[0028] Preferably, in step (2), the granularity alignment module will be used in subsequent steps. The present application introduces a new granularity alignment module under the point supervision video segment positioning framework, which aims to adaptively adjust the granularity relationship between different modalities. By exploring the semantic relationship between cross-modalities, this module enriches the original modality representation and supplements the existing semantic alignment with information of different granularity, ensuring the symmetry of inter-modal interaction. Unlike existing methods that interact with the smallest granularity units (frames and words) in cross-modal representation unconsciously, the present application mainly uses attention perception matrix to capture potential cross-modal semantic granularity information. Specifically, it includes:

[0029] I、The present application uses a matrix composed of cross-modal attention scores as a granularity perception matrix, as follows:

[0030]

[0031] where is the cross-modal attention score matrix, T and L are the number of frames and words, respectively, and s ij is the attention score between the i-th frame and the j-th word; the score in each column represents the attention of the word to the video frame;

[0032] II、Use the max function to aggregate the information in each row to obtain the potential prior distribution of the query words with respect to the video frames The formula is:

[0033]

[0034] III、Calculate the cosine similarity between the query sentence and the original video frame to obtain the prior distribution of the complete query sentence with respect to the video frame,

[0035]

[0036] where cos(·) is the cosine similarity function, for the enhanced visual representation in step 1012, for the enhanced text representation in step 1022;

[0037] IV. The optimal granularity information using local self-attention perception and global granularity information is collected, which is averaged and used as an adaptive visual granularity perception feature, and the process is represented as: for the potential prior distribution of the complete query sentence relative to the video frame, so that the visual granularity perception vector The vector can be used to enhance the original visual representation;

[0038] V. On this basis, the obtained features are used to reconstruct the cross-modal fusion with text features,

[0039]

[0040] for the enhanced visual representation in step 1012, for the re-modeled visual features;

[0041] After re-aggregation, the present application realizes the combination of self-attention-based fine-grained features and multi-scale visual perception features. This enhances the fine-grained entity consistency and focuses on the overall perception, thereby improving the representation.

[0042] VI. Apply the obtained features to cross-modal semantic interaction. After the above steps, we use the previous semantic knowledge to optimize the visual word correlation; finally, use the weighted average method to control the original modal feature information, so as to prevent noise interference caused by excessive introduction of cross-modal information, as follows:

[0043]

[0044] Wherein α is the average weighting factor;

[0045] For the text modality, the enhanced text representation is input into the granularity corresponding alignment module in step (2) to obtain the multi-granularity representation of the text modality Therefore, this step obtains the final multi-granularity representation and

[0046] Further preferably, step (2), the processing of the enhanced text representation and the enhanced visual representation specifically includes capturing intra-model semantics, including:

[0047] Step 201, the present application uses self-attention mechanism to use and The attention-weighted features are generated for the weighted visual features after the self-attention layer, for the weighted text features after the self-attention layer, is the enhanced visual representation in step 1012, is the enhanced text representation in step 1022;

[0048] Step 202, similarly, the present application uses a cross-modal multi-head attention mechanism to obtain a cross-modal representation, i.e. where the expression of Attn(·) is as follows:

[0049]

[0050]

[0051] Step 203, in order to serve subsequent modules, the present application extracts the cross-modal attention scores of the last layer as cross-modal attention scores, respectively: q→v and A v→q , A q→v is the granularity-aware matrix obtained when the video modality is used, as shown in equation (1), A v→q is the granularity-aware matrix obtained when the text modality is used, which will be further described in subsequent steps. The present application inputs the features and the attention scores into the granularity correspondence alignment module recorded in steps I-VI to obtain multi-granularity representations

[0052]

[0053]

[0054] where GCA(·) is the granularity correspondence alignment module, and so far we have obtained multi-granularity representations GCA represents the entire step I-VI, so this multi-granularity representation is finally calculated in step VI, and this representation is a composite representation of multi-layer granularity information, which is the output of GCA here and is helpful for cross-modal semantic alignment; equation (5) is the last step calculation formula for describing the GCA process when the video modality is used, and equation (8) is the composite output representation, and the two are not in conflict.

[0055] ​​​​Step 204, in order to integrate the semantic information of the whole sentence, we use the maximum pooling function F(·) to obtain the text representation form As follows:

[0056]

[0057] Preferably, in step (3), the segment positioning aims to locate the start and end time stamps according to the obtained multi-granularity visual representation and the text representation ; in the training stage, unlike the fully supervised scene, a random point in the target video segment can be obtained, but this point cannot be obtained in the inference stage. Since there is a difference between the two stages, the present application will describe the two stages respectively, which specifically includes:

[0058] Training stage; in the training process, the present application uses a sliding window to slice and generate candidate segments; a Gaussian distribution is used to measure the cosine similarity of the time features and the semantic features of the segments; the time features are defined as:

[0059]

[0060] where i represents the i-th frame, τ m is the index of the supervised frame, σ is a hyperparameter, and L v is the length of the video;

[0061] The semantic similarity score S i represents and the cosine similarity; these time and semantic analyses are integrated and applied to the scale correspondence learning strategy of the present application.

[0062] Inference stage; inference involves two main steps, including identifying the best matching key point of the video and query and expanding from the point to obtain the predicted video segment most similar to the query; specifically, first, by calculating the cosine similarity between the feature vector of the text query and the feature vector of each frame in the video, the frame with the highest similarity is identified as the best matching key point; then, using a sliding window, a plurality of video segments are constructed around the key point, and the frame features of each segment are aggregated to obtain the overall feature vector of the segment; next, the cosine similarity between the feature vectors of these segments and the text query is calculated again, and the segments are sorted according to the similarity scores, and finally the segment with the highest score is selected as the predicted video segment most similar to the query.

[0063] Preferably, in step (4), the scale corresponding learning is used to improve the understanding ability of the overall model (except the feature extraction model) under different scale segments, and the application uses the potential frame-segment corresponding semantic information to construct the learning goal of the overall model, because the single frame supervision information is insufficient to obtain effective segment semantics. Specifically, the optimization goal includes three parts, the first part models the semantic and temporal information of the fusion features from a global perspective; the second part uses point annotations to compare the differences between potential positive and negative samples in different intervals to effectively capture information about the action changes in the interval; and the third part aims to use the single point information of the most similar global segment to the query to enrich the prior knowledge and enhance the guidance of cross-modal semantics.

[0064] Step 401, global alignment loss; the application uses cross-entropy loss to make the similarity score of the obtained query sentence and the video frame represent the information provided by the proximity supervision frame in terms of time distance and semantic distance, and the formula is as follows:

[0065]

[0066] where G i is the Gaussian distribution weight of the i-th frame, S i is the semantic similarity score of the i-th frame, and L g is the global alignment loss.

[0067] Step 402, frame-segment corresponding loss; due to the lack of boundary annotations, the single point information is insufficient to represent the interval matching the query. The application uses a sliding window method to construct video segments containing point annotations as potential positive examples; the application constructs a binary label to facilitate interval perception: the segment containing the point annotation is marked as 1, i.e., a positive sample, and other segments are marked as 0, i.e., a negative sample, as follows,

[0068]

[0069] where M i is the i-th segment of the sliding window, y i is the positive and negative sample label; due to the potential differences in the time information of these segments, the application introduces a Gaussian distribution weight to further construct a soft label which can be expressed as:

[0070]

[0071] where G(s) and G(e) are M iGaussian distribution weights at the beginning and end; since there are inevitably multiple positive sample matrices belonging to the query, it is inappropriate to regard the similarity score learning as a 1-in-N classification problem of cross-entropy loss. To this end, the present application uses Kullback-Leibler divergence to construct the frame-moment correspondence loss As shown below,

[0072]

[0073] wherein is the cosine similarity of the moment M i and the query feature , and in the formula, represents the frame-to-moment correspondence loss designed by the present application, and N represents the number of moments obtained by the sliding window;

[0074] Step 403, moment-frame correspondence loss

[0075] Step 4031, at the beginning of the positioning process, focus on identifying the points that are most closely aligned with the query, so that the model can identify and emphasize subtle point differences. The present application separates the basic information from the positive sample moments and focuses on feature variance. Specifically, the present application extracts detailed point data reflecting cross-moment semantic labels, and uses these data to enrich point annotations with inherent differences;

[0076] This process can be represented as follows,

[0077]

[0078] wherein s and e are the start and end indices of the moment M i , S j is the cosine similarity score of the video frame with index j and the query feature modality;

[0079] Step 4032, for label construction, the present application uses hard sample labels and soft labels with Gaussian weights, i.e., formula (12) and formula (13); then the present application uses KL divergence to analyze the differences between positive samples and negative samples, and uses these insights to refine the alignment process, as shown below:

[0080]

[0081] represents the moment-to-frame correspondence loss designed by the present application;

[0082] Step 404, by combining the global loss Lg and the corresponding loss L c , (the corresponding loss includes frame-fragment correspondence loss, fragment-frame correspondence loss), the final loss of model optimization is obtained, as follows:

[0083]

[0084] Wherein β and γ are used to balance the degree of attention between fragments and frames, wherein β and γ are both set to 0.5, avoiding the model from overemphasizing sample information;

[0085] By obtaining the learning goal of the final loss, firstly, in the training process, the overall model in steps (1) and (2) above is optimized. Secondly, in the inference stage, as described in step (3), the cosine similarity between the feature vector of the text query and the feature vector of each frame in the video is calculated, and the frame with the highest similarity is identified as the best matching key point. Then, taking the key point as the center, a plurality of video fragments are constructed around it using a sliding window, and the frame features of each fragment are aggregated to obtain the overall feature vector of the fragment. Next, the cosine similarity between the feature vectors of these fragments and the text query is calculated again, and the fragments are sorted according to the similarity scores, and finally the fragment with the highest score is selected as the predicted video fragment most similar to the query.

[0086] The second aspect of the present application provides a point-level supervised video fragment positioning system based on correspondence learning, comprising:

[0087] The feature extraction module is configured to extract features from the unpruned video and the query text through a pre-trained model, and then through a fully connected layer, to obtain enhanced visual representation and enhanced text representation;

[0088] The granularity correspondence alignment module is configured to input the cross-modal text representation and the cross-modal visual representation and the corresponding last layer cross-modal attention score to obtain multi-granularity text representation and multi-granularity visual representation;

[0089] The semantic capture module is configured to process the enhanced text representation and the enhanced visual representation obtained through the feature extraction module, and then input the processed cross-modal text representation, cross-modal visual representation and corresponding last layer cross-modal attention score into the granularity correspondence alignment module to obtain multi-granularity text representation and multi-granularity visual representation. Finally, the multi-granularity text representation is maximally pooled to obtain a representation that integrates the semantic information of the entire sentence;

[0090] The segment positioning module is configured to: in a training stage, measure different video segments by combining key frames with Gaussian distribution; and in an inference stage, identify the best matching key point of a video and a query, and then expand from the point to obtain a predicted video segment most similar to the query.

[0091] The scale correspondence learning module is configured to optimize the model by using a model loss composed of a global alignment loss, a frame-segment correspondence loss and a segment-frame correspondence loss.

[0092] The third aspect of the present application provides a computer readable storage medium having a program stored thereon, the program being executed by a processor to implement the steps of the point-level supervised video segment positioning method based on correspondence learning according to the first aspect of the present application.

[0093] The fourth aspect of the present application provides an electronic device comprising a memory, a processor and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the point-level supervised video segment positioning method based on correspondence learning according to the first aspect of the present application.

[0094] The present application has the following advantages:

[0095] (1) The present application improves the ability of a video segment positioning model under point supervision by using display granularity and implicit scale correspondence learning.

[0096] (2) The present application introduces an innovative granular communication alignment module. Specifically, the module improves the explicit correspondence relationship between different granularities in different modalities, thereby solving the problem of fine-grained mismatch and coarse-grained matching hindering, so that the model can have better performance.

[0097] (3) Under the framework of point supervision, the present application develops a scale correspondence learning strategy. Through the strategy, implicit semantic scales can be captured in correspondence learning, thereby solving the existing implicit scale perception problem.

[0098] (4) The present application provides an efficient training method, which can obtain a high-performance video segment positioning model at a low cost, and realizes a good balance between cost and performance. BRIEF DESCRIPTION OF DRAWINGS

[0099] Figure 1 The present application provides a method flowchart. DETAILED DESCRIPTION

[0100] The present application is further described below by way of examples and in conjunction with the accompanying drawings, but is not limited thereto.

[0101] It should be noted that the following detailed description is exemplary in nature and is intended to provide further description of the application. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0102] It should be noted that in the embodiments of the present application, data related to audio and images are involved, and when the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of countries and regions.

[0103] It should be noted that the terms used herein are only for the purpose of describing specific embodiments, and are not intended to limit the exemplary embodiments according to the present application. As used herein, the singular form is intended to include the plural form unless the context clearly indicates otherwise, and it should also be understood that when the terms "comprise" and / or "include" are used in the specification, there is a feature, step, operation, device, component and / or combination thereof.

[0104] Embodiment 1:

[0105] As shown in Figure 1 The embodiment discloses a point-level supervised video segment positioning method based on correspondence learning, which comprises:

[0106] (1) The unpruned video and the query text are first extracted by a pre-trained model, and then the enhanced visual representation and the enhanced text representation are obtained through a full connection layer.

[0107] Step (1) comprises the following steps: feature extraction; before multimodal interaction, we extract visual and text features, and then learnable position embedding is embedded into the extracted features, including:

[0108] Step 101, obtaining visual representation

[0109] Step 1011, for the unpruned video V, we use a pre-trained CNN model to extract visual features;

[0110] The pre-trained CNN model is C3D or I3D, wherein the structure of the C3D model comprises 8 convolutional layers, 5 pooling layers and 2 fully connected layers, and finally a softmax output layer, the size of each convolution kernel is 3x3x3, the step is 1, the window size of all the pooling layers is 2x2x2, and the step is also 2; the structure of the I3D model comprises a 3D convolutional layer with a kernel size of 7x7x7 and a step of 2, followed by a 3D max pooling layer with a kernel size of 3x3x3 and a step of 2, then two convolutional layers with a step of 1, the convolution kernel sizes are 1x1x1 and 3x3x3 respectively, followed by a max pooling layer with a kernel size of 3x3x3 and a step of 2, then a plurality of 3D Inception modules are stacked, the modules contain convolution kernels of different sizes, including 1x1x1, 3x3x3 and 5x5x5, finally, a global average pooling layer is used to output to a softmax layer.

[0111] Step 1012, then the extracted features are dimensionally mapped by using a trainable fully connected (FC) layer to obtain the corresponding visual representation, denoted as V=v1, v2,..., v T , wherein T represents the number of interval-sampled frames in the video, v i represents the i-th frame; in order to perceive the time of the video, a learnable position embedding (PE) is added to the model, that is, a corresponding learnable position vector is added to the visual feature of each time step, and an enhanced visual representation is obtained , wherein d is the dimension of the visual representation;

[0112] Step 102, obtaining a text representation

[0113] Step 1021, for a given query Q, a pre-trained Glove model is used to extract text features; wherein the Glove e The structure of the model comprises constructing a word-word co-occurrence matrix from a large corpus, and defining a weighted loss function to minimize the difference between word vectors; the weighted loss function is represented as , wherein X ij is the co-occurrence frequency of word i and word j, w i and are the word and context word vectors, b i and are bias terms, α is an average weighting factor; f(X ij ) is a weighting function for controlling the sensitivity of the loss to different word frequencies; the model learns the low-dimensional representation of the word by minimizing the difference between the inner product of the word vector and the logarithm of the co-occurrence frequency;

[0114] Step 1022, adopt bidirectional gate recurrent unit Bi-GRU to obtain the potential sequence order of the original sentence, and obtain a new text representation Q={q1, q2,..., q L} where L represents the number of words in the query; similar to the visual representation, we obtain an enhanced text representation by adding the corresponding learnable position vector to the text representation

[0115] wherein the structure of the Bi-GRU model comprises two layers of bidirectional GRU (Gated Recurrent Unit) layers, each layer containing 128 hidden units, followed by a fully connected layer, and finally a softmax output layer for classification.

[0116] (2) Granularity alignment module, which will be used in subsequent steps. The present application introduces a new granularity alignment module under the point supervision video segment positioning framework, which aims to adaptively adjust the granularity relationship between different modalities. By exploring the semantic relationship between cross modalities, this module enriches the original modality representation and supplements the existing semantic alignment through information of different granularities, ensuring the symmetry of inter-modal interaction. Unlike existing methods that interact with the smallest granularity units (frames and words) in cross-modal representation unconsciously, the present application mainly uses attention perception matrix to capture potential cross-modal semantic granularity information. Specifically, it includes:

[0117] I. The present application uses a matrix composed of cross-modal attention scores as a granularity perception matrix, as follows:

[0118]

[0119] wherein is the cross-modal attention score matrix, T and L are the number of frames and words respectively, and s ij is the attention score between the i-th frame and the j-th word; the score in each column represents the attention degree of the word to the video frame;

[0120] II. The max function is used to aggregate the information in each row to obtain the potential prior distribution of the query words with respect to the video frames The formula is:

[0121]

[0122] III. Calculate the cosine similarity between the query sentence and the original video frame to obtain the prior distribution of the complete query sentence with respect to the video frame,

[0123]

[0124] wherein cos(·) is the cosine similarity function, for the enhanced visual representation in step 1012, for the enhanced text representation in step 1022;

[0125] IV. The best granularity information using local self-attention perception and global granularity information is used, which is averaged and used as an adaptive visual granularity perception feature, and the process is represented as: for the potential prior distribution of the complete query sentence relative to the video frame, so that the visual granularity perception vector The vector can be used to enhance the original visual representation;

[0126] V. On this basis, the obtained features are used to reconstruct the cross-modal fusion with text features,

[0127]

[0128] for the enhanced visual representation in step 1012, for the re-modeled visual features;

[0129] After re-aggregation, the present application realizes the combination of self-attention-based fine-grained features and multi-scale visual perception features. This enhances the fine-grained entity consistency and focuses on the overall perception, thereby improving the representation.

[0130] VI. The obtained features are applied to cross-modal semantic interaction. After the above steps, we use the previous semantic knowledge to optimize the visual word correlation; finally, the weighted average method is used to control the original modal feature information, so as to prevent noise interference caused by excessive introduction of cross-modal information, as follows:

[0131]

[0132] wherein α is the average weighting factor;

[0133] For the text modality, the enhanced text representation is input into the granularity corresponding alignment module in step (2) to obtain the multi-granularity representation of the text modality Therefore, this step obtains the final multi-granularity representation and

[0134] The enhanced text representation and the enhanced visual representation obtained by the feature extraction module are further processed by a cross-modal multi-head attention mechanism, and the cross-modal text representation and the cross-modal visual representation obtained after processing and the cross-modal attention score obtained by the last layer of the cross-modal multi-head attention mechanism are input into a granularity corresponding alignment module for processing to obtain multi-granularity text representation and multi-granularity visual representation; finally, the multi-granularity text representation is maximum-pooled to obtain a representation integrating the semantic information of the entire sentence; specifically including the following steps:

[0135] Step 201, the self-attention mechanism is used in the present application to generate attention-weighted features by using and and is the weighted visual feature after the self-attention layer, is the weighted text feature after the self-attention layer, is the enhanced visual representation in step 1012, is the enhanced text representation in step 1022.

[0136] Step 202, similarly, the present application uses a cross-modal multi-head attention mechanism to obtain a cross-modal representation, that is, and wherein the expression of Attn(·) is as follows:

[0137]

[0138]

[0139] Step 203, in order to serve subsequent modules, the present application extracts the cross-modal attention score of the last layer as a cross-modal attention score, which is A q→v and A v→q , A q→v is the granularity perception matrix obtained when the video modality is as formula (1), A v→q is the granularity perception matrix obtained when the text modality is, which will be further described in the subsequent steps. The present application inputs the features and the attention score into the granularity corresponding alignment module recorded in I-VI to obtain multi-granularity representation and

[0140]

[0141]

[0142] ​Among them, GCA(·) is the granularity correspondence alignment module. So far, we have obtained multi-granularity representations through multimodal interaction. and GCA represents the entire steps I-VI, so this multi-granularity representation is finally calculated by step VI. This representation is a composite representation of multi-layer granularity information. Here, it is the output of GCA, which helps to align cross-modal semantics. Formula (5) is the last step calculation formula of the GCA process when describing the video modality, and formula (8) is the composite output representation. The two are not contradictory.

[0143] Step 204: In order to integrate the semantic information of the entire sentence, we use the maximum pooling function F(·) to obtain the text representation as follows:

[0144]

[0145] (3) Fragment localization, including training and reasoning stages; the purpose of fragment localization is to obtain multi-granular visual representations and text representation To locate the start and end timestamps; in the training phase, unlike the fully supervised scenario, a random point in the target video segment can be obtained, but this point cannot be obtained during inference. Due to the differences between these two phases, this invention will explain these two phases separately, specifically including:

[0146] During the training phase, Gaussian distribution is combined with key frames to measure different video clips; sliding windows are used to slice And generate candidate segments; use Gaussian distribution to measure the cosine similarity of the temporal features and semantic features of the segments; the temporal feature is defined as:

[0147]

[0148] Where i represents the i-th frame, τ m is the index of the supervision frame, σ is a hyperparameter, L v is the length of the video;

[0149] Semantic similarity score S i represent and These temporal and semantic analyses are integrated and applied to the scale correspondence learning strategy of the present invention.

[0150] In the inference phase, the best matching key point of the video and the query is first identified, and then the prediction video segment most similar to the query is obtained by expanding from the point; the inference involves two main steps, including identifying the best matching key point of the video and the query and expanding from the point to obtain the prediction video segment most similar to the query; specifically, first, by calculating the cosine similarity between the feature vector of the text query and the feature vector of each frame in the video, the frame with the highest similarity is identified as the best matching key point; then, taking this key point as the center, a plurality of video segments are constructed around it using a sliding window, the frame features of each segment are aggregated to obtain the overall feature vector of the segment; then, the cosine similarity between the feature vectors of these segments and the text query is calculated again, and the segments are sorted according to the similarity scores, and finally the segment with the highest score is selected as the prediction video segment most similar to the query.

[0151] (4) The model is optimized by using a model loss composed of a global alignment loss, a frame-segment correspondence loss and a segment-frame correspondence loss;

[0152] Scale correspondence learning, in order to improve the understanding ability of the overall model (except the feature extraction model) under different scale segments, the present application uses the latent frame-segment correspondence semantic information to construct the learning goal of the overall model, because the single frame supervision information alone is not enough to obtain effective segment semantics. Specifically, the optimization goal includes three parts, the first part models the semantic and temporal information of the fused features from a global perspective; the second part uses point annotations to compare the differences between the latent positive and negative samples in different intervals to effectively capture the information about the action changes in the interval; the third part aims to use the single point information of the global segment most similar to the query, thereby enriching the prior knowledge and enhancing the guidance of cross-modal semantics;

[0153] Step 401, global alignment loss; the present application uses cross-entropy loss to make the similarity score of the obtained query statement and the video frame represent the information provided by the close supervision frame in terms of time distance and semantic distance, and the formula is as follows:

[0154]

[0155] where G i is the Gaussian distribution weight of the i-th frame, S i is the semantic similarity score of the i-th frame, L g is the global alignment loss;

[0156] Step 402, frame-moment correspondence loss; due to the lack of boundary annotations, the information of a single point is insufficient to represent the interval matching the query. The present application uses a sliding window method to construct video moments containing point annotations as potential positive examples; the present application constructs binary labels to facilitate interval awareness: moments containing point annotations are marked as 1, i.e. positive samples, and other moments are marked as 0, i.e. negative samples, as follows,

[0157]

[0158] where M i is the i-th moment of the sliding window, y i is the positive and negative sample label; due to the potential differences in the time information of these moments, the present application introduces Gaussian distribution weights to further construct soft labels which can be expressed as:

[0159]

[0160] where G(s) and G(e) are the Gaussian distribution weights at the beginning and end of M i ; due to the inevitable existence of multiple positive samples belonging to the query, it is inappropriate to regard the similarity score learning as a 1-in-N classification problem of cross-entropy loss. To this end, the present application uses Kullback-Leibler divergence to construct frame-moment correspondence loss as follows,

[0161]

[0162] where is the cosine similarity of the moment M i and the query feature , in the formula, represents the frame-to-moment correspondence loss designed by the present application, and N represents the number of moments obtained by the sliding window;

[0163] Step 403, moment-frame correspondence loss

[0164] Step 4031, at the beginning of the positioning process, focus on identifying the point that aligns most closely with the query, so that the model can identify and emphasize subtle point differences. The present application separates the basic information from the positive sample moments and focuses on feature variance. Specifically, the present application extracts detailed point data reflecting cross-moment semantic labels, and uses these data to enrich point annotations with inherent differences;

[0165] This process can be expressed as follows,

[0166]

[0167] where s and e are the start and end indices of the segment M i j cosine similarity score of the video frame indexed j with the query feature

[0168] Step 4032, for label construction, the present application uses hard sample labels and soft labels with Gaussian weights, i.e. formula (12) and formula (13); then the present application uses KL divergence to analyze the difference between positive samples and negative samples, and uses these insights to refine the alignment process, as follows:

[0169]

[0170] representing the correspondence loss from moment to frame designed by the present application;

[0171] Step 404, by combining the global loss L g and the correspondence loss L c (correspondence loss includes frame-segment correspondence loss, segment-frame correspondence loss), the final loss for model optimization is obtained, as follows:

[0172]

[0173] where β and γ are used to balance the degree of attention between segments and frames, where β and γ are both set to 0.5 to avoid the model from overemphasizing sample information.

[0174] (5) importing the video to be positioned and the text into the optimized model to obtain the positioned video segment.

[0175] Embodiment 2:

[0176] The present embodiment discloses a point-level supervised video segment positioning system based on correspondence learning, comprising:

[0177] The feature extraction module is configured to: first extract features from the unpruned video and the query text through a pre-trained model, and then obtain enhanced visual representation and enhanced text representation through a fully connected layer.

[0178] The granularity correspondence alignment module is configured to: input the cross-modal text representation and the cross-modal visual representation and the corresponding last layer cross-modal attention score to obtain multi-granularity text representation and multi-granularity visual representation.

[0179] ​​The semantic capture module is configured to process the enhanced text representation and the enhanced visual representation obtained by the feature extraction module, and input the processed cross-modal text representation and cross-modal visual representation and the cross-modal attention score of the corresponding last layer into the granularity corresponding alignment module to obtain multi-granularity text representation and multi-granularity visual representation. Finally, the multi-granularity text representation is maximum-pooled to obtain a representation integrating the semantic information of the entire sentence.

[0180] The segment positioning module is configured to: 1) in the training stage, measure different video segments by using a Gaussian distribution combined with key frames. 2) in the inference stage, identify the best matching key point of the video and the query, and then expand from the point to obtain a predicted video segment most similar to the query.

[0181] The scale correspondence learning module is configured to optimize the model by using a model loss composed of a global alignment loss, a frame-segment correspondence loss, and a segment-frame correspondence loss.

[0182] Embodiment 3

[0183] An objective of this embodiment is to provide a computer-readable storage medium.

[0184] The computer-readable storage medium stores a computer program, and the program is executed by a processor to implement the steps in the point-level supervised video segment positioning method based on correspondence learning according to Embodiment 1 of the present disclosure.

[0185] Embodiment 4

[0186] An objective of this embodiment is to provide an electronic device.

[0187] The electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor, and the processor executes the program to implement the steps in the point-level supervised video segment positioning method based on correspondence learning according to Embodiment 1 of the present disclosure.

[0188] The steps and methods involved in the above embodiments 2, 3, and 4 correspond to Embodiment 1, and the specific implementation can be referred to the relevant description part of Embodiment 1. The term “computer-readable storage medium” should be understood as including a single medium or multiple media of one or more instruction sets; it should also be understood as including any medium capable of storing, encoding, or carrying instruction sets for execution by a processor and causing the processor to perform any method in the present disclosure.

[0189] Those skilled in the art should understand that the modules or steps of the present application described above can be realized by a general computer device, or alternatively, they can be realized by program codes executable by a computing device, so that they can be stored in a storage device and executed by a computing device, or they can be respectively manufactured into individual integrated circuit modules, or a plurality of modules or steps among them can be manufactured into a single integrated circuit module. The present application is not limited to any specific combination of hardware and software.

[0190] The specific embodiments of the present application described above in conjunction with the accompanying drawings are not intended to limit the protection scope of the present application. Those skilled in the art should understand that various modifications or changes made by those skilled in the art on the basis of the technical solutions of the present application without creative labor are still within the protection scope of the present application.

Claims

1. A point-level supervised video segment localization method based on correspondence learning, characterized in that, Comprise: (1) the unpruned video and query text through a pre-training model to extract features, and then through the full connection layer, obtain enhanced visual representation and enhanced text representation; (2) the enhanced text representation and enhanced visual representation obtained by the feature extraction module are further processed by the cross-modal multi-head attention mechanism, and then the cross-modal text representation, cross-modal visual representation obtained after processing and the cross-modal attention score obtained by the last layer cross-modal multi-head attention mechanism are input into the granularity corresponding alignment module for processing, obtaining multi-granularity text representation and multi-granularity visual representation; Finally, the maximum pooling is performed on the multi-granularity text representation to obtain the representation integrating the whole sentence semantic information; The granularity corresponding alignment module comprises: Ⅰ, use the matrix composed of cross-modal attention scores as the granularity perception matrix, as follows: wherein is a cross-modal attention score matrix, T and L are the number of frames and words, respectively, and s ij is the attention score between the i-th frame and the j-th word; the scores in each column represent the degree of attention of the words to the video frames; II. Using a max function to aggregate information in each row to obtain a potential prior distribution of query words with respect to video frames The formula is: Ⅲ, calculate the cosine similarity between the query sentence and the original video frame to obtain the prior distribution of the complete query sentence relative to the video frame, where cos(·) is the cosine similarity function, is an enhanced visual representation, is an enhanced textual representation; IV. Using the best granularity information of local self-attention perception and global granularity information collected, the information is averaged and used as adaptive visual granularity perception features, and the process is represented as: For the potential prior distribution of the complete query sentence relative to the video frame, the visual granularity perception vector is obtained Ⅴ, on this basis, the obtained features are used to reconstruct the cross-modal fusion with text features, for an enhanced visual representation, for a remodeled visual feature; Ⅵ, the obtained features are applied to cross-modal semantic interaction, after the above steps, the previous semantic knowledge is used to optimize the visual word correlation; finally, the weighted average method is used to control the original modal feature information, as follows: Wherein, alpha is the average weighting factor; For the text modality, the enhanced text representation is input to a granularity alignment module in step (2) to obtain a multi-granularity representation of the text modality Thus, this step obtains a final multi-granularity representation through multi-modal interaction and (3) segment positioning, including training phase and inference phase; In the training phase, the different video segments are measured by using the Gaussian distribution combined with the key frame; In the inference phase, the best matching key point of the video and the query is first identified, and then the point is expanded to obtain the predicted video segment most similar to the query; (4) the model is optimized by using the model loss composed of global alignment loss, frame-segment correspondence loss and segment-frame correspondence loss; (5) the video to be positioned and the text are imported into the optimized model to obtain the positioned video segment.

2. The method of claim 1, wherein, Step (1) comprises the following steps: feature extraction; before multi-modal interaction, extract visual and text features, and then embed learnable positions into the extracted features, including: Step 101, obtaining visual representation Step 1011, for the unpruned video V, we use a pre-trained CNN model to extract visual features; Step 1012, then obtain the corresponding visual representation by using a layer of trainable fully connected layer to dimensionally map the extracted features, denoted by V=v1,v2,…,v T , where T represents the number of frames in the video interval sampling, v i represents the i-th frame; In the model, a learnable position embedding is added, that is, for each time step of visual features, a corresponding learnable position vector is added, and an enhanced visual representation is obtained where d is the dimension of the visual representation; Step 102, obtaining text representation Step 1021, for a given query Q, we use a pre-trained Glove model to extract text features; wherein the structure of the Glove model includes constructing a word-word co-occurrence matrix from a large corpus, defining a weighted loss function to minimize the difference between word vectors; the weighted loss function is represented as Wherein, X ij is the co-occurrence frequency of word i and word j, w i and are word and context word vectors, respectively, b i and are bias terms, α is the average weighting factor; f(X ij ) is a weighting function used to control the sensitivity of the loss to different word frequencies; the model learns a low-dimensional representation of words by minimizing the difference between the inner product of word vectors and the logarithm of co-occurrence frequency; Step 1022, adopt bidirectional gate recurrent unit Bi-GRU to obtain the potential sequence order of the original sentence, and obtain a new text representation Q={q1, q2, …, q L}, wherein L represents the number of words in the query; add the corresponding learnable position vector to the text representation to obtain an enhanced text representation The structure of the Bi-GRU model comprises two layers of bidirectional GRU layers, each layer contains 128 hidden units, followed by a fully connected layer, and finally classified by a softmax output layer.

3. The method of claim 2, wherein, In step 1011, the pre-trained CNN model is C3D or I3D, wherein the structure of the C3D model includes 8 convolutional layers, 5 pooling layers and 2 fully connected layers, and finally a softmax output layer, the size of each convolution kernel is 3x3x3, the step is 1, the window size of all the pooling layers is 2x2x2, and the step is also 2; the structure of the I3D model includes a 3D convolutional layer with a kernel size of 7x7x7 and a step of 2, followed by a 3D max pooling layer with a kernel size of 3x3x3 and a step of 2, then two convolutional layers with a step of 1, the convolution kernel sizes are 1x1x1 and 3x3x3 respectively, followed by a max pooling layer with a kernel size of 3x3x3 and a step of 2, then a plurality of 3D Inception modules are stacked, which contain convolution kernels of different sizes, including 1x1x1, 3x3x3 and 5x5x5, finally, a global average pooling layer is used to output to the softmax layer.

4. The method of claim 1, wherein, Step (2), the processing of the enhanced text representation and the enhanced visual representation specifically includes: Step 201, using by self-attention layer and attention-weighted features are generated and for weighted visual features after the self-attention layer, for weighted text features after the self-attention layer, for enhanced visual representation, for enhanced text representation; Step 202, obtaining a cross-modal representation using a cross-modal multi-head attention mechanism, i.e. and where the expression of Attn(·) is as follows: Step 203, extract the cross-modal attention score of the last layer as the cross-modal attention score, respectively: A q →v and A v→q , A q→v is the granularity perception matrix obtained when the video mode is, A v→q is the granularity perception matrix obtained when the text mode is, input the feature and the attention score into the granularity corresponding alignment module recorded in I-VI, and obtain the multi-granularity representation and where GCA(·) is a granularity corresponding alignment module, and so far, multi-granularity representations have been obtained through multi-modal interaction and GCA represents the entire steps I-VI, and the multi-granularity representation is finally calculated by step VI, and the representation is a composite representation of multi-layer granularity information, which is the output of GCA here; Step 204, in order to integrate the semantic information of the whole sentence, we use the maximum pooling function F(·) to obtain the text representation form As follows:

5. The method of claim 1, wherein, In step (3), the segment positioning is based on the obtained multi-granular visual representation and textual representation to locate the start and end time stamps; Specifically, it includes: Training phase; during the training process, a sliding window is used to slice and generate candidate segments; a Gaussian distribution is used to measure the cosine similarity of the temporal and semantic features of the segments; the temporal feature is defined as: where i denotes the i-th frame, τ m is the index of the supervision frame, σ is a hyperparameter, L v is the length of the video; semantic similarity score S i representative and cosine similarity between The inference stage; the inference involves two steps, including identifying the best matching key point of the video and the query and expanding from the point to obtain the predicted video segment most similar to the query; specifically, first, by calculating the cosine similarity between the feature vector of the text query and the feature vector of each frame in the video, the frame with the highest similarity is identified as the best matching key point; then, taking this key point as the center, a plurality of video segments are constructed around it using a sliding window, the frame features of each segment are aggregated to obtain the overall feature vector of the segment; then, the cosine similarity between the feature vectors of these segments and the text query is calculated again, and the segments are sorted according to the similarity scores, and finally the segment with the highest score is selected as the predicted video segment most similar to the query.

6. The method of claim 1, wherein, In step (4), scale correspondence learning uses latent frame-segment correspondence semantic information to construct the learning goal of the overall model, specifically, the optimization goal includes three parts, the first part models the semantic and temporal information of the fused features from a global perspective; the second part uses point annotations to compare the differences between latent positive and negative samples in different intervals; the third part uses the single point information of the global segment most similar to the query; Step 401, global alignment loss; cross-entropy loss is used to make the similarity score of the obtained query sentence and video frame represent the information provided by the close supervision frame in terms of time distance and semantic distance, the formula is as follows: where G i is the Gaussian distribution weight of the i-th frame, S i is the semantic similarity score of the i-th frame, L g is the global alignment loss; Step 402, frame-segment correspondence loss; using the sliding window method, the video segment containing the point annotation is constructed as the latent positive example; binary labels are constructed to promote interval awareness: the segment containing the point annotation is marked as 1, which is the positive sample, and other segments are marked as 0, which are negative samples, as follows, where M i is the i-th segment of the sliding window, y i is the positive and negative sample labels; Gaussian distribution weights are introduced to further construct soft labels is represented as: wherein G(s) and G(e) are M i Gaussian distribution weights at the beginning and end; frame-fragment correspondence loss is constructed using Kullback-Leibler divergence As shown below, wherein is segment M i and query features cosine similarity, in which, represents the frame-to-segment correspondence loss designed by the present application, and N represents the number of segments obtained by sliding window. Step 403, segment-frame correspondence loss Step 4031, detailed point data reflecting cross-segment semantic labels are extracted, and these data are used to enrich the point annotations with inherent differences; This process is represented as follows, where s and e are the start and end indices of segment M i S j is the cosine similarity score of the jth video frame with the query feature modality; Step 4032, for label construction, hard sample labels and soft labels with Gaussian weights are used, i.e. formula (12) and formula (13); then KL divergence is used to analyze the difference between positive samples and negative samples, and these insights are used to refine the alignment process as follows: representing the designed segment-to-frame correspondence loss of the present invention; Step 404, obtaining a global loss L g and a corresponding loss L c The corresponding loss includes frame-fragment corresponding loss and fragment-frame corresponding loss, and a final loss for model optimization is obtained as follows: Where β and γ are used to balance the degree of attention between segments and frames, where β and γ are both set to 0.

5.

7. A system for the point-level supervised video segment localization method based on correspondence learning according to claim 1, characterized in that, Comprise: The feature extraction module is configured to: first extract features from the unpruned video and the query text through a pre-trained model, and then through a fully connected layer, obtain enhanced visual representation and enhanced text representation; The granularity correspondence alignment module is configured to: input the cross-modal text representation and the cross-modal visual representation and the corresponding last layer cross-modal attention score to obtain multi-granularity text representation and multi-granularity visual representation; The semantic capture module is configured to: first process the enhanced text representation and the enhanced visual representation obtained through the feature extraction module, and then input the processed cross-modal text representation, cross-modal visual representation and the corresponding last layer cross-modal attention score into the granularity correspondence alignment module to obtain multi-granularity text representation and multi-granularity visual representation; finally, maximum pooling is performed on the multi-granularity text representation to obtain a representation that integrates the entire sentence semantic information; The segment positioning module is configured to: in the training stage, use Gaussian distribution combined with key frames to measure different video segments; In the inference stage, the best matching key point of the video and the query is first identified, and then expanded from the point to obtain the predicted video segment most similar to the query; The scale correspondence learning module is configured to: use the model loss composed of global alignment loss, frame-segment correspondence loss and segment-frame correspondence loss to optimize the model.

8. A computer readable storage medium, characterized in that, A program is stored thereon, which is executed by the processor to realize the steps in the point-level supervised video segment positioning method based on correspondence learning according to claim 1.

9. An electronic device, characterized by A memory, a processor and a program stored on the memory and executable on the processor are included, and the processor executes the program to realize the steps in the point-level supervised video segment positioning method based on correspondence learning according to claim 1. A memory, a processor and a program stored on the memory and executable on the processor are included, and the processor executes the program to realize the steps in the point-level supervised video segment positioning method based on correspondence learning according to claim 1.

Images