Point-level supervision video clip positioning method and system based on corresponding learning and application

Abstract

The invention relates to a point-level supervision video clip positioning method and system based on corresponding learning and application, and belongs to the technical field of target video clip positioning, and the method comprises the steps: firstly extracting features through a pre-training model, obtaining an enhanced visual representation and an enhanced text representation, processing the enhanced visual representation and the enhanced text representation, and obtaining a pre-training model; obtaining a multi-granularity text representation and a multi-granularity visual representation; finally, the multi-granularity text representation is subjected to maximum pooling, and representation integrating semantic information of the whole sentence is obtained; in the training stage, Gaussian distribution is combined with key frames, and different video clips are measured; in the reasoning stage, the best matching key point of the video and the query is identified firstly, and then the point is expanded to obtain a predicted video clip most similar to the query; and a model loss optimization model formed by three parts of global alignment loss, frame-fragment corresponding loss and fragment-frame corresponding loss is utilized. According to the method, the granularity and scale problems existing in video clip positioning under point supervision are solved, and the model performance is effectively improved.

Description

Technical Field

[0001] The present invention relates to a point-level supervised video segment positioning method, system and application based on correspondence learning, belonging to the technical field of target video segment positioning (VML). Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] The task of locating the target video clip is to return the corresponding clip of a given video based on the user's description information. The process mainly includes extracting video features, extracting text features, processing features, locating video clips, and other steps. This task has strong application value in real-world scenarios, such as finding specific clips in surveillance videos to help the police quickly solve cases; helping to quickly review online videos to prevent illegal video uploads from causing adverse effects; helping users quickly find clips of interest in long videos, etc. In view of this, many researchers are committed to studying how to effectively locate the target video clip. A key task to efficiently solve this task is to obtain a video clip localization model with excellent performance through efficient training. Specifically, the problem is mainly divided into two aspects: 1. How to process the data set 2. How to use the data set for efficient training so that the model can learn more knowledge, thereby obtaining a video clip localization model with good performance and generalization ability. In order to solve these problems, many methods have been proposed, which can be mainly divided into three categories: full supervision, weak supervision, and point supervision. Full supervision is to accurately calibrate the real boundary of the video clip corresponding to the query statement. A large number of practices have proved that because the full supervision method can provide rich information, this method can often obtain a model with relatively excellent performance. However, the disadvantage of full supervision is that it is necessary to manually calibrate the real boundaries corresponding to the query statements one by one, which is too expensive. Therefore, the weak supervision method is proposed. Specifically, weak supervision does not perform any annotation on the video, thus greatly reducing the cost of obtaining the data set. But at the same time, due to the sparse information provided by the data set, the performance of the model trained by weak supervision is often far behind that of the model trained by full supervision. In view of the shortcomings of the first two methods, the point supervision method is proposed. Specifically, the point supervision method will annotate one frame in the real boundary corresponding to the query statement. This is low-cost because we do not need to give the real boundary consisting of accurate start and end marks, but only need to annotate any frame in the corresponding real boundary. At the same time, because of the additional annotation information, the point supervision model often shows better performance than the weak supervision model. As mentioned above, point supervision achieves a good balance between model performance and annotation cost. Compared with fully supervised annotation, it is more practical and flexible, and costs only 1 / 6 of the full supervision method; compared with weak supervision, it can achieve better performance at a slightly higher cost. Based on the above advantages, point supervision has gradually become a hot field in video clip localization, and more and more researchers are committed to exploring efficient point supervision solutions.

[0004] Previous studies to enhance the understanding of video and text can be divided into two main approaches: granular modeling and scale modeling. Granular modeling involves depicting the relationship between various features (e.g., frame-word, video clip-sentence) through explicit boundaries present in video-text information, facilitating the distinction of data at different entity granularities. In contrast, scale modeling comes from the complex semantic logic and inherent hierarchical structure within the text and visual domains. This approach recognizes the implicit semantic scale in the interaction of video and language, which exists in an alternative modal space. However, existing point-supervised target video clip localization models still have certain limitations:

[0005] 1) Fine-grained mismatch. Because it is a single-frame annotation, there may be words in the query that are obviously not corresponding to the frame we annotated. This is because the query describes a process, while the annotated frame is only a moment in this process, so it is very likely that the moment only contains part of the information in the query. This problem often prevents us from determining key frames.

[0006] 2) Coarse-grained matching obstacles. The incompleteness of interval annotation information hinders the correspondence between fragments and query sentences at the coarse-grained level. Undoubtedly, this challenge poses a significant obstacle to effective language and vision alignment.

[0007] 3) Implicit scale awareness. Query sentences of the same video may vary in semantic scale. This problem is particularly evident in point-supervised methods. Since only single-frame annotations are used, resulting in a lack of effective context modeling constraints, these methods find it difficult to capture temporal relationships in videos. Summary of the invention

[0008] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a point supervision method based on low annotation cost, a target video clip localization method and system trained by explicit granularity and implicit scale correspondence learning, thereby improving the performance of the video clip localization model under point supervision.

[0009] The technical solution of the present invention is as follows:

[0010] A first aspect of the present invention provides a point-level supervised video segment localization method based on correspondence learning, comprising:

[0011] (1) The untrimmed video and query text are firstly extracted with the pre-trained model, and then the features are obtained through the fully connected layer to obtain enhanced visual representation and enhanced text representation;

[0012] (2) The enhanced text representation and enhanced visual representation obtained by the feature extraction module are further processed by the cross-modal multi-head attention mechanism, and then the cross-modal text representation, cross-modal visual representation and the cross-modal attention score obtained by the last layer of the cross-modal multi-head attention mechanism are input into the granularity alignment module for processing to obtain multi-granularity text representation and multi-granularity visual representation;

[0013] Finally, the multi-granular text representation is max-pooled to obtain a representation that integrates the semantic information of the entire sentence.

[0014] (3) fragment localization, including training and inference stages;

[0015] In the training phase, Gaussian distribution is used in combination with key frames to measure different video clips;

[0016] In the inference phase, the best matching key point between the video and the query is first identified and then expanded from that point to obtain the predicted video segment that is most similar to the query;

[0017] (4) Optimizing the model using a model loss consisting of global alignment loss, frame-fragment correspondence loss, and fragment-frame correspondence loss;

[0018] (5) Import the video and text to be located into the optimized model to obtain the located video clip.

[0019] Preferably, step (1) comprises the following steps: Feature extraction: Before multimodal interaction, we extract visual and text features, and then embed the learnable position into the extracted features, including:

[0020] Step 101: Obtain visual representation

[0021] Step 1011, for the untrimmed video V, we use the pre-trained CNN model to extract visual features;

[0022] Further preferably, the pre-trained CNN model is C3D or I3D, wherein the structure of the C3D model includes 8 convolutional layers, 5 pooling layers and 2 fully connected layers, and finally a softmax output layer, the size of each convolution kernel is 3x3x3, the step size is 1, and the window size of all pooling layers is 2x2x2, and the step size is also 2; the structure of the I3D model includes a 3D convolutional layer with a kernel size of 7x7x7 and a step size of 2, followed by a 3D maximum pooling layer with a kernel size of 3x3x3 and a step size of 2, followed by two convolutional layers with a step size of 1, and the convolution kernel sizes are 1x1x1 and 3x3x3 respectively, followed by a maximum pooling layer with a kernel size of 3x3x3 and a step size of 2, and then, multiple 3D Inception modules are stacked, which contain convolution kernels of different sizes, including 1x1x1, 3x3x3 and 5x5x5, and finally, through a global average pooling layer, output to the softmax layer.

[0023] Step 1012: Then, by using a trainable fully connected (FC) layer, the extracted features are dimensionally mapped to obtain the corresponding visual representation, which is symbolically represented as V = v1, v2, ..., v T , where T represents the number of frames sampled at intervals in the video, v i In order to perceive the time of the video, we added a learnable position embedding (PE) to the model, that is, for each time step of the visual feature, a corresponding learnable position vector is added to obtain an enhanced visual representation Where d is the dimension of the visual representation;

[0024] Step 102: Get text representation

[0025] Step 1021, for a given query Q, we use the pre-trained GloVe model to extract text features; wherein the structure of the GloVe model includes constructing a word-word co-occurrence matrix from a large-scale corpus and defining a weighted loss function to minimize the difference in word vectors; the weighted loss function is expressed as Among them, X ij is the number of co-occurrences of word i and word j, w i and are the word vectors of the word and context, respectively, and b i and is the bias term, α is the average weighting factor; f(X ij ) is a weighting function used to control the sensitivity of the loss to different word frequencies; the model learns the low-dimensional representation of words by minimizing the difference between the inner product of the word vector and the logarithm of the number of co-occurrences;

[0026] Step 1022: Use a bidirectional gated recurrent unit Bi-GRU to obtain the potential sequence order of the original sentence and obtain a new text representation Q = {q1, q2, ..., q L}, where L represents the number of words in the query; similar to the visual representation, we obtain an enhanced text representation by adding the corresponding learnable position vector to the text representation

[0027] The structure of the Bi-GRU model includes two bidirectional GRU (Gated Recurrent Unit) layers, each containing 128 hidden units, followed by a fully connected layer, and finally classified through a softmax output layer.

[0028] Preferably, in step (2), a granularity correspondence alignment module is used in subsequent steps. The present invention introduces a new granularity correspondence alignment module under the framework of point-supervised video clip positioning, which aims to adaptively adjust the granularity relationship between different modalities. By exploring the semantic relationship between cross-modalities, the module enriches the original modal representation and supplements the existing semantic alignment with information of different granularities, ensuring the symmetry of inter-modal interaction. Unlike existing methods that unconsciously interact with the smallest granularity units (frames and words) in cross-modal representations, the present invention mainly uses the attention perception matrix to capture potential cross-modal semantic granularity information. Specifically, it includes:

[0029] I. The present invention uses a matrix composed of cross-modal attention scores as a granularity perception matrix, as shown below:

[0030]

[0031] in is the cross-modal attention score matrix, T and L are the number of frames and words respectively, and s ij is the attention score between the i-th frame and the j-th word; the score in each column represents the degree of attention of the word to the video frame;

[0032] II. Use the maximum function to aggregate the information in each row to obtain the potential prior distribution of query terms relative to video frames. The formula is:

[0033]

[0034] III. Calculate the cosine similarity between the query sentence and the original video frame to obtain the prior distribution of the complete query sentence relative to the video frame.

[0035]

[0036] where cos(·) is the cosine similarity function, is the enhanced visual representation in step 1012, The text representation enhanced in step 1022;

[0037] IV. Using the best granularity information from local self-attention perception and the pooled global granularity information, these information are averaged and used as adaptive visual granularity perception features. The process is expressed as: is the potential prior distribution of the complete query sentence relative to the video frame, so that the visual granularity perception vector can be obtained This vector can be used to enhance the original visual representation;

[0038] V. On this basis, the obtained features are used to reconstruct the cross-modal fusion with text features.

[0039]

[0040] is the enhanced visual representation in step 1012, For remodeled visual features;

[0041] After re-aggregation, the present invention realizes the combination of fine-grained features based on self-attention and multi-scale visual perception features. This enhances fine-grained entity consistency and focuses on overall perception, thereby improving representation.

[0042] VI. Apply the obtained features to cross-modal semantic interaction. After the above steps, we use the previous semantic knowledge to optimize the visual word correlation. Finally, we use the weighted average method to control the original modality feature information to prevent noise interference caused by excessive introduction of cross-modal information. The details are as follows:

[0043]

[0044] Where α is the average weighting factor;

[0045] For text mode, the present invention will enhance the text representation Input to the granularity correspondence alignment module in step (2) to obtain a multi-granularity representation of the text modality Therefore, this step obtains the final multi-granularity representation through multi-modal interaction. and

[0046] Further preferably, step (2), processing the enhanced text representation and the enhanced visual representation specifically includes capturing the semantics within the model, including:

[0047] Step 201, the present invention adopts the self-attention mechanism through the self-attention layer and Generates attention-weighted features and is the weighted visual feature after the self-attention layer, is the weighted text feature after the self-attention layer, is the enhanced visual representation in step 1012, The text representation enhanced in step 1022;

[0048] Step 202: Similarly, the present invention uses a cross-modal multi-head attention mechanism to obtain a cross-modal representation, i.e. and The expression of Attn(·) is as follows:

[0049]

[0050]

[0051] Step 203: In order to serve the subsequent modules, the present invention extracts the cross-modal attention scores of the last layer as cross-modal attention scores, which are: A q→v and A v→q , A q→v The granularity perception matrix obtained when it is video mode is as shown in formula (1), A v→q The granularity perception matrix obtained when the text mode is used will be further described in the subsequent steps. The present invention inputs the features and attention scores into the granularity corresponding alignment module recorded in I-VI to obtain a multi-granularity representation and

[0052]

[0053]

[0054] Where GCA(·) is the granularity correspondence alignment module. So far, we have obtained multi-granularity representations through multimodal interaction. and GCA represents the entire steps I-VI, so this multi-granularity representation is finally calculated by step VI. This representation is a composite representation of multi-layer granularity information. Here, it is the output of GCA, which helps cross-modal semantic alignment. Formula (5) is the last step calculation formula of the GCA process when describing the video modality, and formula (8) is the composite output representation. The two are not contradictory.

[0055] Step 204: In order to integrate the semantic information of the entire sentence, we use the maximum pooling function F(·) to obtain the text representation as follows:

[0056]

[0057] Preferably, in step (3), the fragments are positioned, and the purpose of the fragment positioning is to obtain a multi-granular visual representation. and text representation To locate the start and end timestamps; in the training phase, unlike the fully supervised scenario, a random point in the target video segment can be obtained, but this point cannot be obtained during inference. Due to the differences between these two phases, the present invention will explain these two phases separately, specifically including:

[0058] Training phase: During the training process, the present invention uses a sliding window to slice And generate candidate segments; Gaussian distribution is used to measure the cosine similarity of the temporal features and semantic features of the segments; the temporal features are defined as:

[0059]

[0060] Where i represents the i-th frame, τ m is the index of the supervision frame, σ is a hyperparameter, and L v is the length of the video;

[0061] Semantic similarity score S i represent and cosine similarity between; these temporal and semantic analyses are integrated and applied to the scale correspondence learning strategy of the present invention.

[0062] Reasoning phase: Reasoning involves two main steps, including identifying the best matching key point between the video and the query and expanding from this point to obtain the predicted video segment that is most similar to the query. Specifically, first, by calculating the cosine similarity between the feature vector of the text query and the feature vector of each frame in the video, the frame with the highest similarity is identified as the best matching key point. Then, with this key point as the center, multiple video segments are constructed around it using a sliding window, and the frame features of each segment are aggregated to obtain the overall feature vector of the segment. Next, the cosine similarity between the feature vectors of these segments and the text query is calculated again, and the segments are sorted according to the similarity score. Finally, the segment with the highest score is selected as the predicted video segment that is most similar to the query.

[0063] Preferably, in step (4), scale correspondence learning, in order to improve the overall model (except the feature extraction model) in the understanding ability of the fragments of different scales, the present invention uses the potential frame-fragment correspondence semantic information to construct the learning goal of the overall model, because the single-frame supervision information alone is not enough to obtain effective fragment semantics. Specifically, the optimization goal includes three parts. The first part models the semantic and temporal information of the fused features from a global perspective; the second part uses point annotations to compare the differences between potential positive and negative samples in different intervals to effectively capture information about the changes in actions within the interval; the third part aims to utilize the single-point information of the global fragment that is most similar to the query, thereby enriching the prior knowledge and enhancing the guidance of cross-modal semantics;

[0064] Step 401, global alignment loss: The present invention uses cross entropy loss to make the obtained similarity score representation of the query statement and the video frame close to the information provided by the supervisory frame in terms of temporal distance and semantic distance. The formula is as follows:

[0065]

[0066] Among them G i is the Gaussian distribution weight of the i-th frame, S i is the semantic similarity score of the i-th frame, L g is the global alignment loss;

[0067] Step 402, frame-segment correspondence loss; due to the lack of boundary annotations, the information of a single point is not enough to represent the interval matching the query. The present invention uses a sliding window method to construct video segments containing point annotations as potential positive examples; the present invention constructs binary labels to promote interval perception: segments containing point annotations are marked as 1, which is a positive sample, and other segments are marked as 0, which is a negative sample, as shown below,

[0068]

[0069] Among them, M i is the i-th segment of the sliding window, y i are positive and negative sample labels; due to the potential differences in the temporal information of these fragments, the present invention introduces Gaussian distribution weights to further construct soft labels It can be expressed as:

[0070]

[0071] in G(s) and G(e) are M iGaussian distribution weights at the beginning and end; Since there will inevitably be multiple positive sample moments belonging to the query, it is inappropriate to regard similarity score learning as a 1-in-N classification problem with cross entropy loss. To this end, the present invention uses Kullback-Leibler divergence to construct a frame-segment correspondence loss As shown below,

[0072]

[0073] in It is segment M i and query features The cosine similarity of , in the formula, represents the correspondence loss from frame to moment designed by the present invention, and N represents the number of fragments obtained by sliding the window;

[0074] Step 403: Segment-frame correspondence loss

[0075] Step 4031, at the beginning of the localization process, we will focus on identifying the points that are most closely aligned with the query, so that the model can recognize and emphasize subtle point differences. The present invention separates basic information from positive sample segments and focuses on feature variance. Specifically, the present invention extracts detailed point data that reflects the semantic labels across segments, and uses this data to enrich point annotations with inherent differences;

[0076] This process can be expressed as follows,

[0077]

[0078] where s and e are fragments of M i The start and end index of S j The video frame with index j and the query feature cosine similarity score of the modalities;

[0079] Step 4032: For label construction, the present invention uses hard sample labels and soft labels with Gaussian weights, i.e., formula (12) and formula (13); then the present invention uses KL divergence to analyze the difference between positive samples and negative samples, and uses these insights to refine the alignment process, as shown below:

[0080]

[0081] represents the correspondence loss from moment to frame designed by the present invention;

[0082] Step 404: By combining the global loss Lg and the corresponding loss L c , (the corresponding losses include frame-fragment corresponding losses and fragment-frame corresponding losses), and the final loss of model optimization is obtained as follows:

[0083]

[0084] β and γ are used to balance the attention between clips and frames. Both β and γ are set to 0.5 to avoid the model over-emphasizing sample information.

[0085] By obtaining the learning target of the final loss, first, during the training process, the overall model in the above steps (1) and (2) is optimized. Secondly, in the inference stage, as described in step (3), the cosine similarity between the feature vector of the text query and the feature vector of each frame in the video is calculated, and the frame with the highest similarity is identified as the key point of the best match. Then, with the key point as the center, multiple video clips are constructed around it using a sliding window, and the frame features of each clip are aggregated to obtain the overall feature vector of the clip. Next, the cosine similarity between the feature vectors of these clips and the text query is calculated again, and the clips are sorted according to the similarity score. Finally, the clip with the highest score is selected as the predicted video clip most similar to the query.

[0086] A second aspect of the present invention provides a point-level supervised video segment positioning system based on correspondence learning, comprising:

[0087] The feature extraction module is configured to: first extract features from the untrimmed video and the query text through a pre-trained model, and then obtain enhanced visual representation and enhanced text representation through a fully connected layer;

[0088] The granularity correspondence alignment module is configured to: input the cross-modal text representation and the cross-modal visual representation and the corresponding cross-modal attention score of the last layer to obtain multi-granularity text representation and multi-granularity visual representation;

[0089] The semantic capture module is configured as follows: first, the enhanced text representation and enhanced visual representation obtained by the feature extraction module are processed, and then the processed cross-modal text representation, cross-modal visual representation and the corresponding cross-modal attention score of the last layer are input into the granularity correspondence alignment module to obtain multi-granularity text representation and multi-granularity visual representation; finally, the multi-granularity text representation is subjected to maximum pooling to obtain a representation that integrates the semantic information of the entire sentence;

[0090] The segment localization module, in order to adapt to the two different stages of training and inference, is configured as follows: in the training stage, Gaussian distribution combined with key frames is used to measure different video segments; in the inference stage, the best matching key point between the video and the query is first identified and then expanded from this point to obtain the predicted video segment that is most similar to the query;

[0091] The scale correspondence learning module is configured to optimize the model using a model loss consisting of three parts: global alignment loss, frame-fragment correspondence loss, and fragment-frame correspondence loss.

[0092] The third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in a method for point-level supervised video segment localization based on correspondence learning as described in the first aspect of the present invention.

[0093] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in a method for point-level supervised video segment localization based on correspondence learning as described in the first aspect of the present invention are implemented.

[0094] The beneficial effects of the present invention are:

[0095] (1) The present invention improves the capability of the video segment localization model under point supervision by learning the correspondence between explicit granularity and implicit scale.

[0096] (2) The present invention introduces an innovative granular communication alignment module. Specifically, the module improves the explicit correspondence between different granularities between different modalities, thereby solving the problem of fine-grained mismatch and coarse-grained matching obstruction, allowing the model to perform better.

[0097] (3) Under the framework of point supervision, the present invention develops a scale correspondence learning strategy, which can capture the implicit semantic scale in correspondence learning, thus solving the existing implicit scale perception problem.

[0098] (4) The present invention provides an efficient training method that can obtain a high-performance video clip positioning model at a low cost, achieving a good balance between cost and performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Figure 1 The figure is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0100] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings, but is not limited thereto.

[0101] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0102] It should be noted that the embodiments of the present application involve data related to audio and images. When the above embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0103] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0104] Embodiment 1:

[0105] like Figure 1 As shown, this embodiment discloses a point-level supervised video segment positioning method based on correspondence learning, including:

[0106] (1) The untrimmed video and query text are firstly extracted with a pre-trained model, and then passed through a fully connected layer to obtain enhanced visual representation and enhanced text representation.

[0107] Step (1) includes the following steps: Feature extraction; before multimodal interaction, we extract visual and text features, and then embed the learnable positions into the extracted features, including:

[0108] Step 101: Obtain visual representation

[0109] Step 1011, for the untrimmed video V, we use the pre-trained CNN model to extract visual features;

[0110] The pre-trained CNN model is C3D or I3D, where the structure of the C3D model includes 8 convolutional layers, 5 pooling layers and 2 fully connected layers, and finally a softmax output layer. The size of each convolution kernel is 3x3x3, the stride is 1, and the window size of all pooling layers is 2x2x2, and the stride is also 2; the structure of the I3D model includes a 3D convolutional layer with a kernel size of 7x7x7 and a stride of 2, followed by a 3D maximum pooling layer with a kernel size of 3x3x3 and a stride of 2, followed by two convolutional layers with a stride of 1, and the convolution kernel sizes are 1x1x1 and 3x3x3 respectively, followed by a maximum pooling layer with a kernel size of 3x3x3 and a stride of 2. Next, multiple 3D Inception modules are stacked, which contain convolution kernels of different sizes, including 1x1x1, 3x3x3 and 5x5x5. Finally, through a global average pooling layer, the output is sent to the softmax layer.

[0111] Step 1012: Then, by using a trainable fully connected (FC) layer, the extracted features are dimensionally mapped to obtain the corresponding visual representation, which is symbolically represented as V = v1, v2, ..., v T , where T represents the number of frames sampled at intervals in the video, v i In order to perceive the time of the video, we added a learnable position embedding (PE) to the model, that is, for each time step of the visual feature, a corresponding learnable position vector is added to obtain an enhanced visual representation Where d is the dimension of the visual representation;

[0112] Step 102: Get text representation

[0113] Step 1021: For a given query Q, we use the pre-trained Glove model to extract text features; e The structure of the model includes constructing a word-word co-occurrence matrix from a large-scale corpus and defining a weighted loss function to minimize the difference in word vectors; the weighted loss function is expressed as Among them, X ij is the number of co-occurrences of word i and word j, w i and are the word vectors of the word and context, respectively, and b i and is the bias term, α is the average weighting factor; f(X ij ) is a weighting function used to control the sensitivity of the loss to different word frequencies; the model learns the low-dimensional representation of words by minimizing the difference between the inner product of the word vector and the logarithm of the number of co-occurrences;

[0114] Step 1022: Use a bidirectional gated recurrent unit Bi-GRU to obtain the potential sequence order of the original sentence and obtain a new text representation Q = {q1, q2, ..., q L}, where L represents the number of words in the query; similar to the visual representation, we obtain an enhanced text representation by adding the corresponding learnable position vector to the text representation

[0115] The structure of the Bi-GRU model includes two bidirectional GRU (Gated Recurrent Unit) layers, each containing 128 hidden units, followed by a fully connected layer, and finally classified through a softmax output layer.

[0116] (2) Granularity correspondence alignment module, which will be used in subsequent steps. The present invention introduces a new granularity correspondence alignment module under the framework of point-supervised video clip localization, which aims to adaptively adjust the granularity relationship between different modalities. By exploring the semantic relationship between cross-modalities, this module enriches the original modal representation and supplements the existing semantic alignment with information of different granularities, ensuring the symmetry of inter-modal interaction. Unlike existing methods that unconsciously interact with the smallest granularity units (frames and words) in cross-modal representation, the present invention mainly uses the attention perception matrix to capture potential cross-modal semantic granularity information. Specifically, it includes:

[0117] I. The present invention uses a matrix composed of cross-modal attention scores as a granularity perception matrix, as shown below:

[0118]

[0119] in is the cross-modal attention score matrix, T and L are the number of frames and words respectively, and s ij is the attention score between the i-th frame and the j-th word; the score in each column represents the degree of attention of the word to the video frame;

[0120] II. Use the maximum function to aggregate the information in each row to obtain the potential prior distribution of query terms relative to video frames. The formula is:

[0121]

[0122] III. Calculate the cosine similarity between the query sentence and the original video frame to obtain the prior distribution of the complete query sentence relative to the video frame.

[0123]

[0124] where cos(·) is the cosine similarity function, is the enhanced visual representation in step 1012, The text representation enhanced in step 1022;

[0125] IV. Using the best granularity information from local self-attention perception and the pooled global granularity information, these information are averaged and used as adaptive visual granularity perception features. The process is expressed as: is the potential prior distribution of the complete query sentence relative to the video frame, so that the visual granularity perception vector can be obtained This vector can be used to enhance the original visual representation;

[0126] V. On this basis, the obtained features are used to reconstruct the cross-modal fusion with text features.

[0127]

[0128] is the enhanced visual representation in step 1012, For remodeled visual features;

[0129] After re-aggregation, the present invention realizes the combination of fine-grained features based on self-attention and multi-scale visual perception features. This enhances fine-grained entity consistency and focuses on overall perception, thereby improving representation.

[0130] VI. Apply the obtained features to cross-modal semantic interaction. After the above steps, we use the previous semantic knowledge to optimize the visual word correlation. Finally, we use the weighted average method to control the original modality feature information to prevent noise interference caused by excessive introduction of cross-modal information. The details are as follows:

[0131]

[0132] Where α is the average weighting factor;

[0133] For text mode, the present invention will enhance the text representation Input to the granularity correspondence alignment module in step (2) to obtain a multi-granularity representation of the text modality Therefore, this step obtains the final multi-granularity representation through multi-modal interaction. and

[0134] The enhanced text representation and enhanced visual representation obtained by the feature extraction module are further processed by the cross-modal multi-head attention mechanism, and then the processed cross-modal text representation, cross-modal visual representation and the cross-modal attention score obtained by the last layer of the cross-modal multi-head attention mechanism are input into the granularity correspondence alignment module for processing to obtain multi-granular text representation and multi-granular visual representation; finally, the multi-granular text representation is subjected to maximum pooling to obtain a representation that integrates the semantic information of the entire sentence; specifically, the steps include:

[0135] Step 201, the present invention adopts the self-attention mechanism through the self-attention layer and Generates attention-weighted features and is the weighted visual feature after the self-attention layer, is the weighted text feature after the self-attention layer, is the enhanced visual representation in step 1012, The text representation enhanced in step 1022;

[0136] Step 202: Similarly, the present invention uses a cross-modal multi-head attention mechanism to obtain a cross-modal representation, i.e. and The expression of Attn(·) is as follows:

[0137]

[0138]

[0139] Step 203: In order to serve the subsequent modules, the present invention extracts the cross-modal attention scores of the last layer as cross-modal attention scores, which are: A q→v and A v→q , A q→v The granularity perception matrix obtained when it is video mode is as shown in formula (1), A v→q The granularity perception matrix obtained when the text mode is used will be further described in the subsequent steps. The present invention inputs the features and attention scores into the granularity corresponding alignment module recorded in I-VI to obtain a multi-granularity representation and

[0140]

[0141]

[0142] Where GCA(·) is the granularity correspondence alignment module. So far, we have obtained multi-granularity representations through multimodal interaction. and GCA represents the entire steps I-VI, so this multi-granularity representation is finally calculated by step VI. This representation is a composite representation of multi-layer granularity information. Here, it is the output of GCA, which helps cross-modal semantic alignment. Formula (5) is the last step calculation formula of the GCA process when describing the video modality, and formula (8) is the composite output representation. The two are not contradictory.

[0143] Step 204: In order to integrate the semantic information of the entire sentence, we use the maximum pooling function F(·) to obtain the text representation as follows:

[0144]

[0145] (3) Fragment localization, including training and reasoning stages; the purpose of fragment localization is to obtain multi-granular visual representations. and text representation To locate the start and end timestamps; in the training phase, unlike the fully supervised scenario, a random point in the target video segment can be obtained, but this point cannot be obtained during inference. Due to the differences between these two phases, the present invention will explain these two phases separately, specifically including:

[0146] During the training phase, Gaussian distribution is combined with key frames to measure different video clips; sliding windows are used to slice And generate candidate segments; Gaussian distribution is used to measure the cosine similarity of the temporal features and semantic features of the segments; the temporal features are defined as:

[0147]

[0148] Where i represents the i-th frame, τ m is the index of the supervision frame, σ is a hyperparameter, and L v is the length of the video;

[0149] Semantic similarity score S i represent and cosine similarity between; these temporal and semantic analyses are integrated and applied to the scale correspondence learning strategy of the present invention.

[0150] In the inference stage, the best matching key point between the video and the query is first identified and then expanded from this point to obtain the predicted video segment that is most similar to the query; inference involves two main steps, including identifying the best matching key point between the video and the query and expanding from this point to obtain the predicted video segment that is most similar to the query; specifically, first, by calculating the cosine similarity between the feature vector of the text query and the feature vector of each frame in the video, the frame with the highest similarity is identified as the best matching key point; then, with this key point as the center, multiple video segments are constructed around it using a sliding window, and the frame features of each segment are aggregated to obtain the overall feature vector of the segment; then, the cosine similarity of the feature vectors of these segments and the text query is calculated again, the segments are sorted according to the similarity score, and finally the segment with the highest score is selected as the predicted video segment that is most similar to the query.

[0151] (4) Optimizing the model using a model loss consisting of global alignment loss, frame-fragment correspondence loss, and fragment-frame correspondence loss;

[0152] Scale correspondence learning, in order to improve the overall model (except the feature extraction model) in the understanding of fragments of different scales, the present invention uses the potential frame-fragment correspondence semantic information to construct the learning goal of the overall model, because the single-frame supervision information alone is not enough to obtain effective fragment semantics. Specifically, the optimization goal includes three parts. The first part models the semantic and temporal information of the fused features from a global perspective; the second part uses point annotations to compare the differences between potential positive and negative samples in different intervals to effectively capture information about the changes in actions within the interval; the third part aims to utilize the single-point information of the global fragment that is most similar to the query, thereby enriching the prior knowledge and enhancing the guidance of cross-modal semantics;

[0153] Step 401, global alignment loss: The present invention uses cross entropy loss to make the obtained similarity score representation of the query statement and the video frame close to the information provided by the supervisory frame in terms of temporal distance and semantic distance. The formula is as follows:

[0154]

[0155] Among them G i is the Gaussian distribution weight of the i-th frame, S i is the semantic similarity score of the i-th frame, L g is the global alignment loss;

[0156] Step 402, frame-segment correspondence loss; due to the lack of boundary annotations, the information of a single point is not enough to represent the interval matching the query. The present invention uses a sliding window method to construct video segments containing point annotations as potential positive examples; the present invention constructs binary labels to promote interval perception: segments containing point annotations are marked as 1, which is a positive sample, and other segments are marked as 0, which is a negative sample, as shown below,

[0157]

[0158] Among them, M i is the i-th segment of the sliding window, y i are positive and negative sample labels; due to the potential differences in the temporal information of these fragments, the present invention introduces Gaussian distribution weights to further construct soft labels It can be expressed as:

[0159]

[0160] in G(s) and G(e) are M i Gaussian distribution weights at the beginning and end; Since there will inevitably be multiple positive sample moments belonging to the query, it is inappropriate to regard similarity score learning as a 1-in-N classification problem with cross entropy loss. To this end, the present invention uses Kullback-Leibler divergence to construct a frame-segment correspondence loss As shown below,

[0161]

[0162] in It is segment M i and query features The cosine similarity of , in the formula, represents the correspondence loss from frame to moment designed by the present invention, and N represents the number of fragments obtained by sliding the window;

[0163] Step 403: Segment-frame correspondence loss

[0164] Step 4031, at the beginning of the localization process, we will focus on identifying the points that are most closely aligned with the query, so that the model can recognize and emphasize subtle point differences. The present invention separates basic information from positive sample segments and focuses on feature variance. Specifically, the present invention extracts detailed point data that reflects the semantic labels across segments, and uses this data to enrich point annotations with inherent differences;

[0165] This process can be expressed as follows,

[0166]

[0167] where s and e are fragments of M i The start and end index of S j The video frame with index j and the query feature cosine similarity score of the modalities;

[0168] Step 4032: For label construction, the present invention uses hard sample labels and soft labels with Gaussian weights, i.e., formula (12) and formula (13); then the present invention uses KL divergence to analyze the difference between positive samples and negative samples, and uses these insights to refine the alignment process, as shown below:

[0169]

[0170] represents the correspondence loss from moment to frame designed by the present invention;

[0171] Step 404: By combining the global loss L g and the corresponding loss L c , (the corresponding losses include frame-fragment corresponding losses and fragment-frame corresponding losses), and the final loss of model optimization is obtained as follows:

[0172]

[0173] β and γ are used to balance the attention between clips and frames, and both β and γ are set to 0.5 to prevent the model from over-emphasizing sample information.

[0174] (5) Import the video and text to be located into the optimized model to obtain the located video clip.

[0175] Embodiment 2:

[0176] This embodiment discloses a point-level supervised video segment positioning system based on correspondence learning, including:

[0177] The feature extraction module is configured to first extract features from the untrimmed video and query text through a pre-trained model, and then obtain enhanced visual representation and enhanced text representation through a fully connected layer.

[0178] The granularity correspondence alignment module is configured to: input the cross-modal text representation and the cross-modal visual representation and the corresponding cross-modal attention score of the last layer to obtain multi-granularity text representation and multi-granularity visual representation.

[0179] The semantic capture module is configured to: process the enhanced text representation and enhanced visual representation obtained by the feature extraction module, and then input the processed cross-modal text representation, cross-modal visual representation and the corresponding cross-modal attention score of the last layer into the granularity alignment module to obtain multi-granular text representation and multi-granular visual representation. Finally, the multi-granular text representation is max-pooled to obtain a representation that integrates the semantic information of the entire sentence.

[0180] In order to adapt to the two different stages of training and inference, the segment localization module is configured as follows: 1) In the training stage, Gaussian distribution is used in combination with key frames to measure different video segments. 2) In the inference stage, the best matching key point between the video and the query is first identified and then expanded from this point to obtain the predicted video segment that is most similar to the query.

[0181] The scale correspondence learning module is configured to optimize the model using a model loss consisting of three parts: global alignment loss, frame-fragment correspondence loss, and fragment-frame correspondence loss.

[0182] Embodiment 3:

[0183] The purpose of this embodiment is to provide a computer-readable storage medium.

[0184] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in a method for point-level supervised video segment localization based on correspondence learning as described in Example 1 of the present disclosure.

[0185] Embodiment 4:

[0186] The purpose of this embodiment is to provide an electronic device.

[0187] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in a method for point-level supervised video segment localization based on correspondence learning as described in Example 1 of the present disclosure are implemented.

[0188] The steps involved in the apparatuses of the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method of the present invention.

[0189] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0190] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A point-level supervised video segment localization method based on correspondence learning, characterized in that: include: (1) The untrimmed video and query text are firstly extracted with the pre-trained model, and then the features are obtained through the fully connected layer to obtain enhanced visual representation and enhanced text representation; (2) The enhanced text representation and enhanced visual representation obtained by the feature extraction module are further processed by the cross-modal multi-head attention mechanism, and then the cross-modal text representation, cross-modal visual representation and the cross-modal attention score obtained by the last layer of the cross-modal multi-head attention mechanism are input into the granularity alignment module for processing to obtain multi-granularity text representation and multi-granularity visual representation; Finally, the multi-granular text representation is max-pooled to obtain a representation that integrates the semantic information of the entire sentence. (3) fragment localization, including training and inference stages; In the training phase, Gaussian distribution is used in combination with key frames to measure different video clips; In the inference phase, the best matching key point between the video and the query is first identified and then expanded from that point to obtain the predicted video segment that is most similar to the query; (4) Optimizing the model using a model loss consisting of three parts: global alignment loss, frame-fragment correspondence loss, and fragment-frame correspondence loss; (5) Import the video and text to be located into the optimized model to obtain the located video clip.

2. The method for point-level supervised video segment localization based on correspondence learning according to claim 1, characterized in that: Step (1) includes the following steps: feature extraction; before multimodal interaction, extracting visual and textual features, and then embedding the learnable positions into the extracted features, including: Step 101: Obtain visual representation Step 1011, for the untrimmed video V, we use the pre-trained CNN model to extract visual features; Step 1012, then use a trainable fully connected layer to perform dimension mapping on the extracted features to obtain the corresponding visual representation, which is symbolically represented as V = v1, v2, ..., v T , where T represents the number of frames sampled at intervals in the video, v i represents the i-th frame; a learnable position embedding is added to the model, that is, a corresponding learnable position vector is added to the visual feature of each time step, and an enhanced visual representation is obtained Where d is the dimension of the visual representation; Step 102: Get text representation Step 1021, for a given query Q, we use the pre-trained GloVe model to extract text features; wherein the structure of the GloVe model includes constructing a word-word co-occurrence matrix from a large-scale corpus and defining a weighted loss function to minimize the difference in word vectors; the weighted loss function is expressed as Among them, X ij is the number of co-occurrences of word i and word j, w i and are the word vectors of the word and context, respectively, and b i and is the bias term, α is the average weighting factor; f(X ij ) is a weighting function used to control the sensitivity of the loss to different word frequencies; the model learns the low-dimensional representation of words by minimizing the difference between the inner product of the word vector and the logarithm of the number of co-occurrences; Step 1022: Use a bidirectional gated recurrent unit Bi-GRU to obtain the potential sequence order of the original sentence and obtain a new text representation Q = {q1, q2, ..., q L }, where L represents the number of words in the query; the corresponding learnable position vector is added to the text representation to obtain an enhanced text representation The structure of the Bi-GRU model includes two bidirectional GRU layers, each containing 128 hidden units, followed by a fully connected layer, and finally classified through a softmax output layer.

3. The method for point-level supervised video segment localization based on correspondence learning according to claim 2, characterized in that: In step 1011, the pre-trained CNN model is C3D or I3D, wherein the structure of the C3D model includes 8 convolutional layers, 5 pooling layers, and 2 fully connected layers, and finally a softmax output layer, each convolution kernel size is 3x3x3, the step size is 1, and the window size of all pooling layers is 2x2x2, and the step size is also 2; the structure of the I3D model includes a 3D convolutional layer with a kernel size of 7x7x7 and a step size of 2, followed by a 3D maximum pooling layer with a kernel size of 3x3x3 and a step size of 2, and then two convolutional layers with a step size of 1, and the convolution kernel sizes are 1x1x1 and 3x3x3 respectively, followed by a maximum pooling layer with a kernel size of 3x3x3 and a step size of 2, and then multiple 3D Inception modules, these modules contain convolution kernels of different sizes, including 1x1x1, 3x3x3 and 5x5x5. Finally, they pass through a global average pooling layer and output to the softmax layer.

4. The method for point-level supervised video segment localization based on correspondence learning according to claim 1, characterized in that: In step (2), the granularity alignment module includes: I. Use the matrix composed of cross-modal attention scores as the granularity-aware matrix as follows: in is the cross-modal attention score matrix, T and L are the number of frames and words respectively, and s ij is the attention score between the i-th frame and the j-th word; the score in each column represents the degree of attention of the word to the video frame; II. Use the maximum function to aggregate the information in each row to obtain the potential prior distribution of the query word relative to the video frame The formula is: III. Calculate the cosine similarity between the query sentence and the original video frame to obtain the prior distribution of the complete query sentence relative to the video frame. where cos(·) is the cosine similarity function, For enhanced visual representation, For enhanced text representation; IV. Using the best granularity information from local self-attention perception and the aggregated global granularity information, these information are averaged and used as adaptive visual granularity perception features. The process is expressed as: is the potential prior distribution of the complete query sentence relative to the video frame, and the visual granularity perception vector is obtained V. On this basis, the obtained features are used to reconstruct the cross-modal fusion with text features. For enhanced visual representation, For remodeled visual features; VI. Apply the obtained features to cross-modal semantic interaction. After the above steps, use the previous semantic knowledge to optimize the visual word relevance; finally, use the weighted average method to control the original modality feature information, as follows: Where α is the average weighting factor; For text mode, the enhanced text representation Input to the granularity correspondence alignment module in step (2) to obtain a multi-granularity representation of the text modality Therefore, this step obtains the final multi-granularity representation through multi-modal interaction. and 5. The method for point-level supervised video segment localization based on correspondence learning according to claim 4, characterized in that: Step (2), processing the enhanced text representation and the enhanced visual representation specifically includes: Step 201: Use the self-attention layer and Generates attention-weighted features and is the weighted visual feature after the self-attention layer, is the weighted text feature after the self-attention layer, For enhanced visual representation, For enhanced text representation; Step 202: Use a cross-modal multi-head attention mechanism to obtain a cross-modal representation, i.e. and The expression of Attn(·) is as follows: Step 203: extract the cross-modal attention scores of the last layer as cross-modal attention scores, which are: A q →v and A v→q , A q→v is the granularity perception matrix obtained when the video mode is used, A v→q The granularity perception matrix obtained when the text mode is used is input into the granularity corresponding alignment module described in Ⅰ-Ⅵ to obtain the multi-granularity representation. and Among them, GCA(·) is the granularity correspondence alignment module. So far, multi-granularity representations have been obtained through multimodal interaction. and GCA represents the entire steps I-VI. This multi-granularity representation is finally calculated by step VI. This representation is a composite representation of multiple layers of granularity information, which is the output of GCA here. Step 204: In order to integrate the semantic information of the entire sentence, we use the maximum pooling function F(·) to obtain the text representation as follows:

6. The method for point-level supervised video segment localization based on correspondence learning according to claim 1, characterized in that: In step (3), the fragment location is based on the obtained multi-granular visual representation and text representation To locate the start and end timestamps; specifically: Training phase; during training, use sliding windows to slice And generate candidate segments; Gaussian distribution is used to measure the cosine similarity of the temporal features and semantic features of the segments; the temporal features are defined as: Where i represents the i-th frame, τ m is the index of the supervision frame, σ is a hyperparameter, and L v is the length of the video; Semantic similarity score S i represent and The cosine similarity between ; Reasoning phase: Reasoning involves two steps, including identifying the best matching key point between the video and the query and expanding from this point to obtain the predicted video segment that is most similar to the query. Specifically, first, by calculating the cosine similarity between the feature vector of the text query and the feature vector of each frame in the video, the frame with the highest similarity is identified as the best matching key point. Then, with this key point as the center, multiple video segments are constructed around it using a sliding window, and the frame features of each segment are aggregated to obtain the overall feature vector of the segment. Next, the cosine similarity between the feature vectors of these segments and the text query is calculated again, and the segments are sorted according to the similarity score. Finally, the segment with the highest score is selected as the predicted video segment that is most similar to the query.

7. The method for point-level supervised video segment localization based on correspondence learning according to claim 1, characterized in that: In step (4), scale correspondence learning uses the potential frame-segment correspondence semantic information to construct the learning goal of the overall model. Specifically, the optimization goal includes three parts: the first part models the semantic and temporal information of the fused features from a global perspective; the second part uses point annotations to compare the differences between potential positive and negative samples in different intervals; the third part uses the single point information of the global segment that is most similar to the query; Step 401, global alignment loss: cross entropy loss is used to make the obtained similarity score representation of the query statement and the video frame close to the information provided by the supervision frame in terms of temporal distance and semantic distance. The formula is as follows: Among them G i is the Gaussian distribution weight of the i-th frame, S i is the semantic similarity score of the i-th frame, L g is the global alignment loss; Step 402, frame-segment correspondence loss; using the sliding window method, construct video segments containing point annotations as potential positive examples; construct binary labels to promote interval perception: segments containing point annotations are marked as 1, that is, positive samples, and other segments are marked as 0, that is, negative samples, as shown below, Among them, M i is the i-th segment of the sliding window, y i are positive and negative sample labels; Gaussian distribution weights are introduced to further construct soft labels It is expressed as: Where W(·)=G(s)*G(e), G(s) and G(e) are M i Gaussian weights at the start and end; Kullback-Leibler divergence is used to construct the frame-segment correspondence loss As shown below, in It is segment M i and query features The cosine similarity of , in the formula, represents the corresponding loss from frame to fragment designed by the present invention, and N represents the number of fragments obtained by sliding the window; Step 403: Segment-frame correspondence loss Step 4031: extract detailed point data reflecting cross-segment semantic labels, and use these data to enrich point annotations with inherent differences; This process is represented as follows, where s and e are fragments of M i The start and end index of S j The video frame with index j and the query feature cosine similarity score of the modalities; Step 4032, for label construction, use hard sample labels and soft labels with Gaussian weights, i.e., formula (12) and formula (13); then use KL divergence to analyze the difference between positive and negative samples, and use these insights to refine the alignment process as follows: represents the corresponding loss from fragment to frame designed by the present invention; Step 404: By combining the global loss L g and the corresponding loss L c , the corresponding losses include frame-fragment corresponding loss and fragment-frame corresponding loss, and the final loss of model optimization is obtained as follows: β and γ are used to balance the attention between clips and frames, and both β and γ are set to 0.

5.

8. A system for the point-level supervised video segment localization method based on correspondence learning according to claim 1, characterized in that: include: The feature extraction module is configured to: first extract features from the untrimmed video and the query text through a pre-trained model, and then obtain enhanced visual representation and enhanced text representation through a fully connected layer; The granularity correspondence alignment module is configured to: input the cross-modal text representation and the cross-modal visual representation and the corresponding cross-modal attention score of the last layer to obtain multi-granularity text representation and multi-granularity visual representation; The semantic capture module is configured as follows: first, the enhanced text representation and enhanced visual representation obtained by the feature extraction module are processed, and then the processed cross-modal text representation, cross-modal visual representation and the corresponding cross-modal attention score of the last layer are input into the granularity correspondence alignment module to obtain multi-granularity text representation and multi-granularity visual representation; finally, the multi-granularity text representation is subjected to maximum pooling to obtain a representation that integrates the semantic information of the entire sentence; In order to adapt to the two different stages of training and inference, the segment localization module is configured as follows: in the training stage, Gaussian distribution is used in combination with key frames to measure different video segments; The inference phase first identifies the best matching key point between the video and the query and then expands from that point to obtain the predicted video segment that is most similar to the query; The scale correspondence learning module is configured to optimize the model using a model loss consisting of three parts: global alignment loss, frame-fragment correspondence loss, and fragment-frame correspondence loss.

9. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, the steps in the point-level supervised video segment positioning method based on correspondence learning as described in claim 1 are implemented.

10. An electronic device, characterized in that The method comprises a memory, a processor and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the point-level supervised video segment positioning method based on correspondence learning as claimed in claim 1 are implemented.

Images