Video temporal sentence localization method based on four constraints and partial supervision

Abstract

The video temporal sentence localization method based on four constraints and partial supervision comprises the following steps: high-level semantic features are extracted from query text and video by using a deep neural network; a time stamp corresponding to the query event is obtained from the video features by fusing a frame-by-frame algorithm and a candidate frame algorithm, and the video features fused with the query text features are input into an event detector; a four-constraint model is constructed to facilitate learning of a semi-supervised temporal sentence localization task; a partial-full merging framework is used to predict the temporal localization result according to the features, so as to flexibly handle two types of supervision, i.e., partial supervision and full supervision; partial supervision branch optimization and full supervision branch optimization are performed, the partial supervision branch optimization comprises optimization of the four-constraint model and optimization of a localization pseudo-label localization loss constructed according to partial supervision information; and the full supervision branch optimization is performed by using a localization learning framework and relying on supervised learning optimization of the localization pseudo-label.

Description

Technical fields:

[0001] This invention relates to a video temporal sentence localization method based on quadruple constraints and partial supervision. Background technology:

[0002] With the rapid development of communication technology, the Internet and social media, video data is gradually permeating people's daily lives. Unlike traditional information media such as text and pictures, video, in the form of image streams supplemented by auditory information, can more intuitively and vividly show the events that are happening and the situations that are in, allowing people to have a deeper perception.

[0003] The ability to quickly locate interesting and valuable video content is crucial for video management and storage, as well as improving the user experience of searching and watching videos. Therefore, researchers have conducted extensive research on video key action detection, aiming to discover key events and human behaviors in videos, classify them, and pinpoint their temporal location within the video. However, in this application scenario, the types of events and actions that can be categorized into the detection target are limited, failing to cover the vast and diverse content in videos. Detected events are always confined to a specific set of categories. Natural language, on the other hand, can express richer meanings and is not limited by categories.

[0004] To address the aforementioned issues, a method is adopted where, given a video and a natural language sentence describing its content, the temporal location of the target video segment described by the sentence needs to be predicted within the video—this is known as temporal localization of the natural language sentence within the video. Temporal sentence localization typically employs two approaches: fully supervised temporal sentence localization, where each event is labeled with precise temporal boundaries; although significant progress has been made in this area, accurately labeling event boundaries is time-consuming and highly subjective. For events with advanced semantic concepts, it is difficult to rigorously label the same event boundaries for different annotators, requiring expensive annotation costs. The other approach is weakly supervised temporal sentence localization, where during training, only text queries are provided for each untrimmed video segment without any temporal annotations. This makes it difficult to extract useful text-event correspondence features from limited supervisory information, and its localization performance cannot meet practical application requirements. Summary of the Invention:

[0005] This invention provides a video temporal sentence localization method based on quadruple constraints and partial supervision. The method is rationally designed, proposing a novel quadruple constraint mechanism to comprehensively shape the alignment representation of both events and queries. Improvements are made in three aspects: task setting, cross-modal representation learning, and application scenarios. It fully utilizes partial label information, covering intra- and inter-sample relationships, and mining and extracting unimodal and multimodal information. This not only improves intra-cluster compactness and inter-cluster separability but also separates events from the background. Furthermore, a partially-fully joint framework is introduced, connected to an additional fully supervised branch bridge, to obtain the significant advantages of the fully supervised training paradigm. It exhibits better robustness to partial criteria, reduces the heavy annotation costs in fully supervised temporal sentence localization methods, and improves the localization performance in weakly supervised temporal sentence localization methods. It can achieve localization results comparable to fully supervised settings under video segment supervision conditions, solving the problems existing in the prior art.

[0006] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0007] A video temporal sentence localization method based on four constraints and partial supervision includes the following steps:

[0008] S1 uses a deep neural network to extract high-level semantic features from the query text and video, providing discriminative video features for event detection;

[0009] S2, the frame-by-frame algorithm and the candidate box algorithm are combined to obtain the timestamp corresponding to the query event from the video features. The video features that are combined with the query text features are input into the event detector to predict the center timestamp and event duration of the video segment corresponding to the event.

[0010] S3. Construct a quadruple constraint model to facilitate the learning of semi-supervised temporal sentence localization tasks. The quadruple constraint model includes an in-sample event-query text multimodal constraint model, a visual unimodal constraint model, and an inter-sample event-query multimodal constraint model and a visual unimodal constraint model.

[0011] S4. A partial-to-complete merging framework is used to predict the temporal localization results based on the above features, so as to flexibly handle two types of supervision: partial supervision and complete supervision.

[0012] S5, optimize the partially supervised branch and the fully supervised branch. The optimization of the partially supervised branch includes the optimization of the quadruple constraint model and the optimization of the localization loss constructed from the localization pseudo-labels using partially supervised information. The optimization of the fully supervised branch is to use a localization learning framework and rely on localization pseudo-labels for supervised learning optimization.

[0013] Extracting high-level semantic features from query text and video using deep neural networks to provide discriminative video features for event detection includes the following steps:

[0014] S1.1, a pre-trained 3D convolutional network is used to extract features from the video stream. The extracted features have two dimensions: the number of frames in the video sequence and the number of feature channels of the video as a visual modality.

[0015] S1.2, feature extraction is performed on the query text using a global vector based on word representation. The extracted text query features have two dimensions: the number of words in the query text and the number of channels in the text features.

[0016] S1.3, a fully connected layer is used to fine-tune the features of the video and query text modalities to obtain the single-modal features of the video stream and query text modalities;

[0017] S1.4 introduces two cross-modal Transformers to apply a cross-modal attention mechanism to the features of the video stream and the query text. Specifically, for the video stream, text features are used as query features, while video features are used as key features and key-value features to obtain cross-modal fused video features; for the query text, video features are used as query features, while text features are used as key features and key-value features to obtain cross-modal fused text features.

[0018] The frame-by-frame method is used to continuously determine whether each frame of a video segment belongs to the query text feature; the candidate box algorithm is used to predict the intermediate timestamp of the time and the corresponding event duration.

[0019] The predicted center timestamp and event duration are converted into the start and end times of the event, and then used to construct location pseudo-labels under partial supervision. For the predicted start time, two computational methods can be used to generate a differentiable time mask to facilitate the calculation of the event-text query representation. The computational methods include approximation using a Gaussian shape and approximation using an improved shape distribution with a flat mask. Based on the generated differentiable time mask, visual features are separated into foreground and background to obtain foreground and background features for subsequent quadruple constraints.

[0020] For the event-query multimodal constraint model between samples, by extracting frame-by-frame video features from the entire video features through pooling operations, the semantic similarity between event features and query text features is higher than that between the entire video features and query text.

[0021] The partially-fully-joined framework achieves direct prediction of localization results during inference by adding an additional fully-supervised branch to the partially-supervised branch.

[0022] The localization system of the video temporal sentence localization method based on quadruple constraints and partial supervision includes a partially supervised branch component and a fully supervised branch component. The partially supervised branch component is trained and learned under quadruple constraints, and the fully supervised branch component is trained and learned using pseudo-localization labels obtained from partially supervised labels.

[0023] This invention employs the aforementioned structure, utilizing deep neural networks to extract high-level semantic features from query text and video, providing discriminative video features for event detection. By fusing frame-by-frame and candidate box algorithms, the timestamps corresponding to the query events are obtained from the video features. These video features, incorporating query text features, are then input into the event detector to predict the center timestamp and event duration of the corresponding video segment. A four-fold constraint model is constructed to facilitate the learning of semi-supervised temporal sentence localization tasks. This model includes an in-sample event-query text multimodal constraint model, a visual unimodal constraint model, and an inter-sample event-query multimodal constraint model and a visual unimodal constraint model. A partial-to-complete merging framework is used to predict temporal localization results based on the aforementioned features, flexibly handling both partial and complete supervision, offering advantages of practicality, efficiency, simplicity, and speed. Attached image description:

[0024] Figure 1 This is a schematic diagram of the process of the present invention.

[0025] Figure 2 This is a schematic diagram of the positioning system of the present invention. Detailed implementation method:

[0026] To clearly illustrate the technical features of this solution, the invention will be described in detail below through specific implementation methods and in conjunction with the accompanying drawings.

[0027] like Figure 1-2 As shown, the video temporal sentence localization method based on four constraints and partial supervision includes the following steps:

[0028] S1 uses a deep neural network to extract high-level semantic features from the query text and video, providing discriminative video features for event detection;

[0029] S2, the frame-by-frame algorithm and the candidate box algorithm are combined to obtain the timestamp corresponding to the query event from the video features. The video features that are combined with the query text features are input into the event detector to predict the center timestamp and event duration of the video segment corresponding to the event.

[0030] S3. Construct a quadruple constraint model to facilitate the learning of semi-supervised temporal sentence localization tasks. The quadruple constraint model includes an in-sample event-query text multimodal constraint model, a visual unimodal constraint model, and an inter-sample event-query multimodal constraint model and a visual unimodal constraint model.

[0031] S4. A partial-to-complete merging framework is used to predict the temporal localization results based on the above features, so as to flexibly handle two types of supervision: partial supervision and complete supervision.

[0032] S5, optimize the partially supervised branch and the fully supervised branch. The optimization of the partially supervised branch includes the optimization of the quadruple constraint model and the optimization of the localization loss constructed from the localization pseudo-labels using partially supervised information. The optimization of the fully supervised branch is to use a localization learning framework and rely on localization pseudo-labels for supervised learning optimization.

[0033] Extracting high-level semantic features from query text and video using deep neural networks to provide discriminative video features for event detection includes the following steps:

[0034] S1.1, a pre-trained 3D convolutional network is used to extract features from the video stream. The extracted features have two dimensions: the number of frames in the video sequence and the number of feature channels of the video as a visual modality.

[0035] S1.2, feature extraction is performed on the query text using a global vector based on word representation. The extracted text query features have two dimensions: the number of words in the query text and the number of channels in the text features.

[0036] S1.3, a fully connected layer is used to fine-tune the features of the video and query text modalities to obtain the single-modal features of the video stream and query text modalities;

[0037] S1.4 introduces two cross-modal Transformers to apply a cross-modal attention mechanism to the features of the video stream and the query text. Specifically, for the video stream, text features are used as query features, while video features are used as key features and key-value features to obtain cross-modal fused video features; for the query text, video features are used as query features, while text features are used as key features and key-value features to obtain cross-modal fused text features.

[0038] The frame-by-frame method is used to continuously determine whether each frame of a video segment belongs to the query text feature; the candidate box algorithm is used to predict the intermediate timestamp of the time and the corresponding event duration.

[0039] The predicted center timestamp and event duration are converted into the start and end times of the event, and then used to construct location pseudo-labels under partial supervision. For the predicted start time, two computational methods can be used to generate a differentiable time mask to facilitate the calculation of the event-text query representation. The computational methods include approximation using a Gaussian shape and approximation using an improved shape distribution with a flat mask. Based on the generated differentiable time mask, visual features are separated into foreground and background to obtain foreground and background features for subsequent quadruple constraints.

[0040] For the event-query multimodal constraint model between samples, by extracting frame-by-frame video features from the entire video features through pooling operations, the semantic similarity between event features and query text features is higher than that between the entire video features and query text.

[0041] The partially-fully-joined framework achieves direct prediction of localization results during inference by adding an additional fully-supervised branch to the partially-supervised branch.

[0042] The localization system of the video temporal sentence localization method based on quadruple constraints and partial supervision includes a partially supervised branch component and a fully supervised branch component. The partially supervised branch component is trained and learned under quadruple constraints, and the fully supervised branch component is trained and learned using pseudo-localization labels obtained from partially supervised labels.

[0043] The working principle of the video temporal sentence localization method based on quadruple constraints and partial supervision in this invention embodiment is as follows: A novel quadruple constraint mechanism is proposed to comprehensively shape the alignment representation of both events and queries. Improvements are made in three aspects: task setting, cross-modal representation learning, and application scenarios. It makes full use of partial label information, covering intra-sample and inter-sample relationships, and mining and extraction of unimodal and multimodal information. This not only improves intra-cluster compactness and inter-cluster separability but also enables the separation of events from the background. Furthermore, a partially-fully joint framework is introduced, connected with an additional fully supervised branch bridge, to obtain the significant advantages brought by the fully supervised training paradigm. It has better robustness to partial criteria, reduces the heavy annotation cost in the fully supervised temporal sentence localization method, and improves the localization performance in the weakly supervised temporal sentence localization method. It can achieve localization results comparable to those of fully supervised settings under video segment supervision conditions. Partial supervision information is used to adjust the relationship between annotation cost and model localization accuracy. The quadruple constraints of intra-sample and inter-sample, unimodal and multimodal are used to enhance the pairing relationship between events and query text.

[0044] This application proposes a compromise partially supervised temporal sentence localization setting under both fully supervised and weakly supervised temporal sentence localization. By weakening the event timestamp of fully supervised temporal sentence localization to a segment of event timestamp supervision signal, or even further reducing it to a single frame supervision signal, this setting can greatly reduce the annotation cost of fully supervised temporal sentence localization, while greatly improving the localization performance of weakly supervised temporal sentence localization.

[0045] For partially supervised temporal sentence localization, this application proposes a quadruple constraint based on intra-sample and inter-sample, unimodal and cross-modal constraints to enhance the association learning between events and query statements in videos. The quadruple constraint enables events and query texts within the same cluster to cluster together, while separating events and query texts within different clusters, and also distinguishes the key foreground and background of events in the video. Furthermore, a framework is proposed to bridge and parallelize fully supervised and partially supervised branches, thereby relying on the localization pseudo-labels obtained from partially supervised training to promote the training of an advanced fully supervised temporal sentence localization architecture. The partially-fully parallel framework can directly predict the start timestamp of events during inference, providing a bridge between partially and fully supervised training.

[0046] In existing technologies, cross-modal temporal regression locator models can be used to jointly model text queries and video clips and output predicted candidate clips; self-supervised learning can also be performed using semantic completion networks.

[0047] The cross-modal temporal regression locator model is a token-supervised temporal sentence localization method. In video encoding, it edits unprocessed video into multiple video segments, extracts and encodes features from these segments, and then performs global pooling to obtain the corresponding video feature representations. On the other hand, in text query encoding, it embeds features from words in the text sentence to obtain word embedding features as text features. Then, it concatenates the video features and text features for multimodal information processing, ultimately obtaining the predicted localization boundary.

[0048] Semantic completion networks are a weakly supervised temporal sentence localization method. In the candidate bounding box generation stage, a series of candidate bounding boxes are predicted using cross-modal fusion representations of video and the query text, and a corresponding confidence score is generated for each bounding box. The top K candidate bounding boxes with the highest scores are then selected. In the semantic completion stage, important words in the query text are masked, and the query text is reconstructed using the cross-modal information of the video frames corresponding to the predicted bounding boxes and the masked sentences. The entire training phase relies on two types of supervision: a query semantic reconstruction loss and a ranking loss between the top K feedbacks of the predicted bounding boxes and the reconstruction loss.

[0049] However, in practical applications, cross-modal temporal regression locator models only consider the cross-modal information interaction between video and text, but do not consider the distinction between foreground and background in the video itself. They require fully supervised localization ground truth information for each query, which consumes a lot of manpower and time for annotation, resulting in high costs in actual deployment and use. While semantic completion networks use visual and text encoders and decoders for cross-modal information extraction and fusion, they do not consider the correspondence between events in the video itself and the query statements, and cannot separate events from the video background. They are mainly based on the reconstructed information at the query statement level, which cannot provide explicit supervision information for the localization of statements in the video. Therefore, they are difficult to achieve good localization performance and lack good robustness in practical applications.

[0050] Based on the problems existing in the prior art, this application mainly addresses how to find a compromise technique under two settings: fully supervised temporal sentence localization and weakly supervised temporal sentence localization, so as to achieve good localization performance of the model with a small amount of additional annotation. In cross-modal representation learning, this invention mainly addresses how to better learn the alignment relationship between video and query statements, thereby better locating the video segment described by the query statement within the video. In terms of application scenarios, this invention mainly addresses how to link the algorithm under partially supervised conditions with the fully supervised method flow, so as to better utilize the advantages and convenience of the fully supervised method flow in application.

[0051] The overall solution mainly includes the following steps: Extracting high-level semantic features from query text and video using deep neural networks to provide discriminative video features for event detection; fusing frame-by-frame algorithms and candidate box algorithms to obtain the timestamps corresponding to the query events from the video features, and inputting the video features fused with query text features into the event detector to predict the center timestamp and event duration of the corresponding video segment; constructing a quadruple constraint model to facilitate learning in semi-supervised temporal sentence localization tasks, the quadruple constraint model including an in-sample event-query text multimodal constraint model, a visual single-modal constraint model, and an inter-sample event-query multimodal constraint model and a visual single-modal constraint model; using a partial-to-full merging framework to predict temporal localization results based on the above features to flexibly handle both partial and full supervision; optimizing the partial and full supervision branches, the partial supervision branch optimization including optimization of the quadruple constraint model and optimization of the localization loss constructed from the localization pseudo-labels of the partial supervision information; the full supervision branch optimization using a localization learning framework and relying on localization pseudo-labels for supervised learning optimization.

[0052] The localization system of this application consists of two components: a partially supervised branch component and a fully supervised branch component. The partially supervised branch is trained and learned by relying on four-fold constraints, while the fully supervised branch is trained and learned by relying on pseudo-localization labels obtained from the partially supervised labels.

[0053] For video features, the center timestamp and length of the event video segment are estimated using an event detector. Then, using the supervision information from partial labels, localization pseudo-labels can be generated. For query text features, Gaussian shape approximation can be used to separate background and foreground features, thereby applying four constraints to the event: intra-sample and inter-sample, unimodal and multimodal. Furthermore, the previously generated localization pseudo-labels can be used as standard localization loss for supervision.

[0054] For the fully supervised branch, pseudo-labels generated using partially supervised information are used to train a state-of-the-art temporal sentence localization framework. This allows the model to fully leverage the advantages of the fully supervised learning paradigm during inference, directly predicting the start timestamp of the video segment corresponding to the query statement. By bridging and paralleling the partially supervised and fully supervised branches, this invention maximizes the advantages brought by partially supervised information.

[0055] Preferably, the process of extracting high-level semantic features from query text and video using deep neural networks to provide discriminative video features for event detection includes the following steps: Feature extraction is performed on the video stream using a pre-trained 3D convolutional network. The extracted features have two dimensions: the number of frames in the video sequence and the number of feature channels in the video as a visual modality. Feature extraction is performed on the query text using a global vector based on word representation. The extracted text query features have two dimensions: the number of words in the query text and the number of channels in the text features. A fully connected layer is used to fine-tune the features of both the video and query text modalities to obtain single-modal features for both the video stream and the query text. Two cross-modal Transformers are introduced to apply a cross-modal attention mechanism to the features of both the video stream and the query text modalities. Specifically, for the video stream, text features are used as query features, while video features are used as key features and key-value features, resulting in cross-modal fused video features. For the query text, video features are used as query features, while text features are used as key features and key-value features, resulting in cross-modal fused text features.

[0056] Furthermore, for the predicted event start time, a temporal mask can be generated to facilitate the computation of the event-text query representation. There are two possible methods to generate this differentiable temporal mask: one is to approximate it using a Gaussian shape, and the other is to approximate it using a modified shape distribution with a flat mask. Based on the generated differentiable mask, visual features can be separated into foreground and background features for subsequent quadruple constraints.

[0057] The four constraints of this application include an intra-sample event-query text multimodal constraint model, a visual unimodal constraint model, and an inter-sample event-query multimodal constraint model and a visual unimodal constraint model. Compared to text, video streams contain rich, fine-grained contextual information. This contextual information enhances the continuity of the video, but also causes events and background to have similar features, thus increasing the difficulty of localization. To address this, this invention proposes an intra-sample visual unimodal constraint, ensuring that the semantic similarity between event features and the overall video features is higher than the semantic similarity between event features and background features.

[0058] For video and query text pairs, samples with similar query text are considered positive pairs, while samples with different query text descriptions are considered negative pairs. For multimodal alignment, events and query texts from the same cluster are considered positive pairs, while events and query texts from different clusters are considered negative pairs. Based on the defined positive and negative pairs, contrastive loss can be used to perform multimodal constraints on the event-query text pairs between samples.

[0059] To model the semantics of clustering in visual modalities, events from the same cluster are treated as positive samples, while events from different clusters are treated as negative samples. Based on the defined visual positive and negative samples, contrastive loss can be used to amplify the differences between visual modal samples.

[0060] The features learned using the aforementioned constraints possess powerful representational capabilities, enabling temporal localization result prediction. A partially-fully merged framework is proposed, which achieves direct prediction of localization results during inference by adding an additional fully supervised branch to the partially supervised branch. This approach offers three advantages.

[0061] Structurally, this framework bridges full and partial supervision, allowing partial supervision to leverage the advantages of existing fully supervised methods; for example, explicit localization optimization objectives, better pyramid structures, and advanced detection paradigms. Functionally, the framework acts as akin to self-training or knowledge extraction. The fully supervised branch can further denoise and refine the pseudo-labels of the partially supervised branch to obtain better localization results. From an annotation perspective, this framework avoids the annotation gap between training and testing in partially supervised methods by using only the fully supervised branch for effective inference.

[0062] Furthermore, the high degree of freedom of partially supervised labels within event intervals poses a significant challenge to the robustness of the framework. With the help of a fully supervised branch, the partially-fully merged framework can achieve consistent effectiveness across multiple label distributions. Moreover, the framework's flexibility in handling both types of supervision, enabling it to learn jointly from a wider range of data, may lead to even stronger performance.

[0063] Furthermore, this application can achieve significant performance improvements with only a small increase in annotation costs. In particular, using video clips for supervision can achieve localization performance comparable to that of fully supervised information.

[0064] To verify the effectiveness of the temporal sentence localization method based on four constraints and partial supervision, experiments were conducted on the Charades-STA and ActivityNet Caption datasets. The Charades-STA dataset contains 9848 videos of daily indoor activities, which can be divided into 5338 training videos and 1334 test videos. The training videos contain 12408 event-query annotations, while the test videos contain 3720 event-query annotations. The ActivityNet Caption dataset contains 19994 videos from different domains. For ease of training and evaluation, the training set provides 37421 event-query annotations, the validation set provides 17505, and the test set provides 17031.

[0065] Based on existing work, “R@K, IoU=M” is used as the evaluation metric, that is, in the first K recalls, the percentage of the intersection-union ratio (IoU) of the predicted event timestamps and the real event timestamps is greater than M; in order to further evaluate the quality of the localized pseudo-labels, the average IoU ratio on the localized ground truth labels is also shown.

[0066] The most important aspect of this invention is the proposal of a partially supervised temporal sentence localization framework, which employs single-frame supervision or short video segments for oversight. To verify the effectiveness of the proposed quadruple constraints, the method of this invention was compared with fully supervised and weakly supervised temporal sentence localization methods.

[0067] Overall, the framework of this invention achieves state-of-the-art performance at most intersection-union ratio (IU) thresholds. For example, on the Charades-STA dataset, it improves the average IU by 4.08% compared to previous state-of-the-art methods, significantly narrowing the performance gap between fully supervised and partially supervised approaches.

[0068] Furthermore, compared to ViGA, this invention achieves greater gains under stringent performance metrics than under lenient metrics, for example, a 0.36% improvement at R@0.3 and an 8.07% improvement at R@0.7. Since stringent metrics are more practical in real-world applications, the positioning performance of this invention has proven to be more refined and accurate.

[0069] It should be noted that this invention can be applied to tasks closely related to video library segment retrieval, such as video retrieval, image retrieval, and video segment retrieval. It can also be used as an extractor of cross-modal features of video, images, and text, and applied to other visual language cross-modal tasks.

[0070] In summary, the video temporal sentence localization method based on quadruple constraints and partial supervision in this invention proposes a novel quadruple constraint mechanism to comprehensively shape the alignment representation of both events and queries. Improvements are made in three aspects: task setting, cross-modal representation learning, and application scenarios. It fully utilizes partial label information, covering intra- and inter-sample relationships, and mining and extracting unimodal and multimodal information. This not only improves intra-cluster compactness and inter-cluster separability but also separates events from the background. Furthermore, a partially-fully joint framework is introduced, connected to an additional fully supervised branch bridge, to gain the significant advantages of the fully supervised training paradigm. It exhibits better robustness to partial criteria, reduces the heavy annotation costs in fully supervised temporal sentence localization methods, and improves localization performance in weakly supervised temporal sentence localization methods. It can achieve localization results comparable to fully supervised settings under video segment supervision conditions. Partial supervision information is used to adjust the relationship between annotation costs and model localization accuracy. The quadruple constraints—intra- and inter-sample, unimodal and multimodal—are used to enhance the pairing relationship between events and query text.

[0071] The above specific embodiments should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, any alternative improvements or modifications made to the embodiments of the present invention shall fall within the scope of protection of the present invention.

[0072] Any aspects of this invention not described in detail are well-known to those skilled in the art.

Claims

1. A method for video temporal sentence localization based on four constraints and partial supervision, characterized in that, The positioning method includes the following steps: S1 uses a deep neural network to extract high-level semantic features from the query text and video, providing discriminative video features for event detection; S2, the frame-by-frame algorithm and the candidate box algorithm are combined to obtain the timestamp corresponding to the query event from the video features. The video features that are combined with the query text features are input into the event detector to predict the center timestamp and event duration of the video segment corresponding to the event. S3. Construct a quadruple constraint model to facilitate the learning of semi-supervised temporal sentence localization tasks. The quadruple constraint model includes an in-sample event-query text multimodal constraint model, a visual unimodal constraint model, and an inter-sample event-query multimodal constraint model and a visual unimodal constraint model. S4. A partial-to-complete merging framework is used to predict the temporal localization results based on the above features, so as to flexibly handle two types of supervision: partial supervision and complete supervision. S5, optimize the partially supervised branch and the fully supervised branch. The optimization of the partially supervised branch includes the optimization of the quadruple constraint model and the optimization of the localization loss constructed from the localization pseudo-labels using partially supervised information. The optimization of the fully supervised branch is to use a localization learning framework and rely on localization pseudo-labels for supervised learning optimization.

2. The video temporal sentence localization method based on four constraints and partial supervision according to claim 1, characterized in that, Extracting high-level semantic features from query text and video using deep neural networks to provide discriminative video features for event detection includes the following steps: S1.1, a pre-trained 3D convolutional network is used to extract features from the video stream. The extracted features have two dimensions: the number of frames in the video sequence and the number of feature channels of the video as a visual modality. S1.2, feature extraction is performed on the query text using a global vector based on word representation. The extracted text query features have two dimensions: the number of words in the query text and the number of channels in the text features. S1.3, a fully connected layer is used to fine-tune the features of the video and query text modalities to obtain the single-modal features of the video stream and query text modalities; S1.4 introduces two cross-modal Transformers to apply a cross-modal attention mechanism to the features of the video stream and the query text. Specifically, for the video stream, text features are used as query features, while video features are used as key features and key-value features to obtain cross-modal fused video features; for the query text, video features are used as query features, while text features are used as key features and key-value features to obtain cross-modal fused text features.

3. The video temporal sentence localization method based on four constraints and partial supervision according to claim 1, characterized in that: The frame-by-frame algorithm is used to continuously determine whether each frame of a video segment belongs to the query text feature; the candidate box algorithm is used to predict the intermediate timestamp of the time and the corresponding event duration.

4. The video temporal sentence localization method based on four constraints and partial supervision according to claim 1, characterized in that: The predicted center timestamp and event duration are converted into the start and end times of the event, and then used to construct location pseudo-labels under partial supervision. For the predicted start time, two computational methods are used to generate a differentiable time mask to facilitate the calculation of the event-text query representation. The computational methods include approximation using a Gaussian shape and approximation using an improved shape distribution with a flat mask. Based on the generated differentiable time mask, visual features are separated into foreground and background features to obtain foreground and background features for subsequent quadruple constraints.

5. The video temporal sentence localization method based on four constraints and partial supervision according to claim 1, characterized in that: For the event-query multimodal constraint model between samples, by extracting frame-by-frame video features from the entire video features through pooling operations, the semantic similarity between event features and query text features is higher than that between the entire video features and query text.

6. The video temporal sentence localization method based on four constraints and partial supervision according to claim 1, characterized in that: The partially-fully-joined framework achieves direct prediction of localization results during inference by adding an additional fully-supervised branch to the partially-supervised branch.

7. The video temporal sentence localization method based on four constraints and partial supervision according to claim 1, characterized in that: The localization system of the video temporal sentence localization method based on quadruple constraints and partial supervision includes a partially supervised branch component and a fully supervised branch component. The partially supervised branch component is trained and learned under quadruple constraints, and the fully supervised branch component is trained and learned using pseudo-localization labels obtained from partially supervised labels.

Images