Zero sample video time positioning method and system

By generating free-form natural language pseudoqueries and using event timing structure to generate high-quality pseudo-events, combined with pseudo-label optimization methods, the problems of simplicity and noise impact of pseudo-queries in the existing technology are solved, and the accuracy of video time positioning is significantly improved .

CN119025706APending Publication Date: 2024-11-26BEIJING INSTITUTE FOR GENERAL ARTIFICIAL INTELLIGENCE
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202311133422.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-05-24
Filing Date
2023-09-04
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the existing zero-sample video time positioning method, pseudo-queries are simple and difficult to explain and ignore the event timing structure, resulting in misalignment of pseudo-events and queries, and fail to effectively alleviate the impact of pseudo-label noise, limiting the robustness and accuracy of the model. sex.

Method used

Free form natural language pseudoqueries are generated based on pre-trained models, high-quality pseudo-events are generated using event timing structures, and noise impact is reduced through pseudo-label filtering and sample weighting optimization methods, and video time positioning model is trained.

Benefits of technology

It improves the model's performance ability on real queries, enhances the quality of pseudo-labels, reduces the impact of noise on model training, and significantly improves the accuracy of video time positioning.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025706A_ABST
    Figure CN119025706A_ABST
Patent Text Reader

Abstract

The invention relates to a zero sample video time positioning method and system. The method comprises the following steps: generating a free-form natural language query for a video by using a pre-trained image description generation model; based on a time sequence structure of the event, namely, the correlation between the interior of the event and the query should be high, and the correlation between the exterior of the event and the query should be low, the event with the maximum difference between the correlation in the event and the correlation outside the event is selected as a pseudo event corresponding to the pseudo query; meanwhile, low-quality pseudo query-event pairs are filtered based on a time sequence structure of the event; and training a video time positioning model by using the screened pseudo query-event pair, and reducing the influence of noise in pseudo query and pseudo events on model training through pseudo tag optimization. According to the method, the video time positioning model can be trained without manual annotation, and the manual annotation cost is reduced.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of video positioning, and in particular to a zero-sample video time positioning method and system based on pseudo-label generation. Background Art

[0002] Video temporal localization aims to locate the most relevant video segments for a given query from an untrimmed video based on a free-form natural language query. The video temporal localization model supports free-form natural language queries, which allows the model to flexibly adapt to the needs of different practical applications. Its potential applications in video surveillance, robotic operation and other fields have attracted increasing attention. In recent years, the performance of video temporal localization has been improved with the help of advanced deep learning techniques and large-scale annotated data.

[0003] In the field of video sentence localization, fully supervised methods are usually used to train models based on manual annotations of the query and start and end timestamps of each video. For example, Zhang et al. proposed the 2D-TAN (2D Temporal Adjacent Networks) algorithm to generate candidate time boxes and sort them using multimodal information of adjacent frames. However, the high cost of manual annotation limits the scalability of fully supervised methods. In addition, research published by Yuan et al. in 2021 also showed that annotation bias in the dataset may also affect the robustness of these models. In order to reduce the annotation cost, weakly supervised methods only use videos and queries to train models. For example, the SCN (Semantic Completion Network) algorithm proposed by Lin et al. trains the model through semantic reconstruction, assuming that a video clip with a high match to the query sentence should be able to reconstruct the entire sentence well. However, weakly supervised methods are still affected by language-related annotation biases, and the annotation cost of natural language queries is also inevitable. Therefore, this method studies zero-shot video temporal localization, using only video data without any manual annotations for video sentence localization, which is more practical but also more challenging.

[0004] In the zero-shot setting, the training process only requires video data. Existing zero-shot methods follow the same process, i.e., finding event proposals in the video, then generating pseudo queries for the events, and finally training the video temporal localization model using the pseudo queries and the corresponding events. For example, the PSVL method proposed by Nam et al. in 2021 first extracts event proposals in the video, and then generates simplified pseudo queries by detecting nouns in the event proposals and finding verbs related to these nouns. In 2021, Gao et al. directly generated pseudo query features in the pre-trained visual language feature space and used the features to train the model directly. However, the pseudo queries in existing methods are either too simple or not very interpretable. In addition, the existing processes do not consider the temporal structure of events, which may lead to misalignment between pseudo events and queries. Finally, these methods ignore the risk of pseudo label noise when using pseudo labels for model training. This method generates free-form interpretable pseudo queries by modeling the temporal structure of events, constructs query-dependent event proposals, and proposes a pseudo label optimization method to mitigate the impact of pseudo label noise. Summary of the invention

[0005] In view of the deficiencies of the prior art, the present invention proposes a zero-sample video time positioning method and system.

[0006] In order to achieve the above purpose, the technical solution adopted by the present invention is as follows:

[0007] A zero-shot video temporal localization method for localizing a video segment most relevant to a query statement comprises the following steps:

[0008] (1) Pseudo query generation: Use the pre-trained model to generate a free-form natural language description based on the video content as a pseudo query for the video;

[0009] (2) Pseudo-event generation: Based on the video content and pseudo-query, pseudo-events are generated using the temporal structure of events to obtain pseudo-query-event pairs;

[0010] (3) Pseudo-label filtering: Based on the temporal structure of events, low-quality pseudo-labels are filtered out, that is, low-quality pseudo-query-event pairs are filtered out;

[0011] (4) Noise-robust video temporal localization model training: The video temporal localization model is trained using filtered pseudo-query-event pairs. The impact of noise in pseudo-queries and pseudo-events on model training is reduced through sample weighting and pseudo-label optimization methods. Video temporal localization is performed based on the trained video temporal localization model.

[0012] Furthermore, step (1) uses a pre-trained BLIP (Bootstrapping Language-Image Pre-training) model to generate a free-form natural language description based on the content of the video frame as a pseudo query for the video.

[0013] Furthermore, step (2) generates a pseudo event using the pseudo query obtained in step (1). Based on the temporal structure of events, that is, the correlation between the event and the query should be higher than the correlation between the event and the query, the event with the largest difference between the correlation within the event and the correlation outside the event is selected as the pseudo event corresponding to the pseudo query, thereby obtaining a pseudo query-event pair.

[0014] Furthermore, step (3) utilizes the temporal structure of events to filter out the low-quality pseudo query-event pairs obtained in step (2).

[0015] Furthermore, step (4) uses the pseudo query-event pairs filtered in step (3) to train the video temporal localization model. In order to reduce the noise in the pseudo labels, this method proposes to estimate the noise and reweight each sample during the training process, while fine-tuning the pseudo labels.

[0016] A zero-sample video time positioning system using the above method comprises:

[0017] A pseudo-query generation module, which is used to generate a free-form natural language description based on the content of the video frame using a pre-trained model as a pseudo-query for the video;

[0018] A pseudo event generation module is used to generate corresponding pseudo events for each pseudo query based on the temporal structure of events, thereby obtaining pseudo query-event pairs;

[0019] The pseudo-label filtering module is used to filter low-quality pseudo-query event pairs based on the temporal structure of events;

[0020] The noise-robust model training module is used to train the video temporal localization model using the filtered pseudo query-event pairs, and reduce the impact of noise in pseudo queries and pseudo events on model training through sample reweighting and pseudo label optimization.

[0021] The beneficial effects of the present invention are:

[0022] The present invention proposes a pseudo-label generation method based on event temporal structure. On the one hand, it can generate free-form natural language pseudo-queries according to video content, and generate pseudo-events using the temporal structure of video events, and train a video time positioning model through pseudo-queries and pseudo-events; on the other hand, the method proposes a pseudo-label optimization method to reduce the impact of noise in pseudo-queries and pseudo-events on model training.

[0023] The reason why the present invention has the above-mentioned invention effect is that: firstly, by generating a free-form natural language description as a pseudo-query, the distribution of the pseudo-query is closer to that of the real query, which improves the performance of the model when facing the real query during the test; secondly, by generating pseudo-events based on the event temporal structure and filtering pseudo-labels, the quality of pseudo-labels is improved. Finally, by optimizing pseudo-labels, the influence of noise in pseudo-labels on the training video time localization model is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of the video time positioning method based on pseudo-label generation of the present invention. DETAILED DESCRIPTION

[0025] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0026] A video time positioning method of the present invention has the following process: Figure 1 As shown, the specific steps include:

[0027] (1) Pseudo query generation

[0028] Given a video V, this method first uniformly samples N frames v1,v2,...,v N Then, a pre-trained BLIP model is used to generate a natural language description for each frame. Since video frames may contain rich content, this method generates multiple descriptions for the same frame to ensure that the description of the video is as complete as possible. Finally, these descriptions c1,c2,...,c M as pseudo query candidates, where M is the number of pseudo query candidates.

[0029] (2) Pseudo-event generation

[0030] In this step, this method fully considers the correlation between the internal and external events and the query, and generates pseudo events (i.e., start and end timestamps) for each pseudo query candidate. Specifically, for each pseudo query candidate c1,...,c M , the BLIP text encoder pre-trained in this method extracts text features Where D is the feature dimension. Then, for each video frame v1,...,v N , use BLIP image encoder to extract image features Since BLIP text and image feature spaces are aligned, the cosine similarity of text and image features can be directly used to measure the correlation between the query and the video frame:

[0031]

[0032] This method believes that for a given query, the most relevant event should satisfy the requirement that the video inside the event is highly relevant to the query, while the video outside the event is less relevant to the query. Therefore, this method uses a sliding window to enumerate possible event proposals. Where N p is the number of event proposals, which are segments in the video where an event may exist. Then, we calculate the average similarity within each event and the average similarity outside each event, and use the difference between them as the quality of each event proposal:

[0033]

[0034] Where Q ik is the quality of the kth event proposal matched by the ith pseudo query, S ij is the correlation between the i-th query and the j-th frame, |p k | is an event proposal k Finally, to ensure that the event corresponding to each query is unique, this method selects the event proposal with the highest quality as the pseudo event label for the i-th query:

[0035]

[0036] (3) Pseudo-label filtering

[0037] Since the quality of a large number of pseudo query-event pairs is uneven, this method will further filter them. A good pseudo query-event pair should not be too general, that is, the correlation with the corresponding event internal video should be as high as possible, and the correlation with the event external video should be as low as possible. This means that the quality of the best event proposal for each query candidate can also be used in the formula to evaluate the quality of the query-event pair.

[0038] Specifically, this method defines the quality of the i-th query-event pair as:

[0039]

[0040] This method does not want too many queries to describe the same event in the video, so this method will further use the non-maximum suppression method to filter out pseudo query-event pairs whose intersection-over-union ratio between events is higher than a certain threshold. c The order of retaining the first K query-event pairs.

[0041] (4) Noise-robust video temporal localization model training:

[0042] In this step, our method can use the generated pseudo queries and their corresponding events to train any fully supervised video temporal localization model. However, most existing fully supervised models are designed for clean training data and may not be robust enough for pseudo labels containing a lot of noise. Therefore, our method designs a sample reweighting and label correction method to reduce the impact of noise on the fully supervised model.

[0043] Sample reweighting: It has been shown that neural networks are first trained to fit clean data and then to fit noise. Therefore, the model’s confidence in its predictions can reflect the noise in the sample. That is, if the model is more confident in its predictions and the predictions are close to the training labels, there is relatively less noise in the data. Specifically, this method uses the video-text matching score between the prediction and the pseudo query given by the fully supervised video localization model as the confidence of the i-th training sample Then, this method calculates the intersection-over-union ratio between the prediction and the pseudo-label and The higher it is, the lower the noise is, and the greater the weight of the sample is. Therefore, this method defines the sample weight as:

[0044]

[0045] Where α is a hyperparameter used to balance and Finally, this method uses the loss function of fully supervised video temporal localization and uses w to reweight the losses of different samples. By reweighting the samples, this method can reduce the negative impact of noisy labels, but this method hopes that noisy labels can also provide useful training signals, so this method further proposes a label optimization method to correct noisy labels.

[0046] Label optimization: During training, if the model is confident enough in its prediction, it is likely that the prediction is the true label. In addition, this method also believes that there should not be much difference between the true label and the pseudo label, so this method also considers the intersection-over-union ratio between the prediction and the pseudo label to prevent the model from being too confident in wrong predictions. Specifically, this method can use the visual-textual matching score of the kth event proposal and the pseudo query in the fully supervised video localization model And the intersection of the k-th event proposal and the pseudo label This method will select the event proposals as optimized pseudo labels for the next round of model training, where β is a hyperparameter. In this way, if the model is confident enough in predicting the correct label, it is possible to optimize the noisy label to the correct label. The overall loss function is of the form:

[0047]

[0048] where w i is the weight of the ith pseudo query-event pair used for training, calculated using the formula for calculating w above, V i is video, c is i is a pseudo query. is a pseudo event optimized using the above label optimization method, L loc is the localization loss function used by the fully supervised video temporal localization model, and B is the batch size.

[0049] The following experimental results show that compared with the existing methods, the video time positioning method of the present invention can achieve higher positioning accuracy.

[0050] This embodiment uses the Charades-STA dataset. The dataset consists of 12,408 and 3,720 Moments of Interest (MoIs) in the training set and validation set, respectively. The videos in this dataset vary in video length, lens motion, viewing angle, video quality, etc., so positioning is quite challenging, which helps to prove the effectiveness of this method.

[0051] Existing method 1: The method proposed by Nam et al. in the document “Zero-shot Natural Language Video Localization”.

[0052] Existing method 2: The method proposed by Wang et al. in the document “Prompt-based Zero-shot Video Moment Retrieval”.

[0053] Existing method three: The method proposed by Kim et al. in the paper “Language-free Training for Zero-shot Video Grounding.”

[0054] The present invention: the method of this embodiment.

[0055] The experiment uses the recall rate under a specific intersection over union (IoU) to evaluate the accuracy of video localization. This metric measures the percentage of predicted video segments with an IoU greater than or equal to a threshold m (IoU@m) between the predicted video segments and the correct results. In the Charades-STA dataset, the values ​​of m are 0.5 and 0.7.

[0056] As can be seen from Table 1, the present invention has achieved better video positioning results in the zero-sample video time positioning task. The existing methods have no way to generate free-form natural language pseudo-queries, and do not consider the temporal structure of events, resulting in low quality of pseudo-labels. The impact of pseudo-label noise on the model is also not considered when using pseudo-labels to train the model. On the one hand, the present invention can generate free-form natural language pseudo-queries based on video content, and use the temporal structure of video events to generate higher quality pseudo-events; on the other hand, a pseudo-label optimization method is proposed to reduce the impact of noise in pseudo-queries and pseudo-events on model training. Therefore, it has better performance.

[0057] Table 1. Comparative experimental results with existing methods (unit: %)

[0058] method IoU=0.3 IoU=0.5 Existing method 1 46.47 31.29 Existing method 2 46.83 33.21 Existing method three 52.95 37.24 The present invention 60.73 40.70

[0059] Based on the same inventive concept, another embodiment of the present invention provides a video time positioning system using the above method, which includes:

[0060] The pseudo-query generation module uses a pre-trained model to generate a free-form natural language description based on the content of the video frame as a pseudo-query for the video;

[0061] The pseudo-event generation module selects the corresponding pseudo-event for each pseudo-query based on the temporal structure of events;

[0062] The pseudo-label filtering module filters low-quality pseudo query-event pairs based on the temporal structure of events;

[0063] The noise robust model training module uses the filtered pseudo query-event pairs to train the video temporal localization model, and reduces the impact of noise in pseudo queries and pseudo events on model training through pseudo label optimization.

[0064] Based on the same inventive concept, another embodiment of the present invention provides an electronic device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present invention.

[0065] Based on the same inventive concept, another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, CD), which stores a computer program. When the computer program is executed by a computer, it implements the various steps of the method of the present invention.

[0066] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A zero-sample video temporal positioning method, characterized in that: The following steps are involved: Use the pre-trained model to generate free-form natural language descriptions based on the content of the video frames as pseudo queries for the video; Based on the video content and pseudo-query, pseudo-events are generated using the temporal structure of events to obtain pseudo-query-event pairs; Utilize the temporal structure of events to filter out low-quality pseudo query-event pairs; The filtered pseudo-query-event pairs are used to train the video time localization model. The impact of noise in pseudo-queries and pseudo-events on model training is reduced through sample reweighting and pseudo-label optimization. Video time localization is performed based on the trained video time localization model.

2. The method according to claim 1, characterized in that The method of using a pre-trained model to generate a free-form natural language description according to the content of a video frame includes: uniformly sampling N video frames, using a pre-trained BLIP model to generate a description for each frame, and using the descriptions as pseudo queries.

3. The method according to claim 2, characterized in that The method of generating pseudo events by using the temporal structure of events is to consider the correlation between the query and the event and the event outside the event, and to generate pseudo events for each pseudo query, including: Use the BLIP model to extract the image features F of the video frames respectively v and the pseudo-query text features F c , directly use the cosine similarity of text and image features to measure the correlation between the query and the video frame: Among them, S ij is the correlation between the i-th query and the j-th frame, M is the number of pseudo queries, and N is the number of video frames.

4. The method according to claim 3, characterized in that Compute the average similarity within each event and the average similarity outside each event, and use the difference between them as the quality of each event proposal: Where Q ik is the quality of the kth event proposal matched by the ith pseudo-query, |p k | is an event proposal k The event proposal with the highest quality is selected as the pseudo event label for the i-th query.

5. The method according to claim 4, characterized in that The method of filtering low-quality pseudo query-event pairs by utilizing the temporal structure of events includes: The quality of the i-th query-event pair is defined as: The non-maximum suppression method is used to filter out query-event pairs whose intersection-over-union ratio between events is higher than a certain threshold; finally, according to the quality Q c The order of retaining the first K query-event pairs.

6. The method according to claim 1, characterized in that The sample weights are defined in the sample reweighting as: in is the confidence of the i-th training sample, is the intersection-over-union ratio between the prediction and the pseudo-label, and α is a hyperparameter used to balance and The impact of We use the loss function for fully supervised video temporal localization and use w to reweight the loss of different samples.

7. The method according to claim 1, characterized in that The pseudo-label optimization includes: Confidence of candidate events output by the fully supervised video localization model And the intersection and union with the pseudo label Select event proposals as optimized pseudo labels for the next round of model training, where β is a hyperparameter.

8. A zero-sample video temporal positioning system, characterized in that: include: A pseudo-query generation module, which is used to generate a free-form natural language description based on the content of the video frame using a pre-trained model as a pseudo-query for the video; A pseudo event generation module is used to generate corresponding pseudo events for each pseudo query based on the temporal structure of events, thereby obtaining pseudo query-event pairs; The pseudo-label filtering module is used to filter low-quality pseudo query-event pairs based on the temporal structure of events; The noise-robust model training module is used to train the video temporal localization model using the filtered pseudo query-event pairs, and reduce the impact of noise in pseudo queries and pseudo events on model training through sample reweighting and pseudo label optimization.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program comprises instructions for executing the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a computer, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Cited By

  • Training-free video corpus moment retrieval method based on self-adaptive calibration mechanism

    CN122153119A