Weak-supervision video clip locating method and system based on large-scale video corpus

Through self-supervised learning and weakly supervised multi-scale comparison learning, mapping video features and text features in the measurement space, the problems of low accuracy and low efficiency of video clip positioning in the existing technology are solved, and fast and accurate in large-scale video databases are achieved position.

WO2024230026A1PCT designated stage expired Publication Date: 2024-11-14SHANDONG JIANZHU UNIV

Patent Information

Application Number
PCT/CN2023/117063
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-11
Filing Date
2023-09-05
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Existing video clip positioning methods require a large number of manually labeled data sets, with low positioning accuracy and low efficiency, especially in large-scale video corpus.

Method used

Self-supervised learning is used to extract semantic information between text and video, and map video features and text features in the measurement space through weakly supervised multi-scale comparison learning, and quickly locate using the similarity of query statements.

Benefits of technology

It reduces the cost of data labeling, improves the accuracy and efficiency of video positioning, and can quickly and accurately locate video clips in large-scale video databases.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN2023117063_14112024_PF_FP_ABST
    Figure CN2023117063_14112024_PF_FP_ABST
Patent Text Reader

Abstract

The present invention relates to the technical field of video data identification. Provided are a weak-supervision video clip locating method and system based on large-scale video corpus. The method comprises the following steps: for an acquired training data set, extracting common semantic information between text and a video by using self-supervised learning, and obtaining a fused semantic video feature on the basis of the semantic information; for the fused semantic video feature and a corresponding text feature, performing multi-scale comparative learning by using a weak supervision method, determining a spatial mapping relationship between the video feature and the text feature, and mapping the spatial mapping relationship into a metric space, so as to obtain a trained metric space; acquiring a query statement, searching the trained metric space for text features similar to that of the query statement, and taking, as a video locating result, a video clip corresponding to a text feature with the highest similarity. By means of the locating method of the present invention, a video clip can be directly, accurately and quickly located from a large-scale video database.
Need to check novelty before this filing date? Find Prior Art

Description

Weakly supervised video clip localization method and system based on large-scale video corpus

[0001] The present invention claims priority to the Chinese patent application filed with the Patent Office of China on May 11, 2023, with application number 202310523581.6 and invention name “Weakly supervised video clip localization method and system based on large-scale video corpus”, the entire contents of which are incorporated by reference into the present invention. Technical Field

[0002] The present invention relates to the technical field related to video data recognition, and in particular to a method and system for weakly supervised video segment positioning based on a large-scale video corpus. Background Art

[0003] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0004] Video segment localization based on a large-scale video corpus is a technology that can locate relevant video segments within a large video database using a query. Video segment localization technology is used in many applications, such as the security field, where it is necessary to locate a segment within a long video to quickly search for the target segment. This technology requires manual identification of the long video to be located and then using a query to locate it. When a video database contains a large number of long videos, manually finding this segment is very laborious.

[0005] The inventors discovered in their research that most existing video clip localization methods use supervised methods to localize video clips within a video corpus. A few weakly supervised methods utilize metric learning, where a model is trained to learn a joint feature space between the video and the query, and the distance between the two is measured within this joint feature space. Existing video clip localization methods suffer from the following problems: Firstly, the dataset used to train the video clip localization task requires annotation of the actual moments of localization, which is a very labor-intensive task. Secondly, when it comes to localizing video clips within a large video corpus, existing methods suffer from low localization accuracy and efficiency.

[0006] Summary of the Invention

[0007] In order to solve the above problems, the present invention proposes a weakly supervised video segment localization method and system based on a large-scale video corpus, which can directly and accurately locate video segments from a large-scale video database.

[0008] In order to achieve the above object, the present invention adopts the following technical solutions:

[0009] One or more embodiments provide a weakly supervised video segment localization method based on a large-scale video corpus, comprising the following steps:

[0010] Based on the obtained training dataset, self-supervised learning is used to extract the common semantic information between text and video, and fusion semantic video features are obtained based on the semantic information;

[0011] In order to integrate semantic video features and corresponding text features, a weakly supervised method is used to perform multi-scale contrastive learning, determine the spatial mapping relationship between video features and text features, and map them into the metric space to obtain the trained metric space.

[0012] Obtain the query statement, search for text features similar to the query statement in the trained metric space, and use the video clip corresponding to the text feature with the highest similarity as the video positioning result.

[0013] One or more embodiments provide a weakly supervised video segment localization system based on a large-scale video corpus, including:

[0014] Common semantic perception module: This module is configured to extract common semantic information between text and video using self-supervised learning based on the acquired training dataset, and obtain fused semantic video features based on the semantic information;

[0015] The spatial mapping module for video and text features is configured to perform multi-scale contrastive learning using a weakly supervised method on the fused semantic video features and the corresponding text features, determine the spatial mapping relationship between the video and text features, and map them into a metric space to obtain a trained metric space.

[0016] Matching module: It is configured to obtain a query statement, search for text features similar to the query statement in the metric space, and search for the video clip with the highest similarity to the text features in the trained metric space based on the text features to obtain the video positioning result.

[0017] An electronic device includes a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of the above method are completed.

[0018] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps of the above method are completed.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] (1) The present invention adopts a weak supervision method and does not rely on the labels of the dataset. Any video data containing titles can be used as training data for this method, which greatly reduces the labeling cost of the dataset.

[0021] (2) Self-supervised learning is used on the training data to extract the common semantic information between text and video, obtain better representation information, and make the feature expressions of the text modality and video modality of the same thing more similar, thereby improving the accuracy of video positioning.

[0022] (3) During the positioning process, priority is given to searching for text features that are similar to the new query statement, rather than directly using the new query statement features to calculate the distance with the video features in the metric space. This can reduce the amount of calculation and greatly improve the efficiency of video positioning.

[0023] The video positioning method of the present invention can be embedded in any visual platform, such as video entertainment, video surveillance, unmanned driving, etc., which can greatly improve the user experience.

[0024] The advantages of the present invention and its additional aspects will be described in detail in the following specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their description are used to explain the present invention but do not constitute a limitation of the present invention.

[0026] FIG1 is a flow chart of a method for identifying common semantic perception information between modalities according to Example 1 of the present invention;

[0027] 2 is a schematic flow chart of a method for spatial mapping of video features and text features according to Example 1 of the present invention;

[0028] FIG3 is a flow chart of a segment-level learning method in multi-scale contrastive learning according to Example 1 of the present invention;

[0029] FIG4 is a flow chart of a video-level learning method in multi-scale contrastive learning according to embodiment 1 of the present invention;

[0030] FIG5 is an overall flow chart of the video clip positioning method according to embodiment 1 of the present invention. DETAILED DESCRIPTION

[0031] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0032] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0033] It should be noted that the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof. It should be noted that, in the absence of conflict, the various embodiments of the present invention and the features in the embodiments can be combined with each other. The embodiments will be described in detail below with reference to the accompanying drawings.

[0034] Example 1

[0035] In the technical solution disclosed in one or more embodiments, as shown in FIG1 to FIG5 , a weakly supervised video segment localization method based on a large-scale video corpus includes the following steps:

[0036] Step 1: Based on the obtained training dataset, self-supervised learning is used to extract the common semantic information between text and video, and fusion semantic video features are obtained based on the semantic information;

[0037] Step 2: For the fused semantic video features and the corresponding text features, a weakly supervised method is used to perform multi-scale contrastive learning, determine the spatial mapping relationship between the video features and the text features, and map them into the metric space to obtain the trained metric space;

[0038] Step 3: Get the query statement, search for text features similar to the query statement in the trained metric space, and use the video clip corresponding to the text feature with the highest similarity as the video positioning result.

[0039] This embodiment employs a weakly supervised approach that doesn't rely on dataset labels. Any video data containing titles can serve as training data for this method, significantly reducing dataset annotation costs. Self-supervised learning is employed on the training data to extract shared semantic information between text and video, resulting in a more robust representation. This approach makes the feature representations of the same subject more similar in text and video modalities, thereby improving the accuracy of video localization. During localization, text features similar to the new query are prioritized, rather than directly using the new query features to calculate their distance to the video features in metric space. This reduces computational effort and significantly improves video localization efficiency.

[0040] In step 1, the training data set includes historical query statements and corresponding video data; the query statements are subjected to feature extraction to obtain text features.

[0041] Step 1 is used to perceive the common semantic information between the query and the video modalities. For example, if the table in the query and the table in the video have the same semantic information, then the feature representation of the table in the query should be similar to the feature representation of the table in the video.

[0042] In some embodiments, a method for extracting common semantic information between text and video using self-supervised learning and obtaining fused semantic video features based on the semantic information, as shown in FIG1 , includes the following steps:

[0043] Step 11: The backbone network can be used to extract features from the video data and the query statement to obtain video features and text features respectively;

[0044] Step 12: Fusing the obtained video features and text features (i.e., features of the query sentence);

[0045] Optionally, the fusion method is to add and dot-product the video features and text features and then concatenate them.

[0046] Step 13: Perform convolution operation on the fused features;

[0047] In this step, the convolution operation can reduce the dimension of the feature data to obtain more detailed information.

[0048] Step 14: Use the video features obtained after convolution to predict the text features, and use the original text features as supervision information for self-supervised training;

[0049] Step 15: Calculate the reconstruction loss based on the predicted text features and the supervision information. Give a larger weight to the video clips whose reconstruction loss is lower than the set value. This weight value is the reconstruction reward.

[0050] Step 16: Weight the reconstruction reward to the video features obtained after convolution to obtain the fused semantic video features.

[0051] This embodiment adopts self-supervised learning and uses video features that integrate text semantic information to re-predict text features, which can obtain more accurate text-video common semantic information, making the feature expressions of the text modality and video modality for the same thing more similar. Therefore, in the video positioning process, the corresponding video clips can be found more accurately through the features of the text modality, thereby improving the accuracy of video positioning.

[0052] Step 2 is the joint metric learning step. Joint metric space learning aims to learn a metric space, put all video clips and query texts into the metric space, and evaluate the video clips corresponding to the query based on the similarity in the space.

[0053] Figure 2 illustrates the spatial mapping of high-level video features and text features in metric space. First, the fused semantic video features obtained in step 1 are fed into a GRU (Long Short-Term Memory) network to obtain temporal information about the video. After each training session, the text features and high-level video features are mapped to the metric space. After several training cycles, the distance between text and highly matched video clips decreases, while that between text and less matched video clips increases. Different query pairs (i.e., different text-video pairs) become increasingly distant in space. A text and a video clip with a high degree of matching constitute a query pair.

[0054] In this embodiment, a better mapping space can be obtained through multi-scale contrastive learning, which includes: segment-level learning and video-level learning; segment-level learning is to shorten the distance between the query text and similar video segments, and to push away the video segments with dissimilar query texts; video-level learning is to push away the distance between the video corresponding to the query statement and other videos.

[0055] In this embodiment, video learning is divided into segment-level and video-level learning. The segment-level learning focuses on learning detailed features within a video, while the video-level learning focuses on learning the entire video. For example, video-level learning can determine whether a video is a sports video or a car video, while segment-level features can determine whether the video is high jump or weightlifting. These two scales of learning can be performed independently.

[0056] In this embodiment, multi-scale contrastive learning is the key to realizing weakly supervised learning, and the training of the metric space is realized through multi-scale contrastive learning.

[0057] The purpose of segment-level learning is to improve the matching score between text features and corresponding video segments. The segment-level learning method includes the following steps:

[0058] Step 21: Feed the fused semantic video features into a GRU (Long Short-Term Memory Network) to obtain the temporal information of the video, and add the obtained temporal information to the fused semantic video features to obtain high-level video features.

[0059] Step 22: The matching scores calculated between the text features and each high-level video segment feature are put into the locator, and the video segments corresponding to the start and end times with scores higher than the set value are marked as positive samples, and the unmarked videos are marked as negative samples;

[0060] The localizer can be a multi-layer perceptron fused with a normalized exponential function, i.e., MLP+softmax. The confidence score of each text segment-query pair is output by the normalization function.

[0061] The calculation formula of the normalization function softmax is as follows:

[0062] Among them, p is the confidence score, x i is the output value of the multilayer perceptron of the i-th segment, and k represents a total of k segments.

[0063] The method for calculating the matching degree between text features and each high-level video segment feature is as follows:

[0064] The Tanh function can be used to calculate the matching degree between features, that is, the matching degree r t Calculated by the following formula: t =Tanh(q t ·k t )

[0065] Where t represents the time step, q t represents high-level video segment features, k t Represents text features.

[0066] Step 23: Setting up a generative adversarial network (GAN network) to generate video segment features similar to the original video for segment-level learning. These generated video segment features serve as negative samples.

[0067] For example, video A contains n segments a1, a1…a n , locate segment a in video A based on query statement B i When training the GAN network, video A and query B are required as input. Here, the GAN network is responsible for generating segment features similar to video A. Here, "original video" refers to video A, that is, the GAN network generates video segment-level features similar to the training video ("original video").

[0068] Optionally, the number of negative samples generated by the Generative Adversarial Network (GAN network) can be set. For example, if the number N is set to 100, 100 additional video clip feature information can be generated.

[0069] Step 24: Based on the identified segment-level positive and negative samples, quantify the similarity between the samples using cosine similarity; and obtain a loss function by minimizing the loss function to maximize the probability that the two positive samples are similar.

[0070] All positive and negative samples in segment-level contrastive learning are segment-level samples, that is, one sample is one video segment.

[0071] The similarity between sample A and sample B is defined as:

[0072] The similarity probability between positive samples is calculated, where a positive sample pair (p i 、pj The probability calculation formula of ) is:

[0073] Among them, p i 、p j is a positive sample, N is the total number of samples, including negative samples. A positive pair is (positive sample, positive sample), a negative pair is (positive sample, negative sample), and e is the base of the natural logarithm, a mathematical constant that is an infinite non-repeating decimal and a transcendental number, approximately 2.71828.

[0074] By minimizing the loss function to maximize the probability of similarity between two positive samples, we can use logarithmic loss:

[0075] By performing comparative learning in the above manner, a metric space with more similar positive samples is obtained;

[0076] Step 25: After obtaining the loss function, the segment-level loss function Loss is calculated using the stochastic gradient descent algorithm. clip Optimize until the loss function converges, then stop optimizing. Repeat about 1000 iterations to obtain the metric space after segment-level optimization.

[0077] Loss convergence means that as the model is trained, the value of the model's loss function gradually decreases until it approaches a stable state.

[0078] In some other embodiments, the loss for segment-level contrastive learning may also be hinge loss.

[0079] Both segment-level learning and video-level learning are training processes, which are carried out in parallel, and the final result is a trained model.

[0080] Video-level contrastive learning is almost identical to segment-level contrastive learning. The only difference is the sample size. The sample for video-level contrastive learning is an entire video, while the sample for segment-level contrastive learning is a video segment.

[0081] Video-level learning is designed to eliminate the interference of other irrelevant video features. The video-level learning method includes the following steps:

[0082] Step 2-1: Use the input video for video-level learning as the positive sample for video-level contrastive learning, and randomly select multiple other video samples as negative samples;

[0083] Optionally, negative samples can be selected as 100 random video samples.

[0084] Step 2-2: Based on the identified video-level positive and negative samples, quantify the similarity between samples using cosine similarity; and maximize the probability of similarity between two positive samples by minimizing the loss function to obtain a loss function.

[0085] In step 2-2, one of the video-level positive samples and negative samples is the entire video.

[0086] The similarity between video-level samples is quantified by cosine similarity, that is, the similarity between video-level sample A and video-level sample B is defined as:

[0087] The similarity probability between positive samples is calculated, where a positive sample pair (p i 、p j The probability calculation formula of ) is:

[0088] Among them, p i 、p j is a positive sample, N is the total number of samples, including negative samples. The positive pair is (positive sample, positive sample), and the negative pair is (positive sample, negative sample).

[0089] By minimizing the loss function to maximize the probability of similarity between two positive samples, the logarithmic loss can be used:

[0090] By performing comparative learning in the above way, we can obtain a metric space with greater distances between different videos.

[0091] Step 2-3: After obtaining the loss function, the video level loss function Loss is calculated by using the stochastic gradient descent algorithm. video Optimize. Stop when the loss function converges. The loss converges after about 1000 iterations.

[0092] Steps 2-4: Obtain the video-level optimized metric space by optimizing the loss function;

[0093] In some other embodiments, the loss of segment-level contrastive learning may also adopt hinge loss;

[0094] In this embodiment, the weakly supervised training process is implemented with the help of contrastive learning, and the GAN network is creatively used to generate similar video clip features, thereby increasing the negative sample data required for contrastive learning.

[0095] In step 3, a query statement is obtained, and text features similar to the query statement are searched in the trained metric space. The video clip corresponding to the text feature with the highest similarity is used as the video positioning result.

[0096] The search for text features similar to the query statement in the metric space is implemented through hash binary codes. The specific steps are as follows:

[0097] Step 31: convert the text features in the metric space into a hash binary code through a hash mapping function, and use the hash binary code as a query index for the text-video query pair;

[0098] Step 32: Match the obtained query statement with a set number M of hash binary codes, where M can be set to 10;

[0099] Step 33: The video segment corresponding to the hash binary code with the highest matching degree is the video segment location. The start time and end time of the located video segment are the output timestamp information.

[0100] In this embodiment, the text hash code is used as the index of the text-video pair, and text features similar to the new query statement are searched first, rather than directly using the new query statement features to calculate the distance with the video features in the metric space. This improves query efficiency, reduces the amount of calculation, speeds up by about 10 times, and reduces time complexity by 90%.

[0101] In this embodiment, considering the positioning speed issue, a strategy of using text hash code as the overall text-video index is adopted, which effectively saves the amount of calculation and greatly increases the positioning speed.

[0102] Example 2

[0103] Based on Example 1, this embodiment provides a weakly supervised video segment localization system based on a large-scale video corpus, including:

[0104] Common semantic perception module: This module is configured to extract common semantic information between text and video using self-supervised learning based on the acquired training dataset, and obtain fused semantic video features based on the semantic information;

[0105] The spatial mapping module for video and text features is configured to perform multi-scale contrastive learning using a weakly supervised method on the fused semantic video features and the corresponding text features, determine the spatial mapping relationship between the video and text features, and map them into a metric space to obtain a trained metric space.

[0106] Matching module: It is configured to obtain a query statement, search for text features similar to the query statement in the metric space, and search for the video clip with the highest similarity to the text features in the trained metric space based on the text features to obtain the video positioning result.

[0107] It should be noted here that the various modules in this embodiment correspond one-to-one to the various steps in Example 1, and the specific implementation processes are the same, which will not be repeated here.

[0108] Example 3

[0109] This embodiment provides an electronic device, including a memory and a processor, and computer instructions stored in the memory and running on the processor. When the computer instructions are run by the processor, the steps described in the method of embodiment 1 are completed.

[0110] Example 4

[0111] This embodiment provides a computer-readable storage medium for storing computer instructions. When the computer instructions are executed by a processor, the steps described in the method of embodiment 1 are completed.

[0112] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A weakly supervised video segment localization method based on a large-scale video corpus, It is characterized in that The steps include: For the acquired training data set, self-supervised learning is used to extract the common semantic information between text and video, and the fused semantic video features are obtained based on the semantic information; The specific steps are: the backbone network can be used to extract features from video data and query statements to obtain video features and text features respectively; The obtained video features and text features, i.e., the features of the query sentence, are fused; The fusion method is to add and dot-product the video features and text features, and then concatenate them; Perform convolution operation on the fused features; The convolution operation can reduce the dimension of feature data to obtain more detailed information; The video features obtained after convolution are used to predict the text features, and the original text features are used as supervision information for self-supervised training; The reconstruction loss is calculated based on the predicted text features and supervision information, and a larger weight value is given to the video clips whose reconstruction loss is lower than the set value, which is the reconstruction reward; The reconstruction reward is weighted to the video features obtained after convolution to obtain the fused semantic video features; In order to integrate the semantic video features and the corresponding text features, a weakly supervised method is used to perform multi-scale contrastive learning, determine the spatial mapping relationship between the video features and the text features and map them to the metric space to obtain the trained metric space. Specifically, multi-scale contrastive learning includes segment-level learning and video-level learning. Segment-level learning is to bring the query text closer to similar video segments and push video segments with dissimilar query texts further away. Video-level learning is to push the video corresponding to the query sentence further away from other videos. The learning of videos is divided into segment level and video level. The segment level is the learning of detailed features in the video. The video level is to learn the entire video; the learning of these two scales can be carried out separately; Multi-scale contrastive learning is the key to weakly supervised learning. It can be used to train the metric space. The purpose of segment-level learning is to make the matching score between text features and corresponding video segments higher. The segment-level learning method includes the following steps: The fused semantic video features are fed into the GRU long short-term memory network to obtain the temporal information of the video, and the obtained temporal information is added to the fused semantic video features to obtain high-level video features; The matching scores calculated between the text features and each high-level video clip feature are put into the locator, and the video clips corresponding to the start and end times with scores higher than the set value are marked as positive samples, and the unmarked videos are used as negative samples; Among them, the locator can use a multi-layer perceptron fused with a normalized exponential function, that is, MLP+softmax; the confidence score of each text segment-query pair is output by the normalization function; The calculation formula of the normalization function softmax is as follows: Among them, p is the confidence score, x i is the output value of the multilayer perceptron of the i-th segment, and k means there are k segments in total; The method for calculating the matching degree between the text feature and each high-level video segment feature is as follows: The Tanh function is used to calculate the matching degree between features, that is, the matching degree r t It is calculated by the following formula: r t =Tanh(q t ·k t ) Where t represents the time step, q t represents the high-level video segment features, k t Represents text features; Set up a Generative Adversarial Network (GAN) network to generate video segment features similar to the original video for segment-level learning, and these generated video segment features are used as negative samples; Based on the identified segment-level positive and negative samples, the similarity between samples is quantified by cosine similarity; the loss function is obtained by minimizing the loss function to maximize the probability of similarity between two positive samples; All positive and negative samples in segment-level contrastive learning are segment-level samples, that is, one sample is one video segment; The similarity between sample A and sample B is defined as: The similarity probability between positive samples is calculated, where a positive sample pair (p i 、p j ) is calculated as: Among them, p i 、p j is a positive sample, N is the total number of samples, including negative samples; a positive pair is a positive sample and a positive sample, a negative pair is a positive sample and a negative sample, e is the base of the natural logarithm, a constant in mathematics, an infinite non-repeating decimal, and a transcendental number, approximately 2.71828; By minimizing the loss function to maximize the probability of similarity between two positive samples, we can use logarithmic loss, that is: By performing comparative learning in the above manner, a metric space with more similar positive samples is obtained; After obtaining the loss function, the segment-level loss function Loss is quantified by the stochastic gradient descent algorithm. clip Optimize until the loss function converges, then stop optimizing. After about 1000 iterations, the metric space after segment-level optimization is obtained. Video-level learning is to eliminate the interference of other irrelevant video features. Video-level learning methods, The steps include: The input video for video-level learning is used as the positive sample for video-level contrastive learning, and multiple other video samples are randomly selected as negative samples; Negative samples can be selected as 100 other random video samples; Based on the recognition of video-level positive and negative samples; quantify the similarity between samples by cosine similarity; maximize the probability of similarity between two positive samples by minimizing the loss function to obtain the loss function; One of the video-level positive and negative samples is the entire video; The similarity between video-level samples is quantified by cosine similarity, that is, the similarity between video-level sample A and video-level sample B is defined as: The similarity probability between positive samples is calculated, where a positive sample pair (p i 、p j ) is calculated as: Among them, p i 、p j is a positive sample, N is the total number of samples, including negative samples; a positive pair is a positive sample and a positive sample, and a negative pair is a positive sample and a negative sample; By minimizing the loss function to maximize the probability of similarity between two positive samples, the logarithmic loss can be used: By performing comparative learning in the above manner, a metric space with greater distances between different videos is obtained; After obtaining the loss function, the video level loss function Loss is quantified by the stochastic gradient descent algorithm. video Enter Optimize the algorithm; stop when the loss function converges. The loss converges after about 1000 iterations. By optimizing the loss function, the video-level optimized metric space is obtained; Get the query statement, search for text features similar to the query statement in the trained metric space, and take the video clip corresponding to the text feature with the highest similarity as the video positioning result.

2. A weakly supervised video segment localization method based on a large-scale video corpus as claimed in claim 1, Features: The obtained video features and text features are fused. The fusion method is to add and dot-product the video features and text features and then concatenate them.

3. The weakly supervised video segment localization method based on a large-scale video corpus as claimed in claim 1, It is characterized in that Searching for text features similar to the query statement in the metric space is implemented through hash binary codes, including the following steps: The text features in the metric space are converted into hash binary codes through a hash mapping function, and the hash binary codes are used as query indexes for text-video query pairs; The obtained query statements are matched with a set number of hash binary codes first; The video clip corresponding to the hash binary code with the highest matching degree is the positioning result.

4. A weakly supervised video segment localization system based on a large-scale video corpus, implemented by using the weakly supervised video segment localization method based on a large-scale video corpus as claimed in claim 1, It is characterized in that include: Common semantic perception module: configured to extract common semantic information between text and video using self-supervised learning for the acquired training data set, and obtain fused semantic video features based on the semantic information; The spatial mapping module of video features and text features is configured to perform multi-scale contrastive learning using a weakly supervised method for fusion semantic video features and corresponding text features, determine the spatial mapping relationship between video features and text features and map them to the metric space to obtain the trained metric space; Matching module: It is configured to obtain a query statement, search for text features similar to the query statement in the metric space, and query the video clip with the highest similarity to the text features in the trained metric space based on the text features to obtain the video positioning result.

5. An electronic device, It is characterized in that The invention comprises a memory and a processor and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps of a weakly supervised video segment localization method based on a large-scale video corpus as described in any one of claims 1 to 3 are completed.

6. A computer-readable storage medium, It is characterized in that Used to store computer instructions, which, when executed by a processor, complete the steps of a weakly supervised video segment localization method based on a large-scale video corpus as described in any one of claims 1 to 3.

Citation Information

Patent Citations

  • Video positioning method and device, and model training method and equipment

    CN113590874A

  • Method and device for positioning language segments in video based on comparative learning

    CN113792594A

  • Video time sequence behavior detection method based on weak supervised learning

    CN114359790A

  • Weak supervision video clip positioning method and system based on large-scale video corpus

    CN116226443A

  • Weakly Supervised Action Selection Learning in Video

    US20220335718A1

Cited By

  • Long-time activity analysis method and system based on hierarchical predefined knowledge learning

    CN119293537A

  • Deep learning-based quick retrieval method for literatures in smart library

    CN119622003A

  • Multimodal method for real-time high-fidelity video transmission based on semantic stream

    CN119728994A

  • Cantonese-mandarin aphasia automatic evaluation method based on transfer learning

    CN119832941A

  • Multi-branch collaborative network-based uncertainty perception pulmonary nodule segmentation method

    CN121095273A