Unsupervised video clip retrieval method based on time sequence anchor point mining and semantic alignment

Abstract

The invention discloses an unsupervised video clip retrieval method based on time sequence anchor point mining and semantic alignment, which comprises a video retrieval training system based on time sequence anchor point mining and point supervised learning, and the system comprises a key anchor point extraction module, a semantic alignment description generation module and a point supervised learning enhancement module. Key time anchor points are extracted from an original unlabeled video sequence through a key anchor point extraction module, then a pseudo-label triple is constructed through a semantic alignment description generation module, a weak supervision video clip retrieval model is trained accordingly, and in the training process, the pseudo-label triple is extracted from the original unlabeled video sequence. Constructing a point supervised contrast learning target through a point supervised learning enhancement module to further optimize the model, so that the finally trained weak supervised video clip retrieval model outputs a clip starting and ending time boundary related to query statement semantics; the problems of low pseudo tag quality, fuzzy positioning boundary and inconsistent semantic space in an unsupervised scene are effectively solved, and the positioning precision and generalization ability of video clip retrieval are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and artificial intelligence technology, and in particular relates to an unsupervised video segment retrieval method based on temporal anchor mining and semantic alignment. Background Technology

[0002] Video Moment Retrieval (VMR) aims to accurately locate specific temporal segments within untrimmed long videos based on natural language queries. This capability lays the technological foundation for tasks requiring fine-grained video understanding, such as automated monitoring and analysis, intelligent video search, and human-computer interaction. By associating linguistic concepts with temporal intervals, VMR enables computer systems to efficiently navigate and parse massive amounts of unstructured video data. Traditional video moment retrieval methods heavily rely on large-scale datasets with precise temporal annotations. However, precisely labeling the start and end boundaries for each "video-query" is not only extremely costly in terms of manpower but also difficult to scale in large-scale scenarios. Unlike image-text pairs that can be massively scraped from the internet, the high annotation cost of video temporal localization severely limits the generalization ability and application of supervised learning models.

[0003] To overcome the annotation bottleneck, research focus has gradually shifted to Zero-Shot VideoMoment Retrieval (ZSVMR). This paradigm aims to work directly on unlabeled raw videos, leveraging the extensive knowledge inherent in pre-trained visual language models (VLMs) to establish temporal-semantic alignment, thereby eliminating reliance on manual annotation. Under unsupervised training, the biggest challenge lies in establishing fine-grained "video-text" associations from unlabeled videos. Currently, there are two main unsupervised implementation paths in academia and industry. The first path is a fully supervised training paradigm based on pseudo-temporal boundary generation, which attempts to directly generate pseudo-labels with temporal start and end boundaries using a large multimodal model, and then uses these to train the retrieval model in a fully supervised manner. However, this approach suffers from significant technical bottlenecks. Since existing multimodal large models are not natively designed for long-video temporal localization tasks, the pseudo-boundaries they generate are often noisy and extremely inaccurate. This causes downstream retrieval models to not only learn incorrect boundary distributions but also have their performance limited by the zero-shot localization capabilities of pre-trained models. In some cases, the trained model performs worse than the original model in direct inference. The second approach is a training-free prediction paradigm based on visual language models. This method abandons model training and directly uses a large language model to calculate the similarity of video frames for the test query, combined with complex post-processing logic to extract segments. However, this method struggles to meet real-time requirements in practical applications. The computational load for frame-by-frame or high-frequency sliding window inference on long videos is enormous, resulting in extremely high inference latency. Furthermore, the similarity curves output by visual language models are often noisy due to visual information redundancy, heavily relying on manually designed screening thresholds and post-processing rules, and lacking robustness in various scenarios. Furthermore, existing unsupervised methods generally face the semantic gap problem. The original video descriptions generated by the model in an unconstrained state often have problems such as grammatical errors, use of obscure words, or redundant visual details. This is significantly different from the concise and standardized declarative query statements used by users in actual retrieval, which further exacerbates the difficulty of video-text semantic alignment in unsupervised training.

[0004] Given the inaccuracies caused by forcibly predicting time boundaries in existing technologies, and the high computational costs of direct inference, how to avoid inaccurate segment boundary prediction and instead utilize the image description capabilities of multimodal models to mine high-quality time anchors, and use these to construct high-precision point-level supervision information to enhance the boundary perception capability of weakly supervised models, has become a key technical path to improve the performance and practical value of unsupervised video segment retrieval. Summary of the Invention

[0005] (1) Technical problems to be solved This invention provides an unsupervised video segment retrieval method based on temporal anchor mining and semantic alignment, aiming to solve the problems of severe pseudo-boundary noise, semantic alignment distortion, and computational redundancy and lack of robustness in the inference stage of existing unsupervised methods.

[0006] (2) Technical solution This invention provides an unsupervised video segment retrieval method based on temporal anchor mining and semantic alignment, comprising the following steps: Step 1: Construct a video retrieval training system based on temporal anchor mining and point-supervised learning. This system includes a key anchor extraction module, a semantic alignment description generation module, and a point-supervised learning enhancement module. Step 2: Input the original video sequence, analyze the temporal saliency of the video frames through the key anchor point extraction module and generate the score curve of the key frame, and then extract the peak frame of the score curve as the key time anchor point. Step 3: The semantic alignment description generation module generates the original action description based on the key time anchor points and their context frame sequences using a visual language model. Then, the large language model is used for semantic correction and rewriting to obtain the standard query statement. Pseudo-label triples are constructed to train the weakly supervised video segment retrieval model. Step 4: Construct a point-supervised contrastive learning objective through the point-supervised learning enhancement module to provide additional point-level supervision information for the weakly supervised video segment retrieval model; Step 5: Input the video sequence to be queried and the user's query statement into the trained weakly supervised video segment retrieval model. Perform cross-modal semantic matching through the trained weakly supervised video segment retrieval model and output the start and end time boundaries of the segments in the video sequence to be queried that are semantically related to the query statement.

[0007] Furthermore, the specific steps for extracting key time anchors using the key anchor extraction module in step 2 are as follows: Step 201: Construct positive cue words to represent dynamic significant events and negative cue words to represent static background environments, forming logically contradictory judgment text pairs; Step 202: Input the video sequence frame by frame into the visual language model, obtain the response probability of each frame to the positive prompt word and the negative prompt word, and record them as positive confidence score and negative confidence score respectively. Step 203: Calculate the final keyframe score of a single frame by the difference between the positive confidence score and the negative confidence score, and obtain the temporal score curve that changes with video time. Step 204: Perform local maxima filtering on the time-series score curve, and determine the video frames that meet the preset threshold and are located at the peak of the time-series score curve as the key time anchor points.

[0008] Furthermore, the design criteria and processing details for the positive confidence score and the negative confidence score in step 202 include the following steps: Step 20201: Input the video sequence to be queried and the positive prompt words into the visual language model; Step 20202: Perform the forward inference process of the visual language model. After receiving the video sequence, the visual language model uses tensor slicing indexing to locate the unnormalized probability distribution vector corresponding to the end of the positive prompt word in the output sequence of the visual language model, and retrieves the value of the first tag "Yes" representing affirmative semantics from the unnormalized probability distribution vector. The numerical value of the second marker "No" representing negation. ; Step 20203, for the value of the first marker "Yes" The value of the second marker "No" Perform Softmax normalization for binary classification to calculate the positive response probability of the current video frame under the current positive cue word. and the positive response probability As the corresponding positive confidence score; The probability of a positive response The calculation formula is as follows:

[0009] in, This indicates that for each element in the input unnormalized probability distribution vector Logits, an operation is performed using the natural constant. The exponential operation with base 0 is equivalent to performing Softmax normalization on Logits. This represents the value of the first marker "Yes" in the probability distribution vector Logits. This represents the value of the second marker "No" in the probability distribution vector Logits. This represents the probability of a positive response for the current video frame under the current positive prompt word. Step 20204: Input the video sequence to be queried and the negative prompt words into the visual language model, process it according to the same logic described in steps 20202 to 20203, obtain the negative response probability representing that the current video frame is in a static background environment, and determine the negative response probability as the corresponding negative confidence score.

[0010] Furthermore, the positive prompt word in step 201 includes an analysis instruction on visual changes in the video sequence, which is used to guide the visual language model to focus on the visual change features in the video frame and to perform a Boolean judgment on whether the video frame has a semantically meaningful action or motion. The negative prompt words contain analysis instructions for the static attributes of the video sequence, which are used to guide the visual language model to analyze the visual static attributes of the scene and perform Boolean judgments on whether the video frame is a background environment without active events.

[0011] Furthermore, the specific steps for constructing pseudo-label triples through the semantic alignment description generation module in step 3 are as follows: Step 301: Using the key time anchor point as the center, expand the preset time radius in both directions forward and backward, and sample the expanded video interval at equal intervals to form a time sequence of images representing the evolution of the action. Step 302: Input the time-series image sequence into the visual language model and configure strong constraint description instructions to guide the visual language model to output a single sentence original action description that conforms to the preset grammar template; Step 303: Input the original action description into the large language model, use the large language model to perform semantic correction and rewriting on the original action description, and generate a standard query statement that is semantically equivalent and expresses standardly. Step 304: Associate and map the generated standard query statement, the corresponding original video sequence, and the key time anchor points to complete the construction of the pseudo-label triplet.

[0012] Furthermore, the specific processing steps for the point-level supervision information provided by the point-supervised learning enhancement module in step 4 are as follows: Step 401: The video sequence is encoded into a frame-by-frame visual feature sequence through a weakly supervised video segment retrieval model, and cross-modal semantic matching is completed by combining the semantic features of the standard query statement to predict and generate multiple candidate time intervals. Step 402: Use key time anchors to determine the candidate time intervals; if the candidate time interval covers the key time anchor on the time axis, then mark it as a positive sample interval; if the candidate time interval does not cover the key time anchor, then mark it as a negative sample interval. Step 403: Aggregate the frame-by-frame visual features corresponding to the positive sample interval and the negative sample interval to generate interval-level features, and construct a point-supervised contrastive learning target based on these features; Step 404: Jointly optimize the point-supervised contrastive learning objective and the supervision objective of the weakly supervised video segment retrieval model, calculate the joint loss function, and train the weakly supervised video segment retrieval model by minimizing the joint loss function.

[0013] Furthermore, the point-supervised learning enhancement module in step 403 constructs the point-supervised contrastive learning objective by including the following steps: Step 40301: Extract the frame visual features corresponding to the key time anchor points from the frame-by-frame visual feature sequence, and use them as the reference anchor point representation. Step 40302: Based on all positive and negative sample intervals determined in step 402, extract the frame-by-frame visual features corresponding to all video frames contained in their time intervals, and calculate the average value of all frame-by-frame visual features in their time intervals through mean pooling, which is used as the interval-level feature of the sample interval. Step 40303: Pair all positive sample intervals with all negative sample intervals to construct a comparison sample pair that covers all positive sample intervals and all negative sample intervals. Step 40304: For each pair of comparison samples, calculate the cosine similarity between the positive sample interval features and the baseline anchor point representation. And the cosine similarity between the negative sample interval features and the baseline anchor point representation. ; Step 40305: Calculate the contrast loss function based on the cosine similarity from step 40304. The contrastive loss function The calculation formula is as follows:

[0014] in, To compare the loss function values, This indicates taking the maximum value. This is the preset interval parameter; The final point-supervised contrastive learning objective is obtained by averaging the contrastive loss values ​​generated from all contrastive sample pairs.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. By using the key anchor point extraction module, logically contradictory judgment text pairs are constructed and the temporal saliency of video frames is analyzed to generate a score curve that reflects the dynamic changes of the video. Then, the peak of the score curve is accurately locked as the key time anchor point, realizing the effective extraction from redundant original video sequences to a high-confidence temporal benchmark. 2. Through the semantic alignment description generation module, the original action description is generated under the guidance of strong constraint instructions. Combined with the large language model, semantic correction and rewriting are performed to generate semantically equivalent and standardized query statements. A pseudo-label triplet of "video-aligned query-time anchor" is constructed, which effectively bridges the semantic gap between automatically generated text and actual retrieval needs. 3. A point-supervised contrastive learning objective based on the benchmark anchor point representation was constructed through the point-supervised learning enhancement module. The frame features of the anchor point position were extracted as the benchmark, and the predicted positive sample intervals and negative sample intervals were paired one by one. The cosine similarity between the features of each interval level and the benchmark anchor point representation was calculated. The contrastive loss function was used to constrain the model to bring the positive sample interval closer and push the negative sample interval further away in the feature space, which significantly enhanced the model's ability to perceive the start and end boundaries of actions. 4. By jointly optimizing and training the point-supervised contrastive learning objective with the weakly supervised video segment retrieval model's own supervised objective, the model achieves both global semantic matching and local temporal precise localization capabilities without the need for manual annotation, thus realizing a systematic improvement in retrieval performance from point-level supervised signals to interval-level retrieval. Attached Figure Description

[0016] Figure 1 A flowchart illustrating the implementation method of the present invention is shown.

[0017] Figure 2 A schematic diagram of the operation of each module in an embodiment of the present invention is shown.

[0018] Figure 3 A schematic diagram illustrating the process by which the point-supervised learning enhancement module of the present invention provides point-level supervision information is shown. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] For the sake of clarity and reference, the technical terms, abbreviations, or acronyms used below are summarized and explained as follows: Logits: Unnormalized probability distribution vector.

[0021] Softmax is a normalization function that transforms an unnormalized probability distribution vector (Logits) into a probability distribution map. It works by applying a normalization function to each element of the input unnormalized probability distribution vector, using the natural constant. Perform exponential operations with base 0 and divide by the sum of the exponents of all elements to compress the output value to 0. Within the interval, and ensure that the algebraic sum of all output elements is 1.

[0022] VLM: Visual Language Model.

[0023] Stride: Stride refers to the number of frame intervals between two adjacent sampling frames when sampling in a video frame sequence. It is used to control the sparsity of sampling, that is, the number of frames skipped between adjacent sampling frames.

[0024] like Figure 1-2 As shown, Figure 1 The flowchart of the implementation method of the present invention is shown. The present invention discloses an unsupervised video segment retrieval method based on temporal anchor point mining and semantic alignment, including the following steps: Step 1: Construct a video retrieval training system based on temporal anchor mining and point-supervised learning. This system includes a key anchor extraction module, a semantic alignment description generation module, and a point-supervised learning enhancement module. Step 2: Using the original unlabeled video sequence as input, the temporal saliency of the video frames is analyzed by the key anchor extraction module and the score curve of the key frame is generated. Then, the peak frame of the score curve is extracted as the key time anchor, thus realizing the effective extraction from the redundant original video sequence to the high confidence temporal benchmark. Step 3: The semantic alignment description generation module generates the original action description based on the key time anchors and their context frame sequences using a visual language model. Then, the large language model is used for semantic correction and rewriting to generate a standard query statement that is semantically equivalent and expresses in a standardized manner. A pseudo-label triplet of "video sequence - standard query statement - key time anchors" is constructed, which effectively bridges the semantic gap between the automatically generated text and the actual retrieval requirements. In this invention, the visual language model is preferably Qwen3-VL, and the large language model is preferably Qwen3.

[0025] Step 4: Train a weakly supervised video clip retrieval model using the pseudo-label triplet constructed in Step 3. During the training process, the weakly supervised video clip retrieval model can predict relevant candidate time intervals. Then, the point-supervised learning enhancement module filters the candidate time intervals predicted by the weakly supervised video clip retrieval model based on key time anchors and constructs a point-supervised contrastive learning objective to provide additional point-level supervision information for the weakly supervised video clip retrieval model. By jointly optimizing and training the point-supervised contrastive learning objective with the supervised objective inherent in the weakly supervised video clip retrieval model, the weakly supervised video clip retrieval model is equipped with both global semantic matching and local temporal precise localization capabilities without the need for manual annotation. This achieves a systematic improvement in retrieval performance from point-level supervised information to interval-level retrieval.

[0026] Step 5: Input the video sequence to be queried and the user's query statement into the trained weakly supervised video segment retrieval model, and use the trained weakly supervised video segment retrieval model to perform cross-modal semantic matching, thereby outputting the start and end time boundaries of segments in the video sequence to be queried that are semantically related to the query statement.

[0027] Specifically, the steps for extracting key time anchors using the key anchor extraction module in step 2 are as follows: Step 201: Construct positive cue words to represent dynamic significant events and negative cue words to represent static background environments, forming logically contradictory judgment text pairs; The positive prompts include analytical instructions for visual changes in the video sequence, such as the amplitude of movement, direction of motion, and frequency of scene changes. These instructions guide the visual language model to focus on visual change features in video frames, thereby enhancing the model's sensitivity to complex dynamic events. Furthermore, they perform Boolean judgments on whether semantically meaningful actions or movements exist in the video frames. The specific details are as follows: Positive prompt words="Analyze this frame sequence. Focus on visual changes.Does this sequence capture a significant dynamic event or a specific action?Answer Yes for meaningful movement. Answer No for static scenes". The negative prompts contain analysis instructions for the static attributes of the video sequence, such as lighting stability, object stillness, and scene consistency. These instructions guide the visual language model to analyze the visual static attributes of the scene, thereby improving the model's ability to filter non-keyframes. Finally, a Boolean judgment is performed to determine whether a video frame represents a background environment without active events. The specific details are as follows: Negative prompt words="Analyze this frame sequence. Is this scene visuallystatic, representing a background environment with no active events? AnswerYes for empty or motionless scenes. Answer No if action is present". Step 202, change the length to T Video sequence of frames VThe visual language model is input frame by frame, and the response probabilities for each frame to the positive and negative prompt words are calculated and recorded as positive confidence scores and negative confidence scores, respectively.

[0028] In this embodiment, the length T The length is 200 frames. In practical applications, this length... T Based on video sequence V The specific duration and frame rate are dynamically adjusted.

[0029] Specifically, the design criteria and processing details for the positive confidence score and the negative confidence score include the following steps: Step 20201: Input the video sequence to be queried and the positive prompt words into the visual language model; Step 20202: Perform the forward inference process of the visual language model. The visual language model receives a length of... T After the video sequence is processed, a hidden state corresponding to the position is generated in its top-level linear decoding layer. This embodiment uses tensor slice indexing to lock the unnormalized probability distribution vector Logits corresponding to the end of the positive cue word in the visual language model output sequence, and retrieves the value of the first tag "Yes" representing affirmative semantics from this unnormalized probability distribution vector Logits. The numerical value of the second marker "No" representing negation. ; Step 20203, for the value of the first marker "Yes" The value of the second marker "No" Perform Softmax normalization for binary classification to calculate the positive response probability of the current video frame under the current positive cue word. The probability of the affirmative response The calculation formula is as follows:

[0030] in, This indicates that for each element in the input unnormalized probability distribution vector Logits, an operation is performed using the natural constant. The exponential operation with base 0 is equivalent to performing Softmax normalization on Logits. This represents the value of the first marker "Yes" in the probability distribution vector Logits. This represents the value of the second marker "No" in the probability distribution vector Logits. This represents the probability of a positive response for the current video frame under the current positive prompt word.

[0031] For example, calculate the first Frame (0 < ≤ T The probability of a positive response Then, the first step needs to be obtained according to the above steps. The value corresponding to "Yes" in the frame. and the corresponding value for "No" Then adjust the values and Perform Softmax normalization to obtain the first... Positive response probability of a frame and the first Positive response probability of a frame Marked as the corresponding number Positive confidence score of the frame .

[0032] Similarly, the steps for calculating the negative confidence score are as follows: Step 20204: Input the video sequence to be queried and the negative prompt words into the visual language model, and process them according to the same logic described in steps 20202 to 20203 above, thereby obtaining the representative of the current [video sequence]. The probability of a negative response when a video frame is in a static background environment. and the obtained negative response probability As the corresponding negative confidence score ; Indicates the first Negative confidence score of the frame, and 0 < ≤ T。

[0033] Step 203, based on the results obtained from the above steps... Positive confidence score of the frame and negative confidence score The difference between the two is calculated to obtain the final keyframe score for a single frame. This yields the temporal score curve that varies with video time, and the keyframe score. The calculation formula is as follows:

[0034] Step 204: Perform local maxima filtering on the time-series score curve to select those that meet the keyframe score requirements. ( Video frames that are at the peak of the time-series score curve (with a preset threshold) are identified as key time anchor points, denoted as . .

[0035] In this embodiment, =0.5.

[0036] Furthermore, the specific steps for constructing the pseudo-label triples through the semantic alignment description generation module in step 3 are as follows: Step 301, using the key time anchor point Centered on the preset time radius, expand forward and backward in both directions. And the expanded video range is divided into steps. k Equal-interval sampling is performed, and the images are stitched together to form a temporal image sequence representing the evolution of the action; Among them, step size k This indicates the number of frame intervals between two adjacent samplings when sampling in a video frame sequence. k This is used to control the sparsity of sampling, i.e., the number of frames skipped between adjacent sampling frames.

[0037] In this embodiment, the time radius =1 frame.

[0038] Step 302: Input the time-series image sequence into the visual language model and configure strong constraint description instructions to guide the visual language model to output a single sentence of original action description that conforms to a preset grammar template. The strong constraint description instruction restricts the output text to begin with a fixed subject, contains only a single core action predicate, and prohibits the use of speculative adverbs. Step 303, describe the original action. Input a large language model, and use the large language model to describe the original action. Semantic correction and rewriting are performed; specifically, this includes correcting grammatical errors, replacing obscure words with common action terms, and transforming fragmented descriptions into standard statements that conform to search habits, generating semantically equivalent and well-expressed standard query statements. ; Step 304, generate the standard query statement The corresponding original video sequence and the key time anchor points The association mapping is constructed into pseudo-label triples of "video sequence - standard query statement - key time anchor" for subsequent training of the weakly supervised video segment retrieval model. That is, pseudo-label triples .

[0039] Furthermore, such as Figure 3 As shown, the specific processing steps of the point-supervised learning enhancement module in step 4 to provide point-level supervision information for the weakly supervised video segment retrieval model are as follows: Step 401: Encode the video sequence into a frame-by-frame visual feature sequence using a weakly supervised video segment retrieval model. Its dimensions are T×D ,in, T For video length, D This is based on video feature dimensions, combined with standard query statements. Semantic features are used to perform cross-modal semantic matching, ultimately predicting and generating multiple candidate time intervals. This provides a foundation for subsequent interval selection and fine-tuning; The candidate time interval The formula is expressed as follows: ; in This represents the set of candidate time intervals predicted by a weakly supervised video segment retrieval model, which contains... One video clip that may be semantically related to the query statement; This represents the index number of the candidate interval, and its value range is... , Indicates the first The starting time point of each candidate interval on the video timeline. Indicates the first The termination time point of each candidate interval on the video timeline.

[0040] Step 402: Use the key time anchors in the pseudo-label triples to determine the candidate time intervals; if the candidate time interval covers the key time anchors on the time axis... If the candidate time interval does not cover the key time anchor point, then mark it as a positive sample interval; If so, then mark it as a negative sample interval; Step 403: Aggregate the frame-by-frame visual features corresponding to the positive sample interval and the negative sample interval to generate interval-level features, and construct a point-supervised contrastive learning objective based on these features. The specific process includes: Step 40301, from the frame-by-frame visual feature sequence Extract the key time anchor points The visual features of the corresponding frame are used as the reference anchor point representation. ,Right now ; in Indicates the first Visual features of a frame.

[0041] Step 40302: Based on all positive and negative sample intervals obtained in step 402, extract the frame-by-frame visual features corresponding to all video frames contained in the time interval. Calculate the average value of all frame-by-frame visual features within the time interval using average pooling, and use this average value as the interval-level feature of that sample interval. The interval-level features The calculation formula is as follows:

[0042] in, This represents the calculated interval-level feature representation. This indicates the starting frame index of the currently processed sample interval. This indicates the end frame index of the currently processed sample interval. Indicates the first Visual features of a frame This represents the summation of the frame-by-frame visual features corresponding to all video frames.

[0043] Step 40303: Pair all positive sample intervals with all negative sample intervals to construct contrast sample pairs covering all positive and negative sample intervals; Step 40304: For each pair of comparison samples, calculate the cosine similarity between the positive sample interval features and the baseline anchor point representation. And the cosine similarity between the negative sample interval features and the baseline anchor point representation. ; Step 40305: Calculate the contrast loss function based on the cosine similarity mentioned above. The contrast loss The calculation formula is as follows:

[0044] in, To compare the loss function values, This indicates taking the maximum value. In this embodiment, the preset interval parameter is used. The value is 0.15; The final point-supervised contrastive learning objective is obtained by averaging the contrastive loss values ​​generated from all contrastive sample pairs. .

[0045] Step 404, the point-supervised contrastive learning target The supervised objectives of weakly supervised video segment retrieval models Perform joint optimization, construct and compute the joint loss function. The joint loss function The calculation formula is as follows:

[0046] in The loss adjustment coefficient is 1 in this embodiment. By minimizing this joint loss function, the weakly supervised video segment retrieval model is trained to optimize the feature extraction and temporal anchor point mining capabilities of the weakly supervised video segment retrieval model. This significantly enhances the model's ability to perceive the start and end boundaries of actions, solving the technical pain points of blurred boundaries and inaccurate positioning in weakly supervised scenarios.

[0047] Step 5: Input the video sequence to be queried and the user's query into the trained weakly supervised video segment retrieval model. Perform cross-modal semantic matching through the trained weakly supervised video segment retrieval model, and output the start and end time boundaries of segments in the video sequence that are semantically related to the query. , in, This represents the starting frame index of the time boundary predicted by the weakly supervised video segment retrieval model.

[0048] This represents the index of the end frame of the time boundary predicted by the weakly supervised video segment retrieval model.

[0049] This invention trains a weakly supervised video clip retrieval model using a key anchor point extraction module, a semantic alignment description generation module, and a point-supervised learning enhancement module. This effectively overcomes the problems of low quality pseudo-labels, ambiguous positioning boundaries, and inconsistent semantic space in unsupervised scenarios, and significantly improves the positioning accuracy and generalization ability of video clip retrieval.

[0050] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0051] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0052] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0053] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An unsupervised video segment retrieval method based on temporal anchor point mining and semantic alignment, characterized in that, Includes the following steps: Step 1: Construct a video retrieval training system based on temporal anchor mining and point-supervised learning. This system includes a key anchor extraction module, a semantic alignment description generation module, and a point-supervised learning enhancement module. Step 2: Input the original video sequence, analyze the temporal saliency of the video frames through the key anchor point extraction module and generate the score curve of the key frame, and then extract the peak frame of the score curve as the key time anchor point. Step 3: The semantic alignment description generation module generates the original action description based on the key time anchor points and their context frame sequences using a visual language model. Then, the large language model is used for semantic correction and rewriting to obtain the standard query statement. Pseudo-label triples are constructed to train the weakly supervised video segment retrieval model. Step 4: Construct a point-supervised contrastive learning objective through the point-supervised learning enhancement module to provide additional point-level supervision information for the weakly supervised video segment retrieval model; Step 5: Input the video sequence to be queried and the user's query statement into the trained weakly supervised video segment retrieval model. Perform cross-modal semantic matching through the trained weakly supervised video segment retrieval model and output the start and end time boundaries of the segments in the video sequence to be queried that are semantically related to the query statement.

2. The unsupervised video segment retrieval method based on temporal anchor mining and semantic alignment according to claim 1, characterized in that, The specific steps for extracting key time anchors using the key anchor point extraction module in step 2 are as follows: Step 201: Construct positive cue words to represent dynamic significant events and negative cue words to represent static background environments, forming logically contradictory judgment text pairs; Step 202: Input the video sequence frame by frame into the visual language model, obtain the response probability of each frame to the positive prompt word and the negative prompt word, and record them as positive confidence score and negative confidence score respectively. Step 203: Calculate the final keyframe score of a single frame by the difference between the positive confidence score and the negative confidence score, and obtain the temporal score curve that changes with video time. Step 204: Perform local maxima filtering on the time-series score curve, and determine the video frames that meet the preset threshold and are located at the peak of the time-series score curve as the key time anchor points.

3. The unsupervised video segment retrieval method based on temporal anchor point mining and semantic alignment according to claim 2, characterized in that, The design criteria and processing details for the positive confidence score and the negative confidence score in step 202 include the following steps: Step 20201: Input the video sequence to be queried and the positive prompt words into the visual language model; Step 20202: Perform the forward inference process of the visual language model. After receiving the video sequence, the visual language model uses tensor slicing indexing to lock the unnormalized probability distribution vector corresponding to the end of the positive prompt word in the output sequence of the visual language model, and retrieves the value of the first tag "Yes" representing the affirmative semantics from the unnormalized probability distribution vector. The numerical value of the second marker "No" representing negation. ; Step 20203, for the value of the first marker "Yes" The value of the second marker "No" Perform Softmax normalization for binary classification to calculate the positive response probability of the current video frame under the current positive cue word. and the positive response probability As the corresponding positive confidence score; The probability of a positive response The calculation formula is as follows: ； in, This indicates that for each element in the input unnormalized probability distribution vector Logits, an operation is performed using the natural constant. The exponential operation with base 0 is equivalent to performing Softmax normalization on Logits. This represents the value of the first marker "Yes" in the probability distribution vector Logits. This represents the value of the second marker "No" in the probability distribution vector Logits. This represents the probability of a positive response for the current video frame under the current positive prompt word. Step 20204: Input the video sequence to be queried and the negative prompt words into the visual language model, process it according to the same logic described in steps 20202 to 20203, obtain the negative response probability representing that the current video frame is in a static background environment, and determine the negative response probability as the corresponding negative confidence score.

4. The unsupervised video segment retrieval method based on temporal anchor point mining and semantic alignment according to claim 3, characterized in that, The positive prompt words in step 201 contain analysis instructions for visual changes in the video sequence, which are used to guide the visual language model to focus on the visual change features in the video frame and to perform Boolean judgments on whether the video frame has semantically meaningful actions or movements. The negative prompt words contain analysis instructions for the static attributes of the video sequence, which are used to guide the visual language model to analyze the visual static attributes of the scene and perform Boolean judgments on whether the video frame is a background environment without active events.

5. The unsupervised video segment retrieval method based on temporal anchor point mining and semantic alignment according to claim 1, characterized in that, The specific steps for constructing pseudo-label triples using the semantic alignment description generation module in step 3 are as follows: Step 301: Using the key time anchor point as the center, expand the preset time radius in both directions forward and backward, and sample the expanded video interval at equal intervals to form a time sequence of images representing the evolution of the action. Step 302: Input the time-series image sequence into the visual language model and configure strong constraint description instructions to guide the visual language model to output a single sentence original action description that conforms to the preset grammar template; Step 303: Input the original action description into the large language model, use the large language model to perform semantic correction and rewriting on the original action description, and generate a standard query statement that is semantically equivalent and expresses standardly. Step 304: Associate and map the generated standard query statement, the corresponding original video sequence, and the key time anchor points to complete the construction of the pseudo-label triplet.

6. The unsupervised video segment retrieval method based on temporal anchor point mining and semantic alignment according to claim 1, characterized in that, The specific processing steps for the point-level supervision information provided by the point-supervised learning enhancement module in step 4 are as follows: Step 401: The video sequence is encoded into a frame-by-frame visual feature sequence through a weakly supervised video segment retrieval model, and cross-modal semantic matching is completed by combining the semantic features of the standard query statement to predict and generate multiple candidate time intervals. Step 402: Use key time anchors to determine the candidate time intervals; if the candidate time interval covers the key time anchor on the time axis, then mark it as a positive sample interval; if the candidate time interval does not cover the key time anchor, then mark it as a negative sample interval. Step 403: Aggregate the frame-by-frame visual features corresponding to the positive sample interval and the negative sample interval to generate interval-level features, and construct a point-supervised contrastive learning target based on these features; Step 404: Jointly optimize the point-supervised contrastive learning objective and the supervision objective of the weakly supervised video segment retrieval model, calculate the joint loss function, and train the weakly supervised video segment retrieval model by minimizing the joint loss function.

7. The unsupervised video segment retrieval method based on temporal anchor point mining and semantic alignment according to claim 6, characterized in that, The point-supervised learning enhancement module in step 403 constructs the point-supervised contrastive learning objective through the following steps: Step 40301: Extract the frame visual features corresponding to the key time anchor points from the frame-by-frame visual feature sequence, and use them as the reference anchor point representation. Step 40302: Based on all positive and negative sample intervals determined in step 402, extract the frame-by-frame visual features corresponding to all video frames contained in their time intervals, and calculate the average value of all frame-by-frame visual features in their time intervals through mean pooling, which is used as the interval-level feature of the sample interval. Step 40303: Pair all positive sample intervals with all negative sample intervals to construct a comparison sample pair that covers all positive sample intervals and all negative sample intervals. Step 40304: For each pair of comparison samples, calculate the cosine similarity between the positive sample interval features and the baseline anchor point representation. And the cosine similarity between the negative sample interval features and the baseline anchor point representation. ; Step 40305: Calculate the contrast loss function based on the cosine similarity from step 40304. The contrastive loss function The calculation formula is as follows: ； in, To compare the loss function values, This indicates taking the maximum value. This is the preset interval parameter; The final point-supervised contrastive learning objective is obtained by averaging the contrastive loss values ​​generated from all contrastive sample pairs.

Images