Training-free video corpus time instant retrieval method based on adaptive calibration mechanism

Abstract

The present application relates to a training-free video corpus time moment retrieval method based on an adaptive calibration mechanism, and belongs to the technical field of computer vision and multi-modal information processing. It comprises: based on the constructed query event chain and video event chain, calculating the event level similarity score, and combining the mean-variance joint scoring mechanism for cross-modal retrieval to obtain candidate proposals; in the time positioning stage, a boundary level association is established among the video candidate proposals through a cooperative mechanism, and a profit-loss dynamic feedback strategy is used to perform adaptation and iteratively optimize the time boundary of the candidate proposals. The present application realizes a closed-loop retrieval process from text semantic analysis to video accurate matching. Compared with the prior art, the present application does not need large-scale labeled data for training or parameter updating, effectively solves the generalization bottleneck and deployment cost problem of video retrieval in open domain scenarios, and significantly improves the accuracy and robustness of time positioning under zero training conditions.

Description

Technical Field

[0001] This invention relates to a training-free video corpus moment retrieval method based on an adaptive calibration mechanism, belonging to the field of computer vision and multimodal information processing technology. Background Technology

[0002] With the rapid development of the internet and multimedia technologies, video data has experienced explosive growth. How to efficiently and accurately locate and retrieve video segments that match specific natural language queries from massive video corpora—a process known as video corpus moment retrieval—has become a crucial research topic in computer vision and multimodal information processing. This technology has wide-ranging applications in intelligent video editing, precise information retrieval, and long-form video content understanding. Most existing video moment retrieval methods rely on large-scale labeled data for fully supervised training or fine-tuning. However, in real-world open-domain applications, user queries exhibit high diversity and complexity, leading to severe domain bias and generalization bottlenecks in training models dependent on specific datasets. Furthermore, collecting and labeling massive amounts of fine-grained text-video segments requires enormous human and time resources. This strong dependence on training data significantly limits the large-scale deployment and practical application of existing retrieval methods.

[0003] To address these issues, researchers have begun exploring zero-shot or training-free video retrieval methods. However, existing training-free methods still have limitations. On one hand, most methods, when dealing with complex natural language queries, often employ holistic feature extraction, lacking in-depth analysis of the semantic structure and event dependencies within the query sentence. This leads to semantic ambiguity during cross-modal feature alignment. On the other hand, in the temporal boundary localization process, existing methods typically rely solely on coarse-grained segment-level similarity for simple truncation, making them susceptible to interference from locally high-scoring anomalies and lacking modeling of the global semantic coherence between candidate proposals. Furthermore, when faced with complex video streams, existing methods lack dynamic feedback and boundary optimization mechanisms, making it difficult to fine-grainedly adjust candidate segments based on cross-modal interaction results, thus resulting in insufficient accuracy and robustness in temporal localization.

[0004] Therefore, how to construct a video corpus moment retrieval method that does not require large-scale labeled data training, can deeply analyze the structured semantics of text and video, and has adaptive boundary calibration capabilities has become a technical problem that urgently needs to be solved in the field of multimodal information processing. Summary of the Invention

[0005] To overcome the problems of weak generalization ability, high training and deployment costs, and insufficient temporal boundary positioning accuracy of existing video corpus time-of-flight retrieval methods in open domain scenarios, this invention proposes a training-free video corpus time-of-flight retrieval method based on an adaptive calibration mechanism, so as to achieve more accurate and efficient cross-modal semantic matching and temporal positioning between complex natural language queries and video segments.

[0006] The technical solution of this invention is: a training-free video corpus moment retrieval method based on an adaptive calibration mechanism, the method comprising:

[0007] Step 1: Input the natural language text query data into the large language model for semantic parsing and structural reconstruction; convert the text sentence structure into an ordered, structured query event chain;

[0008] Step 2: Perform automatic subtitle segmentation and keyframe aggregation on the target video corpus data; divide the continuous video data stream into multiple semantically coherent event units, thereby mapping and representing the semantic stream of the video as a structured video event chain;

[0009] Step 3: Use a vision-language pre-trained network model to extract multimodal features from natural language text queries, query event chains, input videos, and video event chains to construct a unified feature representation across modalities;

[0010] Step 4: Calculate the event-level similarity score between the query and the query event chain and the video event chain; integrate the mean and variance of the event-level similarity scores, and perform cross-modal feature matching through the mean-variance joint scoring mechanism to obtain the overall retrieval score;

[0011] Step 5: For the initially selected video candidate proposals, the sliding window method is used to generate two sets of time segment proposals based on the similarity scores between query features and video features, and the similarity scores between query event chain features and video features, respectively.

[0012] Step 6: Perform interval overlap analysis on the two sets of time segment proposals generated from the same video candidate proposal; determine whether there is an intersection of time intervals between the time segment proposals based on text queries and the time segment proposals based on query event chains;

[0013] Step 7: In the temporal boundary localization stage, a collaborative calibration mechanism is introduced. Based on the interval intersection judgment result, boundary-level associations between video candidate proposals are established, and combined with the gain-loss dynamic feedback strategy, the temporal boundary of the candidate proposals is adaptively and iteratively optimized without updating any model parameters.

[0014] Step 8: Compare and select the optimal time boundary based on the dynamically optimized proposal scores; combine the score corresponding to the optimal time boundary with the overall retrieval score in Step 4 to output the final video time-lapse retrieval and precise positioning results.

[0015] Further, Step 1 includes: for a given natural language text query ,in Representing words, To represent the number of words, the large language model LLaMA 3.1-8B is first used to decompose it into a series of sub-events and then reorder them into a structured query event chain. The process of generating the structured query event chain through the large language model is represented as follows:

[0016] ;

[0017] in, This represents the LLaMA 3.1-8B large language model. Clue words indicating a large model, Indicates the first A textual description of the event. Indicates the number of event chains generated.

[0018] Furthermore, Step 2 specifically includes: for a given video ,in Represents a video frame. To represent the number of video frames, the visual language model BLIP is first used to generate its subtitle sequence. Then, similarity scores are calculated between the subtitle sequences. Video segments associated with the same subtitle are considered independent events, thus forming... Each event unit is an event unit; the core semantic features of each event are preserved, and the intermediate frames of these event units are further selected for aggregation to form a complete video event chain; the process of generating a video event chain through BLIP is represented as follows:

[0019] ;

[0020] ;

[0021] in, The prompt word indicating BLIP, Indicates the first The subtitle description corresponding to the frame, This represents the generated subtitle sequence. This represents the segmentation function, which divides the video... Divided into Each event unit is a separate event unit, and the intermediate frame of each event unit is further selected. Indicates the first event in the video event chain. One event.

[0022] Further, Step 3 specifically includes: using the BLIP model as the vision-language pre-trained network model, and performing feature encoding on natural language text queries, query event chains, input videos, and video event chains respectively, mapping them to a unified feature representation of a unified dimension; the construction process of the unified feature representation is expressed as follows:

[0023] ;

[0024] ;

[0025] ;

[0026] ;

[0027] in Sentence-level features representing text queries. Represents a natural language text query. Indicates the first event in the query event chain. The characteristics corresponding to each event Indicates the first A textual description of the event. Indicates the number of event chains generated. This represents the query event chain feature matrix. Indicates the video number The features corresponding to the frame Represents the video feature matrix. Represents a video frame. Indicates the number of video frames. Indicates the first event in the video event chain. The characteristics corresponding to each event Represents the feature matrix of the video event chain. Indicates the first event in the video event chain. One event.

[0028] Furthermore, Step 4 specifically includes the following steps:

[0029] First, calculate the cosine similarity between the query features and the video event chain features, and take the maximum value as the initial matching score between the query and the video;

[0030] Subsequently, regarding the first For each query event, the initial query event score at that event level is obtained by calculating the cosine similarity between its feature representation and the features of the video event chain and taking the maximum value.

[0031] Finally, the overall retrieval score is obtained by adding the mean of the initial query event scores to the initial matching scores of the query features and video event chain features, and then subtracting the variance of the initial query event scores; the mean-variance joint scoring mechanism is represented as follows:

[0032] ;

[0033] ;

[0034] ;

[0035] ;

[0036] in, This indicates the calculation of cosine similarity. This indicates selecting the maximum value. This represents the initial matching score between the query features and the video event chain features. Sentence-level features representing text queries. Indicates the first event in the video event chain. The characteristics corresponding to each event Indicates the first The initial matching score between the query event features and the video event chain features. Indicates the first event in the query event chain. The characteristics corresponding to each event Indicates the number of event chains generated. This represents the query event score vector. and This represents the mean and variance of the score for the event-level match, i.e., the score for the initial query event. This represents the overall search score.

[0037] Furthermore, Step 5 specifically includes the following steps:

[0038] Step 5.1: First, calculate the similarity score between the query feature and the video feature. Then, generate a series of video proposals using the sliding window method. The score of each proposal is calculated by subtracting the mean of the out-of-proposal scores from the mean of the in-proposal scores. The proposal with the highest score is selected as the final query proposal. The specific process is as follows:

[0039] ;

[0040] ;

[0041] ;

[0042]

[0043] in, This indicates the query similarity score. Sentence-level features representing text queries. Indicates the video number The features corresponding to the frame Represents the video feature matrix. Represents a video frame. Indicates the number of video frames. This represents the sliding window method. Indicates the generated first One suggestion, This represents the total number of proposals. This indicates the query similarity score. All included in the All scores in the proposals This indicates the query similarity score. All not included in the All scores in the proposals This indicates calculating the mean. It is a function defined to evaluate the quality of a proposal. Indicates the final query proposal;

[0044] Step 5.2: Calculate the similarity score between the query event chain features and video features, starting with the first... The similarity scores between video features and query event chain features are calculated, and the maximum value is taken as the score between the video feature and the query event chain. Finally, the final query event chain proposal is generated. The specific process is as follows:

[0045]

[0046]

[0047]

[0048]

[0049] in, Indicates the first Similarity score between the frame and the query event chain. Indicates the video number The features corresponding to the frame Indicates the first event in the query event chain. The characteristics corresponding to each event This represents the similarity score sequence of the query event chain. This represents the similarity score sequence for the query event chain. All included in the All scores in the proposals This represents the similarity score sequence for the query event chain. All not included in the All scores in the proposals This indicates the final query event chain proposal.

[0050] Furthermore, Step 6 specifically includes: for the final query proposal and the final query event chain proposal ,in and These represent the starting frame indices of the query proposal and the query event chain proposal, respectively. and These represent the end frame indices of the query proposal and the query event chain proposal, respectively. The final query proposal and the final query event chain proposal are analyzed to determine if there is any intersection. If there is an intersection, proceed to Step 7; otherwise, skip Step 7 and proceed to Step 8.

[0051] Furthermore, Step 7 specifically includes the following steps:

[0052] Step 7.1, consider satisfying Query suggestions and the final query event chain proposal Intersection Considered a potentially correct region for a given proposal First, a new proposal is constructed by replacing its ending frame. And will propose and suggestions The starting frames are moved to the left respectively Frame, get and ;

[0053] Step 7.2: Define the loss as the decrease in score due to interval contraction, and the gain as the score gain obtained on the corresponding proposal. The specific process is as follows:

[0054] ;

[0055] ;

[0056] in This indicates the query similarity score. This represents the similarity score sequence of the query event chain. It is a function defined to evaluate the quality of a proposal; if the gains exceed the losses, that is:

[0057]

[0058] If the adjustment is deemed effective, the proposal will be updated and repeated; if the loss outweighs the gain, then:

[0059]

[0060] Or when the iteration exceeds the intersection interval Stop immediately;

[0061] Step 7.3, Regarding the event chain proposal First, a new proposal is constructed by replacing its ending frame. And will propose and suggestions The starting frames are moved to the right respectively. Frame, get and The corresponding losses and gains are expressed as follows:

[0062] ;

[0063] ;

[0064] In response to the two proposals and After iterative optimization, the optimized proposal is denoted as... and ;

[0065] Furthermore, Step 8 specifically includes: based on the optimized proposal and Calculate their query similarity scores. Similarity score sequence with query event chain The overall matching score, if Then choose to propose. Otherwise choose Then, the overall matching score of the selected proposals will be compared with the overall search score obtained in Step 4. The scores are summed to obtain the final score for the proposal. The specific process is as follows:

[0066]

[0067] in Indicates the overall search score. It is a function defined to evaluate the quality of a proposal. This is the final score for the proposal.

[0068] The present invention also provides a training-free video corpus moment retrieval system based on an adaptive calibration mechanism, the system comprising: a module for executing the training-free video corpus moment retrieval method based on the adaptive calibration mechanism.

[0069] The beneficial effects of this invention are:

[0070] 1. Improve cross-modal semantic alignment accuracy and structured retrieval accuracy: This invention utilizes a large language model to parse natural language queries, constructs query event chains with semantic trajectories and dependencies, and combines automatic subtitle segmentation and keyframe aggregation to build video event chains, achieving a deep representation from unstructured data to structured events. This event-level bidirectional mapping effectively eliminates semantic ambiguity caused by complex long sentences, enabling precise alignment between text intent and video content at the fine-grained event level, significantly improving the accuracy of retrieval results;

[0071] 2. Overcoming the generalization bottleneck in open-domain scenarios and significantly reducing model deployment costs: This invention adopts a training-free architecture design, completely eliminating the strong dependence of traditional video retrieval methods on massive amounts of manually labeled "text-video clip" data. Without the need for time-consuming large-scale model training and parameter fine-tuning, it can be directly applied to diverse open-domain video streams, effectively avoiding the domain bias problem caused by training on specific datasets, and achieving truly zero-cost deployment and high generalization capabilities.

[0072] 3. Enhance global semantic coherence and effectively suppress local abnormal noise interference: This invention introduces a "mean-variance joint scoring mechanism" in the cross-modal retrieval stage, breaking the limitation of traditional methods that rely solely on a single similarity score for crude truncation. This mechanism can fully consider the scoring variance of contextual paragraphs while evaluating event-level similarity, thereby effectively suppressing abnormally high score interference caused by irrelevant redundant images or local visual abrupt changes, ensuring that the obtained candidate proposals have a high degree of global semantic consistency and logical coherence;

[0073] 4. Achieving Adaptive and Precise Temporal Boundary Positioning, Enhancing the Robustness of Time-Based Retrieval: This invention designs a boundary-level association and collaboration mechanism during the temporal positioning stage and innovatively introduces a "gain-loss dynamic feedback strategy" to perform adaptive calibration. This mechanism ensures that candidate video segments are no longer static segmentation results, but can undergo multiple rounds of dynamic fine-tuning and iterative convergence based on feedback from cross-modal matching. This adaptive calibration greatly improves the accuracy of video time-based extraction, making the final output temporal boundary more closely aligned with real human semantic cognition, and exhibiting strong robustness when facing complex and varied video corpora. Attached Figure Description

[0074] Figure 1This is a schematic diagram of the process structure of the method of the present invention;

[0075] Figure 2 This is a schematic diagram of the query event chain and video event chain generation process of the method of the present invention;

[0076] Figure 3 This is a schematic diagram of the cross-modal feature matching process of the method of the present invention;

[0077] Figure 4 This is a schematic diagram of the proposal generation process of the method of the present invention. Detailed Implementation

[0078] Example 1: As Figures 1-4 As shown, a training-free video corpus moment retrieval method based on an adaptive calibration mechanism includes:

[0079] Step 1: Input the natural language text query data into the large language model for semantic parsing and structural reconstruction; convert the text sentence structure into an ordered, structured query event chain;

[0080] Further, Step 1 includes: for a given natural language text query ,in Representing words, To represent the number of words, the large language model LLaMA 3.1-8B is first used to decompose it into a series of sub-events and then reorder them into a structured query event chain. The process of generating the structured query event chain through the large language model is represented as follows:

[0081] ;

[0082] in, This represents the LLaMA 3.1-8B large language model. Clue words indicating a large model, Indicates the first A textual description of the event. Indicates the number of event chains generated.

[0083] Step 2: Perform automatic subtitle segmentation and keyframe aggregation on the target video corpus data; divide the continuous video data stream into multiple semantically coherent event units, thereby mapping and representing the semantic stream of the video as a structured video event chain;

[0084] Furthermore, Step 2 specifically includes: for a given video ,in Represents a video frame. To represent the number of video frames, the visual language model BLIP is first used to generate its subtitle sequence. Then, similarity scores are calculated between the subtitle sequences. Video segments associated with the same subtitle are considered independent events, thus forming... Each event unit is an event unit; the core semantic features of each event are preserved, and the intermediate frames of these event units are further selected for aggregation to form a complete video event chain; the process of generating a video event chain through BLIP is represented as follows:

[0085] ;

[0086] ;

[0087] in, The prompt word indicating BLIP, Indicates the first The subtitle description corresponding to the frame, This represents the generated subtitle sequence. This represents the segmentation function, which divides the video... Divided into Each event unit is a separate event unit, and the intermediate frame of each event unit is further selected. Indicates the first event in the video event chain. One event.

[0088] Step 3: Use a vision-language pre-trained network model to extract multimodal features from natural language text queries, query event chains, input videos, and video event chains to construct a unified feature representation across modalities;

[0089] Further, Step 3 specifically includes: using the BLIP model as the vision-language pre-trained network model, and performing feature encoding on natural language text queries, query event chains, input videos, and video event chains respectively, mapping them to a unified feature representation of a unified dimension; the construction process of the unified feature representation is expressed as follows:

[0090] ;

[0091] ;

[0092] ;

[0093] ;

[0094] in Sentence-level features representing text queries. Represents a natural language text query. Indicates the first event in the query event chain. The characteristics corresponding to each event Indicates the first A textual description of the event. Indicates the number of event chains generated. This represents the query event chain feature matrix. Indicates the video number The features corresponding to the frame Represents the video feature matrix. Represents a video frame. Indicates the number of video frames. Indicates the first event in the video event chain. The characteristics corresponding to each event Represents the feature matrix of the video event chain. Indicates the first event in the video event chain. One event.

[0095] Step 4: Calculate the event-level similarity score between the query and the query event chain and the video event chain; integrate the mean and variance of the event-level similarity scores, and perform cross-modal feature matching through the mean-variance joint scoring mechanism to obtain the overall retrieval score;

[0096] Furthermore, Step 4 specifically includes the following steps:

[0097] First, calculate the cosine similarity between the query features and the video event chain features, and take the maximum value as the initial matching score between the query and the video;

[0098] Subsequently, regarding the first For each query event, the initial query event score at that event level is obtained by calculating the cosine similarity between its feature representation and the features of the video event chain and taking the maximum value.

[0099] Finally, the overall retrieval score is obtained by adding the mean of the initial query event scores to the initial matching scores of the query features and video event chain features, and then subtracting the variance of the initial query event scores; the mean-variance joint scoring mechanism is represented as follows:

[0100] ;

[0101] ;

[0102] ;

[0103] ;

[0104] in, This indicates the calculation of cosine similarity. This indicates selecting the maximum value. This represents the initial matching score between the query features and the video event chain features. Sentence-level features representing text queries. Indicates the first event in the video event chain. The characteristics corresponding to each event Indicates the first The initial matching score between the query event features and the video event chain features. Indicates the first event in the query event chain. The characteristics corresponding to each event Indicates the number of event chains generated. This represents the query event score vector. and This represents the mean and variance of the score for the event-level match, i.e., the score for the initial query event. This represents the overall search score.

[0105] Step 5: For the initially selected video candidate proposals, the sliding window method is used to generate two sets of time segment proposals based on the similarity scores between query features and video features, and the similarity scores between query event chain features and video features, respectively.

[0106] Furthermore, Step 5 specifically includes the following steps:

[0107] Step 5.1: First, calculate the similarity score between the query feature and the video feature. Then, generate a series of video proposals using the sliding window method. The score of each proposal is calculated by subtracting the mean of the out-of-proposal scores from the mean of the in-proposal scores. The proposal with the highest score is selected as the final query proposal. The specific process is as follows:

[0108] ;

[0109] ;

[0110] ;

[0111]

[0112] in, This indicates the query similarity score. Sentence-level features representing text queries. Indicates the video number The features corresponding to the frame Represents the video feature matrix. Represents a video frame. Indicates the number of video frames. This represents the sliding window method. Indicates the generated first One suggestion, This represents the total number of proposals. This indicates the query similarity score. All included in the All scores in the proposals This indicates the query similarity score. All not included in the All scores in the proposals This indicates calculating the mean. It is a function defined to evaluate the quality of a proposal. Indicates the final query proposal;

[0113] Step 5.2: Calculate the similarity score between the query event chain features and video features, starting with the first... The similarity scores between video features and query event chain features are calculated, and the maximum value is taken as the score between the video feature and the query event chain. Finally, the final query event chain proposal is generated. The specific process is as follows:

[0114]

[0115]

[0116]

[0117]

[0118] in, Indicates the first Similarity score between the frame and the query event chain. Indicates the video number The features corresponding to the frame Indicates the first event in the query event chain. The characteristics corresponding to each event This represents the similarity score sequence of the query event chain. This represents the similarity score sequence for the query event chain. All included in the All scores in the proposals This represents the similarity score sequence for the query event chain. All not included in the All scores in the proposals This indicates the final query event chain proposal.

[0119] Step 6: Perform interval overlap analysis on the two sets of time segment proposals generated from the same video candidate proposal; determine whether there is an intersection of time intervals between the time segment proposals based on text queries and the time segment proposals based on query event chains;

[0120] Furthermore, Step 6 specifically includes: for the final query proposal and the final query event chain proposal ,in and These represent the starting frame indices of the query proposal and the query event chain proposal, respectively. and These represent the end frame indices of the query proposal and the query event chain proposal, respectively. The final query proposal and the final query event chain proposal are analyzed to determine if there is any intersection. If there is an intersection, proceed to Step 7; otherwise, skip Step 7 and proceed to Step 8.

[0121] Step 7: In the temporal boundary localization stage, a collaborative calibration mechanism is introduced. Based on the interval intersection judgment result, boundary-level associations between video candidate proposals are established, and combined with the gain-loss dynamic feedback strategy, the temporal boundary of the candidate proposals is adaptively and iteratively optimized without updating any model parameters.

[0122] Furthermore, Step 7 specifically includes the following steps:

[0123] Step 7.1, consider satisfying Query suggestions and the final query event chain proposal Intersection Considered a potentially correct region for a given proposal First, a new proposal is constructed by replacing its ending frame. And will propose and suggestions The starting frames are moved to the left respectively Frame, get and ;

[0124] Step 7.2: Define the loss as the decrease in score due to interval contraction, and the gain as the score gain obtained on the corresponding proposal. The specific process is as follows:

[0125] ;

[0126] ;

[0127] in This indicates the query similarity score. This represents the similarity score sequence of the query event chain. It is a function defined to evaluate the quality of a proposal; if the gains exceed the losses, that is:

[0128]

[0129] If the adjustment is deemed effective, the proposal will be updated and repeated; if the loss outweighs the gain, then:

[0130]

[0131] Or when the iteration exceeds the intersection interval Stop immediately;

[0132] Step 7.3, Regarding the event chain proposal First, a new proposal is constructed by replacing its ending frame. And will propose and suggestions The starting frames are moved to the right respectively. Frame, get and The corresponding losses and gains are expressed as follows:

[0133] ;

[0134] ;

[0135] In response to the two proposals and After iterative optimization, the optimized proposal is denoted as... and ;

[0136] Step 8: Compare and select the optimal time boundary based on the dynamically optimized proposal scores; combine the score corresponding to the optimal time boundary with the overall retrieval score in Step 4 to output the final video time-lapse retrieval and precise positioning results.

[0137] Furthermore, Step 8 specifically includes: based on the optimized proposal and Calculate their query similarity scores. Similarity score sequence with query event chain The overall matching score, if Then choose to propose. Otherwise choose Then, the overall matching score of the selected proposals will be compared with the overall search score obtained in Step 4. The scores are summed to obtain the final score for the proposal. The specific process is as follows:

[0138]

[0139] in Indicates the overall search score. It is a function defined to evaluate the quality of a proposal. This is the final score for the proposal.

[0140] Furthermore, the present invention also provides a training-free video corpus moment retrieval system based on an adaptive calibration mechanism, the system comprising:

[0141] The query event chain generation model, guided by model prompts, decomposes complex text sentence structures into a series of sub-events and rearranges them in logical order. Ultimately, it transforms natural language into an ordered, structured query event chain, thereby explicitly revealing the semantic trajectory of the original text and the dependencies between events.

[0142] The video event chain generation module is used to divide video clips with the same or highly similar subtitles into independent, semantically coherent event units by calculating the similarity score between adjacent subtitle sequences. To filter out transitional and irrelevant content at event boundaries, the intermediate keyframes of each independent event unit are further extracted and aggregated, thereby mapping the continuous video stream into a structured video event chain;

[0143] The feature extraction module is used to extract features at multiple scales from "natural language text query", "query event chain", "input video sequence", and "video event chain". Through encoding operations, these four types of data with different granularities are mapped to a feature space of the same dimension, generating corresponding text query sentence-level features, query event chain feature matrices, video feature matrices, and video event chain feature matrices, laying the foundation for subsequent matching and alignment.

[0144] The cross-modal feature matching module introduces a mean-variance joint scoring mechanism. It utilizes the initial matching score between the query and the video, subtracting the mean and variance of the query event score to effectively suppress high response scores from local anomalies, enhance the consistency of global semantic features, and ultimately output the overall preliminary retrieval score, while also initially filtering out video candidate proposals.

[0145] The proposal generation module processes two main threads in parallel. First, based on the similarity scores of "text query features and video features," a sliding window is used to calculate the difference between the mean score within a proposal and the mean score outside of a proposal, selecting the time segment with the highest score as the query proposal. Similarly, based on the score sequence of "query event chain features and video features," the same mean difference strategy is used to generate the optimal query event chain proposal.

[0146] The collaborative calibration module treats the intersection interval as a potential correct region and introduces a collaborative calibration mechanism. It moves the proposed boundary frame left / right and calculates the gains and losses from dynamic feedback. Assuming the gains outweigh the losses, the proposed boundary is adaptively and iteratively optimized without updating any model parameters. Finally, the optimized matching score and global retrieval score are comprehensively evaluated to output the optimal temporal boundary and video time-localization result.

[0147] To verify the effectiveness of the method in this invention, ActivityNet-Captions and Charades-STA were used as datasets. The ActivityNet-Captions dataset contains approximately 20,000 YouTube videos with an average duration of 117.6 seconds. Each video is associated with an average of 3.7 text queries, and each query contains approximately 14.8 words. The Charades-STA dataset contains approximately 6,700 indoor activity videos with an average duration of 30 seconds, and each video is associated with 2.4 text queries. The algorithm in this invention is developed based on PyTorch and tested on a single NVIDIA RTX 4090 GPU. BLIP is used as the video and text encoder and is also used as the caption generation module. For the large language model, LLaMA 3.1-8B was used in all experiments. Notably, the number of candidate videos corresponding to each query was reduced to 200 using retrieval scores, and then a collaborative calibration mechanism was applied for further optimization. Hyperparameters Set to 1. This invention uses the Recall@K (IoU=m) metric, which measures the percentage of times in the first K retrieved times whose temporal intersection-union (IoU) with the real time exceeds a threshold m. Specifically, K is set to {1, 5, 10}, and m is set to {0.5, 0.7}. It is worth noting that in the VCMR task, a prediction is considered correct only if (1) the predicted video matches the real video, and (2) the predicted time segment has sufficient overlap with the real segment (the degree of overlap is quantified by temporal IoU).

[0148] Tables 1 and 2 summarize the performance of the present invention on the ActivityNet-Captions dataset and the Charades-STA dataset, respectively, and compare it with the RefCap (B) and RefCap (M) methods. It is worth noting that RefCap has two variant versions, which use BLIP and MiniGPT

[49] as visual language models (VLMs) for generating image captions. For clarity, these two versions are referred to as RefCap (B) and RefCap (M), respectively. Experimental results show that the present invention outperforms the comparison methods on all evaluation metrics such as Recall@K (IoU=m), K∈{1, 5, 10}, m∈{0.3, 0.5,0.7}. On the ActivityNet-Captions dataset shown in Table 1, compared with the RefCap (M) method, the present invention improves the performance of Iou=0.5, Recall@1, Recall@5 and Recall@10 metrics by 1.98%, 2.28% and 1.93%, respectively. At Iou=0.7, the performance of Recall@1, Recall@5, and Recall@10 metrics was improved by 1.33%, 1.57%, and 1.48%, respectively. On the Charades-STA dataset shown in Table 2, compared to the RefCap(M) method, this invention achieved improvements of 0.27%, 0.54%, and 0.91% at Iou=0.5, respectively, for Recall@1, Recall@5, and Recall@10 metrics. At Iou=0.7, the improvements were 0.22%, 0.76%, and 0.94%, respectively, demonstrating its significant performance advantage.

[0149] Table 1 shows the performance of different methods on the ActivityNet-Captions dataset.

[0150] Table 2 shows the performance of different methods on the Charades-STA dataset.

[0151] The specific embodiments of the present invention have been described in detail above. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A training-free video corpus moment retrieval method based on adaptive calibration mechanism, characterized in that: The method includes: Step 1: Input the natural language text query data into the large language model for semantic parsing and structural reconstruction; convert the text sentence structure into an ordered, structured query event chain; Step 2: Perform automatic subtitle segmentation and keyframe aggregation on the target video corpus data; divide the continuous video data stream into multiple semantically coherent event units, thereby mapping and representing the semantic stream of the video as a structured video event chain; Step 3: Use a vision-language pre-trained network model to extract multimodal features from natural language text queries, query event chains, input videos, and video event chains to construct a unified feature representation across modalities; Step 4: Calculate the event-level similarity score between the query and the query event chain and the video event chain; integrate the mean and variance of the event-level similarity scores, and perform cross-modal feature matching through the mean-variance joint scoring mechanism to obtain video candidate proposals; Step 5: For the initially selected video candidate proposals, the sliding window method is used to generate two sets of time segment proposals based on the similarity scores between query features and video features, and the similarity scores between query event chain features and video features, respectively. Step 6: Perform interval overlap analysis on the two sets of time segment proposals generated from the same video candidate proposal; determine whether there is an intersection of time intervals between the time segment proposals based on text queries and the time segment proposals based on query event chains; Step 7: In the temporal boundary localization stage, a collaborative calibration mechanism is introduced. Based on the interval intersection judgment result, boundary-level associations between video candidate proposals are established, and combined with the gain-loss dynamic feedback strategy, the temporal boundary of the candidate proposals is adaptively and iteratively optimized without updating any model parameters. Step 8: Compare and select the optimal time boundary based on the dynamically optimized proposal scores; combine the score corresponding to the optimal time boundary with the overall retrieval score in Step 4 to output the final video time-lapse retrieval and precise positioning results.

2. The training-free video corpus moment retrieval method based on adaptive calibration mechanism according to claim 1, characterized in that: Step 1 includes: for a given natural language text query ,in Representing words, To represent the number of words, the large language model LLaMA 3.1-8B is first used to decompose it into a series of sub-events and then reorder them into a structured query event chain. The process of generating the structured query event chain through the large language model is represented as follows: ； in, This represents the LLaMA 3.1-8B large language model. Clue words indicating a large model, Indicates the first A textual description of the event. Indicates the number of event chains generated.

3. The training-free video corpus moment retrieval method based on adaptive calibration mechanism according to claim 1, characterized in that: The specific steps of Step 2 include: for a given video ,in Represents a video frame. To represent the number of video frames, the visual language model BLIP is first used to generate its subtitle sequence. Then, similarity scores are calculated between the subtitle sequences. Video segments associated with the same subtitle are considered independent events, thus forming... Each event unit is an event unit; the core semantic features of each event are preserved, and the intermediate frames of these event units are further selected for aggregation to form a complete video event chain; the process of generating a video event chain through BLIP is represented as follows: ； ； in, The prompt word indicating BLIP, Indicates the first The subtitle description corresponding to the frame, This represents the generated subtitle sequence. This represents the segmentation function, which divides the video... Divided into Each event unit is a separate event unit, and the intermediate frame of each event unit is further selected. Indicates the first event in the video event chain. One event.

4. The training-free video corpus moment retrieval method based on adaptive calibration mechanism according to claim 1, characterized in that: Step 3 specifically includes: using the BLIP model as a vision-language pre-trained network model, and performing feature encoding on natural language text queries, query event chains, input videos, and video event chains, respectively, mapping them to a unified feature representation of a unified dimension; the construction process of the unified feature representation is as follows: ； ； ； ； in Sentence-level features representing text queries. Represents a natural language text query. Indicates the first event in the query event chain. The characteristics corresponding to each event Indicates the first A textual description of the event. Indicates the number of event chains generated. This represents the query event chain feature matrix. Indicates the video number The features corresponding to the frame Represents the video feature matrix. Represents a video frame. Indicates the number of video frames. Indicates the first event in the video event chain. The characteristics corresponding to each event Represents the feature matrix of the video event chain. Indicates the first event in the video event chain. One event.

5. The training-free video corpus moment retrieval method based on adaptive calibration mechanism according to claim 1, characterized in that: The specific steps in Step 4 include: First, calculate the cosine similarity between the query features and the video event chain features, and take the maximum value as the initial matching score between the query and the video; Subsequently, regarding the first For each query event, the initial query event score at that event level is obtained by calculating the cosine similarity between its feature representation and the features of the video event chain and taking the maximum value. Finally, the overall retrieval score is obtained by adding the mean of the initial query event scores to the initial matching scores of the query features and video event chain features, and then subtracting the variance of the initial query event scores; the mean-variance joint scoring mechanism is represented as follows: ； ； ； ； in, This indicates the calculation of cosine similarity. This indicates selecting the maximum value. This represents the initial matching score between the query features and the video event chain features. Sentence-level features representing text queries. Indicates the first event in the video event chain. The characteristics corresponding to each event Indicates the first The initial matching score between the query event features and the video event chain features. Indicates the first event in the query event chain. The characteristics corresponding to each event Indicates the number of event chains generated. This represents the query event score vector. and This represents the mean and variance of the score for the event-level match, i.e., the score for the initial query event. This represents the overall search score.

6. The training-free video corpus moment retrieval method based on adaptive calibration mechanism according to claim 1, characterized in that: The specific steps in Step 5 include: Step 5.1: First, calculate the similarity score between the query feature and the video feature. Then, generate a series of video proposals using the sliding window method. The score of each proposal is calculated by subtracting the mean of the out-of-proposal scores from the mean of the in-proposal scores. The proposal with the highest score is selected as the final query proposal. The specific process is as follows: ； ； ； ； in, This indicates the query similarity score. Sentence-level features representing text queries. Indicates the video number The features corresponding to the frame Represents the video feature matrix. Represents a video frame. Indicates the number of video frames. This represents the sliding window method. Indicates the generated first One suggestion, This represents the total number of proposals. This indicates the query similarity score. All included in the All scores in the proposals This indicates the query similarity score. All not included in the All scores in the proposals This indicates calculating the mean. It is a function defined to evaluate the quality of a proposal. Indicates the final query proposal; Step 5.2: Calculate the similarity score between the query event chain features and video features, starting with the first... The similarity scores between video features and query event chain features are calculated, and the maximum value is taken as the score between the video feature and the query event chain. Finally, the final query event chain proposal is generated. The specific process is as follows: ； ； ； ； in, Indicates the first Similarity score between the frame and the query event chain. Indicates the video number The features corresponding to the frame Indicates the first event in the query event chain. The characteristics corresponding to each event This represents the similarity score sequence of the query event chain. This represents the similarity score sequence for the query event chain. All included in the All scores in the proposals This represents the similarity score sequence for the query event chain. All not included in the All scores in the proposals This indicates the final query event chain proposal.

7. The training-free video corpus moment retrieval method based on adaptive calibration mechanism according to claim 1, characterized in that: Step 6 specifically includes: for the final query proposal and the final query event chain proposal ,in and These represent the starting frame indices of the query proposal and the query event chain proposal, respectively. and These represent the end frame indices of the query proposal and the query event chain proposal, respectively. The final query proposal and the final query event chain proposal are analyzed to determine if there is any intersection. If there is an intersection, proceed to Step 7; otherwise, skip Step 7 and proceed to Step 8.

8. The training-free video corpus moment retrieval method based on adaptive calibration mechanism according to claim 1, characterized in that: The specific steps in Step 7 include: Step 7.1, consider satisfying Query suggestions and the final query event chain proposal Intersection Considered a potentially correct region for a given proposal First, a new proposal is constructed by replacing its ending frame. And will propose and suggestions The starting frames are moved to the left respectively Frame, get and ; Step 7.2: Define the loss as the decrease in score due to interval contraction, and the gain as the score gain obtained on the corresponding proposal. The specific process is as follows: ; ; in This indicates the query similarity score. This represents the similarity score sequence of the query event chain. It is a function defined to evaluate the quality of a proposal; if the gains exceed the losses, that is: ； If the adjustment is deemed effective, the proposal will be updated and repeated; if the loss outweighs the gain, then: ； Or when the iteration exceeds the intersection interval Stop immediately; Step 7.3, Regarding the event chain proposal First, a new proposal is constructed by replacing its ending frame. And will propose and suggestions The starting frames are moved to the right respectively. Frame, get and The corresponding losses and gains are expressed as follows: ; ; In response to the two proposals and After iterative optimization, the optimized proposal is denoted as... and .

9. The training-free video corpus moment retrieval method based on adaptive calibration mechanism according to claim 1, characterized in that: Step 8 specifically includes: based on the optimized proposal and Calculate their query similarity scores. Similarity score sequence with query event chain The overall matching score, if Then choose to propose. Otherwise choose Then, the overall matching score of the selected proposals will be compared with the overall search score obtained in Step 4. The scores are summed to obtain the final score for the proposal. The specific process is as follows: ； in Indicates the overall search score. It is a function defined to evaluate the quality of a proposal. This is the final score for the proposal.

10. A training-free video corpus moment retrieval system based on an adaptive calibration mechanism, characterized in that, The system includes a module for performing the training-free video corpus moment retrieval method based on an adaptive calibration mechanism as described in any one of claims 1 to 9.

Images