A multi-modal large model-based training-free video retrieval method and system

Abstract

This invention belongs to the field of video retrieval technology, and relates to a training-free video retrieval method and system based on a multimodal large model. It sequentially performs semantic enhancement and temporal causal context expansion on the original query text to generate an enhanced query sequence, alleviating the information density asymmetry between text queries and video temporal dynamics. It extracts global features from candidate videos and calculates the semantic similarity between the global features and the enhanced query sequence, achieving coarse-grained high recall retrieval. This solves the problem of insufficient recall caused by the semantic sparsity of the original query in zero-shot scenarios. The original query text is element-splitting and filtering to obtain the smallest question-and-answer subset covering all semantics. Then, a question-and-answer pair consistency verification is performed on the candidate video set, enabling the model to verify the correspondence between query details and video content through propositional verification. This solves the problem of poor fine-grained matching accuracy caused by insufficient cross-modal alignment granularity.

Description

Technical Field

[0001] This invention belongs to the field of video retrieval technology, and relates to a training-free video retrieval method and system based on a multimodal large model. Background Technology

[0002] Video retrieval is a crucial task in cross-modal information retrieval, involving locating the most relevant video content from massive amounts of video data based on user-provided text queries. It is a key technology connecting natural language semantics with visual content. Video retrieval technology has wide applications in industries such as public safety monitoring, digital media production, and online education.

[0003] Training-free video retrieval technology based on multimodal large models has been increasingly widely applied in many fields in recent years. In particular, video retrieval methods based on large language model prompting strategies can achieve cross-modal semantic alignment without relying on large-scale labeled data and parameter optimization, compared with traditional end-to-end training models. With the explosive growth of multimedia data, the scale of video data has exceeded trillions, and the core problem faced by users has shifted from content scarcity to the difficulty of accurate acquisition. However, traditional video retrieval methods usually rely on end-to-end training on large-scale labeled data to construct a joint embedding space to achieve cross-modal alignment. However, this paradigm faces practical bottlenecks such as high training costs and scarcity of manually labeled data, making it difficult to adapt to the rapid retrieval needs in open scenarios.

[0004] Training-free video retrieval, as an important branch of zero-shot transfer learning, can achieve generalized video retrieval without relying on downstream task labeled data or parameter optimization. It directly performs semantic matching through the cross-modal alignment capabilities embedded in pre-trained models. Specifically, the main goal of training-free video retrieval is to leverage the general visual language representations learned by large-scale pre-trained models to map videos and text to a shared semantic space without updating model parameters, and then complete the retrieval by calculating feature similarity. This method relies on the zero-shot transfer capability of the video base model or the cue learning capability of the multimodal large model, fundamentally avoiding dependence on labeled data and the consumption of training resources.

[0005] While training-free video retrieval methods can achieve zero-shot retrieval by leveraging the generalization ability of pre-trained models, they still have corresponding problems. Most current training-free methods simply use video and text encodings from a basic video model to achieve simple modal alignment. However, user queries often employ abstract semantic expressions, lacking explicit visual entities and complete temporal causal logic chains. This information sparsity of text and the visual richness of video create a severe asymmetry. Furthermore, existing methods lack the ability to explicitly decompose query intent and perform fine-grained verification, making it difficult to verify the correspondence between query details and video content. InternVideo2, a rapidly developing video model in recent years, achieves zero-shot retrieval through multimodal contrastive learning of video, audio, speech, and text, significantly improving cross-modal semantic alignment capabilities compared to the traditional CLIP4Clip model. However, like most training-free video retrieval methods, InternVideo2 directly encodes the original user query for feature matching, failing to compensate for the lack of visual details and causal logic breaks in the query text itself. This results in low recall and poor ranking accuracy in fine-grained retrieval scenarios. Summary of the Invention

[0006] The purpose of this invention is to solve the problems of insufficient recall caused by the sparse semantics of the original query in zero-sample scenarios and insufficient granularity of cross-modal alignment in the prior art, which leads to poor accuracy of fine-grained matching. This invention provides a training-free video retrieval method and system based on a multimodal large model.

[0007] To achieve the above objectives, the present invention employs the following technical solution: A training-free video retrieval method based on a multimodal large model includes the following steps: Obtain the original query text, perform semantic enhancement and temporal causal context expansion on the original query text in sequence to generate an enhanced query sequence, extract global features of candidate videos, calculate the semantic similarity between global features and enhanced query sequence, and select candidate video sets based on semantic similarity; The original query text is split into elements, and an initial set of candidate question-answer pairs is generated based on the split elements. The initial set of candidate question-answer pairs is filtered for semantic consistency and visual verifiability to obtain the minimum question-answer subset that covers all semantics. The minimum question-answer subset is used to perform consistency verification on the candidate video set to obtain the verification score of each video. The video retrieval ranking results are obtained by aggregating the video ranking in the candidate video set with the verification scores of each video.

[0008] A further improvement of the present invention is that: The sequential semantic enhancement and temporal causal context expansion of the original query text include: By sequentially performing semantic debiasing and visual detail expansion on the original query text using a large language model, a visually enhanced query is generated. Based on visually enhanced queries, temporal causal reasoning is performed to infer the preceding and succeeding events of the core event, and to generate preceding event queries and succeeding event queries.

[0009] The process of generating a visually enhanced query by sequentially performing semantic debiasing and visual detail expansion on the original query text using a large language model includes: We define three dimensions: scene background, associated objects, and role attributes. Based on these three dimensions, we activate the visual common sense knowledge of the large language model through role-based prompts. Then, we perform zero-shot extensional reasoning on the semantically debiased query text based on the activated large language model to generate visually enhanced queries with fine-grained visual semantics.

[0010] The process of obtaining the minimum question-and-answer subset that covers all semantics includes... Discriminative element decomposition is performed on visual entities, attributes, actions, and spatiotemporal relationships in the original query text. Based on the decomposed elements, a descriptive initial candidate question-answer pair set is generated. Construct a prompt template containing negative examples of low-quality question-answering. Use the prompt template to remove question-answering pairs from the initial candidate question-answering pair set that do not match the semantics of the original query. Perform visual verifiability screening on the remaining candidate question-answering pair set to remove question-answering pairs that cannot be verified visually and semantically, and generate a question-answering pair set after quality screening. Solve the question-and-answer set after quality screening to generate the smallest subset of questions and answers that covers visual entities, attributes, actions, and spatiotemporal relationships.

[0011] The consistency verification of the candidate video set based on the minimum question-answer subset is performed to obtain the verification score of each video, including: Each video in the candidate video set and each question-answer pair in the minimum question-answer subset are input into the multimodal large model. The linear probability of a word matching the expected answer is output. The linear probability is converted into a matching score for each video and each question-answer pair. The question-answer pairs are then sorted according to the matching score. The ranking results of all question-answer pairs are aggregated using the inverse ranking fusion method to obtain the verification score for each video.

[0012] The step of aggregating the video ranking in the candidate video set with the verification score of each video to obtain the video retrieval ranking result includes: The inverse ranking fusion method is used to calculate the fusion score between the video ranking and the verification score of each video:

[0013] In the formula, Indicating the first stage Ranking; Indicates based on The second-stage ranking is arranged in descending order. This represents the smoothing constant.

[0014] A training-free video retrieval system based on a multimodal large model includes: The candidate video set filtering module is used to obtain the original query text, perform semantic enhancement and temporal causal context expansion on the original query text in sequence, generate an enhanced query sequence, extract global features of candidate videos, calculate the semantic similarity between global features and enhanced query sequence, and filter candidate video sets based on semantic similarity. The candidate video consistency verification module is used to split the original query text into elements, generate an initial candidate question-answer pair set based on the split elements, perform semantic consistency and visual verifiability screening on the initial candidate question-answer pair set, obtain the minimum question-answer subset covering all semantics, and perform consistency verification on the candidate video set based on the minimum question-answer subset to obtain the verification score of each video. The video retrieval and ranking module is used to aggregate the video ranking in the candidate video set with the verification score of each video to obtain the video retrieval and ranking results.

[0015] A computer program product includes a computer program that, when executed by a processor, implements any one of the methods described.

[0016] A terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the steps of any of the methods described above.

[0017] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the methods described herein.

[0018] Compared with the prior art, the present invention has the following beneficial effects: This invention discloses a training-free video retrieval method based on a multimodal large model. It sequentially performs semantic enhancement and temporal causal context expansion on the original query text to generate an enhanced query sequence, alleviating the information density asymmetry between text queries and the temporal dynamics of videos. This allows the expanded query to more comprehensively cover visual elements and complete event chains in the video. Global features of candidate videos are extracted, and the semantic similarity between the global features and the enhanced query sequence is calculated, achieving coarse-grained high recall retrieval. This solves the problem of insufficient recall caused by the semantic sparsity of the original query in zero-shot scenarios. The original query text is element-splitting and filtering to obtain the smallest question-answer subset covering all semantics. Then, a question-answer pair consistency verification is performed on the candidate video set, enabling the model to verify the correspondence between query details and video content through propositional verification. This solves the problem of poor fine-grained matching accuracy caused by insufficient cross-modal alignment granularity. Finally, the video ranking in the candidate video set is aggregated with the verification scores of each video to improve ranking accuracy. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is an overall flowchart of the method of the present invention; Figure 2 This is a schematic diagram of the query optimization process in this invention; Figure 3 This is a schematic diagram of the thought process chain reasoning used to generate the problem in this invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the invention without inventive effort are within the scope of protection of the invention.

[0023] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0024] In the description of the embodiments of the present invention, it should be noted that if terms such as "upper," "lower," "horizontal," or "inner" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship commonly used when the product of the invention is in use, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. Furthermore, terms such as "first" and "second" are only used to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0025] Furthermore, the use of the term "horizontal" does not imply that the component must be absolutely horizontal, but rather that it can be slightly tilted. For example, "horizontal" simply means that its direction is more horizontal than "vertical," and does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0026] In the description of the embodiments of the present invention, it should also be noted that, unless otherwise explicitly specified and limited, the terms "set," "install," "connect," and "link" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in the present invention according to the specific circumstances.

[0027] The present invention will now be described in further detail with reference to the accompanying drawings: See Figures 1 to 3 This invention discloses a training-free video retrieval method based on a multimodal large model. The first stage uses semantically enhanced and causally enhanced extended queries to perform coarse-grained recall of the retrieved videos. The second stage explicitly decomposes the query into fine-grained questions, and then utilizes the capabilities of the multimodal large model to validate and re-rank these questions on the coarse-grained recalled videos. This invention can address the problems of insufficient recall caused by the semantic sparsity of the original query, cross-modal representation alignment bias, and noise interference from coarse-grained similarity matching in zero-shot scenarios.

[0028] Includes the following steps: Step 1: For a given original text query, semantic bias removal, visual detail expansion, and temporal causal contextual reasoning are performed sequentially through a large language model to generate an enhanced query sequence. The enhanced query sequence includes visual enhancement queries, preceding event queries, and subsequent event queries. The cross-modal similarity between candidate videos and each query in the enhanced query sequence is calculated using a pre-trained video base model. The comprehensive similarity score is obtained by max pooling aggregation, and the top K videos are selected as the candidate video set. This step optimizes queries based on visual expansion and temporal information supplementation to achieve high recall in Top-K candidate video filtering. Specifically, it includes the following steps: To address text quality biases, this invention first employs a large language model for query correction. Through spelling correction, terminology standardization, and contextual disambiguation, noise in the query text is transformed into a semantically clear benchmark, establishing a reliable foundation for subsequent cross-modal alignment.

[0029] Furthermore, to address the issue of insufficient visual semantic coverage in queries, this embodiment enhances the original query's visual semantics from three dimensions: scene background, associated objects, and role attributes. The enhanced text is then injected with fine-grained visual information, which on the one hand expands the query's coverage in the visual feature space, improving the recall rate for the target video; on the other hand, it strengthens the query's distinguishability, as the added details differentiate it from semantically mismatched non-target videos.

[0030] To systematically implement the above two-stage enhancement, this embodiment designs a chained suggestion strategy based on Qwen3-8B: In the first stage, to address the issue of query bias correction, constraints are used to generate hints to correct spelling errors, eliminate grammatical ambiguities, and output standardized queries.

[0031] The second stage addresses the issue of insufficient semantic coverage. Based on the results of the first stage, it activates the model's visual common sense knowledge through role-based prompts, performs zero-shot extrapolational reasoning on scene context, related objects, and main character attributes, and generates enhanced semantic queries.

[0032] The core of this design lies in using static detail injection to map the original query from a high-order, simplified action semantic space to text with fine-grained visual semantics, thereby improving the feature discrimination and recall robustness of cross-modal retrieval.

[0033] Furthermore, let the query text be denoted as The video is Traditional video retrieval methods aim to minimize the gap between the query text and the target video. ,in, Typically, temporal average pooling or attention aggregation is used, but the original query usually corresponds to the core event in the video clip, ignoring the cause and effect. This leads to a mismatch between the causal structure of the text representation and the temporal dynamics of the video, and cannot solve the problem of temporal granularity heterogeneity between text and video.

[0034] Based on this, this embodiment reasonably expands the query text in terms of temporal causality, and the specific steps are as follows: Figure 2 As shown. Specifically, this embodiment, based on Qwen3-8B, utilizes the reasoning capabilities of a large model and the logical relationships between video events to perform causal context expansion on the semantically enhanced query text, displaying the preceding events that lead to the inference of the core event. and subsequent events Expanding isolated events into a sequence of causal events The search target was adjusted to .

[0035] This method, by making the query structure explicit, guides the alignment process to focus on the temporal dynamics of the video, thereby improving the retrieval coverage of complete event sequences without increasing the computational overhead on the video side.

[0036] Furthermore, in terms of Prompt design, this embodiment designs two Prompts for cause and effect respectively. Compared with semantic enhancement prompts, the core innovation lies in introducing reasonable uncertainty. By explicitly encouraging the model to reason in reasonable causal contexts similar to the original event, the semantic coverage of the query is expanded, and the model is guided to generate more relevant core features.

[0037] Furthermore, after completing the semantic enhancement and temporal causal context expansion of the query text, this embodiment employs the InternVideo2 model for multimodal feature encoding and similarity calculation. As a large-scale video understanding model, it possesses robust and accurate cross-modal matching capabilities between video and text. The query representation is obtained by encoding using the InternVideo2 text encoder. , , :

[0038]

[0039] (1) In the formula, This indicates the query text has been enhanced with details. , This indicates that the query text, after being enhanced with time-series causality, consists of antecedents and consequences; This represents the feature dimension after text encoding.

[0040] To supplement visual semantics and enhance the robustness of cross-modal alignment, this embodiment introduces automatic speech recognition technology to generate auxiliary text, utilizing semantic information from the audio channel to construct a more complete video representation. Specifically, this includes: For any video in the video collection Deconstruct it into a visual frame sequence With native audio tracks The system performs visual feature extraction and auxiliary subtitle generation, respectively. Visual features are obtained using an 8-frame sparse sampling strategy and are processed by a visual encoder. : (2) For audio signals, this embodiment uses an ASR model to transcribe the speech signal into auxiliary semantic information in text form, denoted as . And it is encoded using the InternVideo2 text encoder. : (3) Subsequently, cross-modal alignment was achieved through the video subtitle fusion module of InternVideo2, that is, by splicing visual and text features, and then the fusion module generated a global video representation. : (4) Finally, the semantic similarity between the global video representation, the enhanced query text representation, and the preceding and following causal extended text is calculated using cosine similarity.

[0041]

[0042] (5) This invention does not employ traditional average polymerization for polymerization with three similarity levels. This method has a semantic dilution effect in some matching scenarios, such as when only a certain query text matches a video. This invention adopts max pooling aggregation to optimize the three similarity ranking methods: (6) Based on the above cross-modal similarity scores The process of selecting Top-K candidate videos is a preliminary step for subsequent retrieval and re-ranking. Its purpose is to reduce the computational complexity of subsequent fine-grained re-ranking while maintaining retrieval recall. Specifically, this includes: video corpus Similarity score of all videos in Perform descending sorting and select the top-ranked scores. The video constitutes the candidate video set : (7) In the formula: ( () represents a descending sorting function based on similarity scores; Indicates the preceding A result selection function.

[0043] Step 2: Based on the large language model, the thought chain prompt decomposes the original query into multiple minimum-coverage question-answer pairs. Each question-answer pair contains a discriminative proposition and its expected answer. The consistency of each question-answer pair in each video in the candidate video set is verified by using a multimodal large model to obtain the verification score of each video. In this step, based on query-splitting question-answer verification, a discriminative question-answer pair set is constructed, and a large visual language model is used to verify consistency and re-rank candidate videos. Specifically, this includes: This embodiment aims to generate a minimal set of question-answer pairs that covers all core semantics of the query. Let the generated effective fine-grained question set be denoted as . , To minimize the number of semantic QA pairs that cover all semantics, the expected set of binary answers for the corresponding question is: : (8) In the formula, This represents the user's original text query; This indicates that a QA in a specified format is used to generate boot instructions.

[0044] Specifically, this embodiment uses Qwen3-8B as the parsing engine, generating a set of questions strongly related to the query through four stages. The reasoning logic for question generation is as follows: Figure 3 As shown, it specifically includes: The first stage involves breaking down the discriminative elements in the original query and extracting the visual entities, attributes, actions, and spatiotemporal relationships that exist within them. The second stage, based on the elements analyzed in the first stage, constructs an initial set of candidate question-answer pairs from the original query. This embodiment restricts the large model to generate descriptive propositions rather than inferential propositions, that is, to judge whether the statement in the question is correct rather than open-ended answers. The advantages of doing so are twofold: first, it reduces the difficulty of answering, as the model only needs to judge whether it is correct, which helps to control the risk of hallucination; second, the answer space can be strictly limited to "Yes" or "No", and the confidence rate of the answer can be obtained by extracting the token distribution of the large model.

[0045] The third stage involves quality screening of the QA pairs generated in stage two. First, a context learning mechanism is used to construct a prompt template containing low-quality question-answer counterexamples to guide the model to eliminate propositions that do not conform to the semantics of the original query. Then, a second screening is performed based on visual verifiability to guide the model to filter out QA pairs that cannot be verified visually and semantically. The fourth stage involves finding the minimum subset of questions and answers that covers all semantics, minimizing this subset while satisfying the four types of verification coverage: existence, attribute, action, and temporal. Final output A fine-grained verification proposition with a standard answer is used for subsequent video verification. Each stage is conducted using a chain-of-thought reasoning approach, requiring the model to output concise intermediate reasoning processes to ensure the reasonableness and transparency of the generated results.

[0046] For the first phase of retrieval A set of candidate videos for the results This invention utilizes the previously generated QA pairs for verification, and calculates and re-ranks the matching scores for each video independently based on the results. Specifically, it is based on the original query generated... The answer to each question is correct. For each candidate video Connect it with each question The process involves combining each input into Qwen3-VL to obtain the matching score for the correct result, completing the entire process. Sub-independent reasoning verification: (9) In the formula: This represents the model's output terms and expected answers. The linear probability of agreement; a higher value indicates a higher output matching score.

[0047] To measure the overall consistency between candidate videos and query descriptions, this invention employs a question-based reordering strategy when aggregating the matching scores of multiple QA pairs, that is, for each question... Considered an independent verification perspective, the number of videos that correctly match the answers is used as the ranking criterion. Specifically, based on... Rankings for each question And the ranking of multiple questions is fused using the inverse ranking fusion RRF method: (10) Step 3: Aggregate the initial ranking of candidate videos in Step 1 with the fine-grained ranking obtained based on the validation score in Step 2 using the inverse ranking fusion (RRF) method to output the final video retrieval ranking result.

[0048] This step aggregates the results using the Reverse Ranking fusion RRF method, outputting the final video retrieval ranking result. This result combines the initial ranking from the first stage with the fine-grained validation from the second stage. The present invention employs the Reverse Ranking fusion RRF method, and the fusion score is shown in the formula: (11) In the formula, Indicating the first stage Ranking; Indicates based on The second-stage ranking is arranged in descending order. Let the smoothing constant be denoted as , following the experimental specifications of the original RRF literature. = 60, a value that was verified as near optimal in the pilot study and is insensitive to parameter selection. The final video is based on... Output in descending order.

[0049] This invention employs a large language model to semantically correct and visually complete the original query, generating an enhanced query rich in background, object, and character attributes. Furthermore, leveraging the temporal causal reasoning capabilities of the large language model, the core event is expanded into a complete causal sequence of cause, current, and consequence, mitigating the information density asymmetry between text queries and video temporal dynamics. Then, a video-based model is used to calculate cross-modal similarity between the enhanced query and candidate videos, achieving coarse-grained, high-recall retrieval. Finally, the query is decomposed into minimum-coverage question-answer pairs using thought chain hints, and a multimodal large model is used to verify the consistency of each candidate video. Finally, a reciprocal ranking fusion method is used to aggregate the two-stage ranking results, generating the final retrieval ranking.

[0050] Furthermore, in this invention, the query optimization module adopts a chain-like suggestion strategy. In the first stage, it corrects spelling errors, obscure expressions, and semantic ambiguities in the original query and outputs a standardized query. In the second stage, based on role-based suggestions, it expands visual semantics from three dimensions: scene context, related objects, and subject attributes, mapping high-level abstract semantics to a fine-grained visual text space, thereby improving the feature discrimination and recall robustness of cross-modal retrieval.

[0051] Furthermore, in this invention, the question-answering verification module first decomposes the query into discriminative elements such as visual entities, attributes, actions, and spatiotemporal relationships through thought chain reasoning, generating binary question-answer pairs; then, it uses a context learning mechanism to eliminate low-quality questions that are semantically inconsistent or visually unverifiable, and solves for the minimum question-answer subset that covers all semantics; finally, it verifies each candidate video one by one based on a multimodal large model, and uses the consistency probability between the model output tokens and the expected answers as the matching score, thereby achieving fine-grained content verification and interpretability reordering.

[0052] This embodiment also discloses a training-free video retrieval system based on a multimodal large model, including: The candidate video set filtering module is used to obtain the original query text, perform semantic enhancement and temporal causal context expansion on the original query text in sequence, generate an enhanced query sequence, extract global features of candidate videos, calculate the semantic similarity between global features and enhanced query sequence, and filter candidate video sets based on semantic similarity. The candidate video consistency verification module is used to split the original query text into elements, generate an initial candidate question-answer pair set based on the split elements, perform semantic consistency and visual verifiability screening on the initial candidate question-answer pair set, obtain the minimum question-answer subset covering all semantics, and perform consistency verification on the candidate video set based on the minimum question-answer subset to obtain the verification score of each video. The video retrieval and ranking module is used to aggregate the video ranking in the candidate video set with the verification score of each video to obtain the video retrieval and ranking results.

[0053] It should be noted that the solutions disclosed in this embodiment of the invention are supported by the Jiangsu Provincial Agricultural Science and Technology Independent Innovation Fund Project, project number: CX(24)3132.

[0054] A schematic diagram of a terminal device according to an embodiment of the present invention. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the various method embodiments described above. Alternatively, when the processor executes the computer program, it implements the functions of each module / unit in the various device embodiments described above.

[0055] The computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present invention.

[0056] The terminal device can be a desktop computer, laptop computer, cloud server, or other device with strong computing power. The terminal device may include, but is not limited to, a processor and memory.

[0057] The optimal choice for the processor is a multi-core high-speed central processing unit (CPU).

[0058] The memory can be used to store the computer program and / or module. The processor implements various functions of the terminal device by running or executing the computer program and / or module stored in the memory and calling the data stored in the memory.

[0059] If the modules / units integrated into the terminal device are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0060] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A training-free video retrieval method based on a multimodal large model, characterized in that, Includes the following steps: Obtain the original query text, perform semantic enhancement and temporal causal context expansion on the original query text in sequence to generate an enhanced query sequence, extract global features of candidate videos, calculate the semantic similarity between global features and enhanced query sequence, and select candidate video sets based on semantic similarity; The original query text is split into elements, and an initial set of candidate question-answer pairs is generated based on the split elements. The initial set of candidate question-answer pairs is filtered for semantic consistency and visual verifiability to obtain the minimum question-answer subset that covers all semantics. The minimum question-answer subset is used to perform consistency verification on the candidate video set to obtain the verification score of each video. The video retrieval ranking results are obtained by aggregating the video ranking in the candidate video set with the verification scores of each video.

2. The training-free video retrieval method based on a multimodal large model according to claim 1, characterized in that, The sequential semantic enhancement and temporal causal context expansion of the original query text include: By sequentially performing semantic debiasing and visual detail expansion on the original query text using a large language model, a visually enhanced query is generated. Based on visually enhanced queries, temporal causal reasoning is performed to infer the preceding and succeeding events of the core event, and to generate preceding event queries and succeeding event queries.

3. The training-free video retrieval method based on a multimodal large model according to claim 2, characterized in that, The process of generating a visually enhanced query by sequentially performing semantic debiasing and visual detail expansion on the original query text using a large language model includes: We define three dimensions: scene background, associated objects, and role attributes. Based on these three dimensions, we activate the visual common sense knowledge of the large language model through role-based prompts. Then, we perform zero-shot extensional reasoning on the semantically debiased query text based on the activated large language model to generate visually enhanced queries with fine-grained visual semantics.

4. The training-free video retrieval method based on a multimodal large model according to claim 1, characterized in that, The process of obtaining the minimum question-and-answer subset that covers all semantics includes... Discriminative element decomposition is performed on visual entities, attributes, actions, and spatiotemporal relationships in the original query text. Based on the decomposed elements, a descriptive initial candidate question-answer pair set is generated. Construct a prompt template containing negative examples of low-quality question-answering. Use the prompt template to remove question-answering pairs from the initial candidate question-answering pair set that do not match the semantics of the original query. Perform visual verifiability screening on the remaining candidate question-answering pair set to remove question-answering pairs that cannot be verified visually and semantically, and generate a question-answering pair set after quality screening. Solve the set of question-and-answer pairs after quality screening to generate the smallest subset of question-and-answer pairs that covers visual entities, attributes, actions, and spatiotemporal relationships.

5. The training-free video retrieval method based on a multimodal large model according to claim 4, characterized in that, The consistency verification of the candidate video set based on the minimum question-answer subset is performed to obtain the verification score of each video, including: Each video in the candidate video set and each question-answer pair in the minimum question-answer subset are input into the multimodal large model. The linear probability of a word matching the expected answer is output. The linear probability is converted into a matching score for each video and each question-answer pair. The question-answer pairs are then sorted according to the matching score. The ranking results of all question-answer pairs are aggregated using the inverse ranking fusion method to obtain the verification score for each video.

6. The training-free video retrieval method based on a multimodal large model according to claim 1, characterized in that, The step of aggregating the video ranking in the candidate video set with the verification score of each video to obtain the video retrieval ranking result includes: The inverse ranking fusion method is used to calculate the fusion score between the video ranking and the verification score of each video: In the formula, Indicating the first stage Ranking; Indicates based on The second-stage ranking is arranged in descending order. This represents the smoothing constant.

7. A training-free video retrieval system based on a multimodal large model, characterized in that, include: The candidate video set filtering module is used to obtain the original query text, perform semantic enhancement and temporal causal context expansion on the original query text in sequence, generate an enhanced query sequence, extract global features of candidate videos, calculate the semantic similarity between global features and enhanced query sequence, and filter candidate video sets based on semantic similarity. The candidate video consistency verification module is used to split the original query text into elements, generate an initial candidate question-answer pair set based on the split elements, perform semantic consistency and visual verifiability screening on the initial candidate question-answer pair set, obtain the minimum question-answer subset covering all semantics, and perform consistency verification on the candidate video set based on the minimum question-answer subset to obtain the verification score of each video. The video retrieval and ranking module is used to aggregate the video ranking in the candidate video set with the verification score of each video to obtain the video retrieval and ranking results.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-6.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1-6.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1-6.

Images