A text video retrieval method based on multi-level prompt learning and cross-modal supervision, a storage device and a retrieval device

Abstract

The application discloses a text video retrieval method based on multi-level prompt learning and cross-modal supervision, a storage device and a retrieval device, relates to the technical field of computer vision and cross-modal retrieval, and comprises the following steps: introducing local frame-level and global video-level prompts, combining a parameter sharing strategy to realize joint modeling of video local details and global time sequence semantics; evaluating the discrimination ability of visual and text features by single-modal prediction supervision and dividing sample pair types, combining multi-modal contrast learning and multi-modal fusion supervision to realize accurate alignment of cross-modal features, and fusing single-modal supervision loss, multi-modal contrast loss and multi-modal fusion loss to construct a total loss function for training the model. The text video retrieval method based on multi-level prompt learning and cross-modal supervision, the storage device and the retrieval device provided by the application can improve the semantic matching precision between text and video, and enhance the robustness and generalization ability of the model in a complex scene.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and cross-modal retrieval technology, and more specifically, to a text video retrieval method, storage device, and retrieval device based on multi-level cue learning and cross-modal supervision. Background Technology

[0002] Text-based video retrieval is a common method that retrieves videos from large video databases that semantically best match the text descriptions provided in natural language. It is also a significant research area in computer vision. This task requires models to possess cross-modal semantic alignment capabilities while effectively modeling video temporal sequences to handle complex dynamic changes and inter-frame interactions within the video.

[0003] Early text-to-video retrieval methods typically extracted features from both text and video separately, employing contrastive learning for cross-modal matching within a shared embedding space. These methods often used average pooling or max pooling to obtain video representations from video frames. While these methods offer high retrieval efficiency, they struggle to effectively characterize the temporal semantics of keyframes in videos. Subsequent research has built upon these methods by introducing recurrent neural networks and temporal attention mechanisms to achieve weighted aggregation of video frame features, thereby enhancing the modeling ability for target actions and changes between frames. However, these methods still suffer from high computational costs and unstable temporal semantic modeling when dealing with long videos or complex scenes.

[0004] In recent years, with the rapid development of Transformer architecture and visual-language models, CLIP-based methods have gradually become one of the important supporting methods. This approach typically treats a video as a sequence of multiple video frames and obtains corresponding video representations through frame-level feature extraction and temporal aggregation. Although pre-trained models like CLIP have achieved promising results in this field, the increasing number of model parameters leads to higher fine-tuning costs, hindering further improvements to existing methods. Furthermore, because CLIP is pre-trained on large-scale image-text pairs, the model's visual encoder primarily learns static image semantics, lacking overall semantic modeling of the video. Therefore, when CLIP models encounter long videos or videos with uneven semantic distribution, they often rely too heavily on local frame information, affecting the model's ability to understand and retrieve the entire video. In addition, when applying pre-trained models to downstream visual-language tasks, the input video is usually processed as a set of frames, and video features are obtained through simple frame-level aggregation, ignoring the continuous semantics and dynamic representations of the video. This makes it difficult for the model to effectively capture information such as pose changes of targets and semantic relationships across frames.

[0005] Therefore, how to construct a lightweight and high-precision text video retrieval method that reduces model training costs and memory usage while improving the model's ability to model video temporal semantics and the accuracy of cross-modal feature alignment has become a key issue in current research. Summary of the Invention

[0006] The purpose of this invention is to provide a text video retrieval method, storage device, and retrieval device based on multi-level prompting learning and cross-modal supervision, which can improve the semantic matching accuracy between text and video and enhance the robustness and generalization ability of the model in complex scenarios.

[0007] This invention provides a text video retrieval method based on multi-level prompting learning and cross-modal supervision, comprising the following steps: S1: Construct a visual encoder and a text encoder based on the pre-trained CLIP model. The visual encoder is used to extract features from the video frame sequence to obtain visual feature representations, and the text encoder is used to perform sequence modeling of natural language descriptions to obtain text feature representations. S2: Obtain video-text pair samples, and use the visual encoder and text encoder to encode and aggregate the video-text pair samples to obtain text semantic features, frame-level visual features and video global representation; S3: Based on the text semantic features, frame-level visual features and video global representation, the video representation features and text representation features are enhanced using a multi-level cueing learning module to obtain enhanced video representation features and text representation features. S4: Based on the enhanced video representation features and text representation features, perform cross-modal supervision constraints to obtain positive sample pairs, negative sample pairs, imperfect sample pairs, and single-modal supervision loss; S5: Based on the positive sample pairs, negative sample pairs, and imperfect sample pairs, calculate the multimodal contrastive loss using multimodal contrastive learning; concatenate the video representation features and text representation features to obtain fusion features, supervise multimodal contrastive learning using the fusion features, and calculate the multimodal fusion loss; S6: Construct a total loss function based on the single-modal supervision loss, multimodal contrast loss, and multimodal fusion loss. Use the total loss function to train the text video retrieval model end-to-end to obtain the trained retrieval model. Use the trained retrieval model to perform text video retrieval.

[0008] The present invention also provides a storage device that stores instructions and data for implementing the above-mentioned text video retrieval method based on multi-level prompting learning and cross-modal supervision.

[0009] The present invention also provides a text-to-retrieval device, comprising: a processor and a storage device; the processor loads and executes instructions and data in the storage device for the aforementioned text-to-video retrieval method based on multi-level prompting learning and cross-modal supervision.

[0010] The text video retrieval method, storage device, and retrieval device based on multi-level prompting learning and cross-modal supervision provided by this invention have the following beneficial effects: This invention addresses the technical problems of existing text-video retrieval models, such as insufficient ability to model video temporal semantics, poor cross-modal alignment, high model fine-tuning costs, and large GPU memory consumption. It constructs a multi-level prompting learning module, introducing a joint modeling mechanism of global video-level prompts and local frame-level prompts. Combined with a parameter-sharing strategy, it achieves joint modeling of local video details and global temporal semantics, replacing the traditional simple frame-level feature aggregation method. Without disrupting the original input structure of the pre-trained model, it significantly improves the model's ability to model video temporal semantics and local details. The multi-level prompting combined with the lightweight learning strategy of parameter sharing effectively guides the visual and language encoders to learn feature representations with cross-modal semantic relevance, enhancing the robustness of text-video feature alignment and improving the accuracy of cross-modal bidirectional retrieval.

[0011] This invention constructs a cross-modal supervision and constraint module. It evaluates the discriminative ability of visual and textual features and classifies sample pairs through single-modal prediction supervision. Then, it combines multimodal contrastive learning and multimodal fusion supervision to achieve accurate alignment of cross-modal features. Finally, it constructs a total loss function that integrates single-modal supervision loss, multimodal contrastive loss, and multimodal fusion loss for end-to-end model training. The single-modal prediction supervision mechanism and modality reliability-aware multimodal contrastive learning strategy introduced in this invention can fully exploit the discriminative information of visual and linguistic single-modal features in different samples. Through confidence-driven sample pair partitioning and weighting, robust cross-modal joint representation learning is achieved, significantly reducing the interference of weaker discriminative modalities on model training and improving the model's adaptability and generalization ability to complex scenes, long videos, and videos with uneven semantic distribution. Furthermore, by jointly optimizing the single-modal supervision loss, multimodal fusion loss, and multimodal contrast loss, and introducing the multimodal fusion supervision signal of feature splicing, fine-grained semantic constraint modeling is performed on cross-modal matching samples. This effectively improves the discriminativeness and symmetry of cross-modal features, and enhances the model's adaptability and global ranking ability for imperfect text-video matching pairs in real-world application scenarios.

[0012] Experimental results show that the method of this invention significantly outperforms existing technologies on mainstream text-video retrieval datasets. This invention can effectively improve the semantic matching accuracy between text and video, enhance the robustness and generalization ability of the model in complex scenarios, and achieve efficient and stable cross-modal retrieval performance while ensuring low computational overhead. It achieves lightweight model design while possessing stronger video semantic understanding capabilities, cross-modal retrieval accuracy, and engineering application feasibility. It realizes high-precision bidirectional text-to-video retrieval under lightweight model design and is suitable for various application scenarios such as video retrieval, intelligent retrieval question answering, multimedia content analysis, and cross-modal text-video matching tasks. Attached Figure Description

[0013] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a flowchart of the text video retrieval method based on multi-level prompting learning and cross-modal supervision provided by the present invention; Figure 2 This is a framework diagram of the text video retrieval method based on multi-level prompting learning and cross-modal supervision provided by the present invention; Figure 3 This is a flowchart of the modal reliability constraint strategy module provided by the present invention; Figure 4 This is a schematic diagram of the modular retrieval system provided by the present invention; Figure 5 This is a partial schematic diagram of the video frame attention visualization provided by the present invention; Figure 6 This is a schematic diagram of the operation of the hardware device provided by the present invention. Detailed Implementation

[0014] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0015] Figure 1 A schematic diagram of the text video retrieval method based on multi-level cue learning and cross-modal supervision in this embodiment is shown. In this embodiment, the text video retrieval method based on multi-level cue learning and cross-modal supervision includes the following steps: S1: Construct a visual encoder and a text encoder based on the pre-trained CLIP model. The visual encoder is used to extract features from the video frame sequence to obtain visual feature representations, and the text encoder is used to perform sequence modeling of natural language descriptions to obtain text feature representations. In one exemplary embodiment, the visual feature is represented as:

[0016] in, Represents the set of feature vectors of a video. This indicates the number of video frames obtained from sampling. Indicates the feature embedding dimension. The dimension of the feature matrix. This represents the obtained global visual features. Indicates obtained through sampling Frame characteristics of video; The text features are represented as follows:

[0017] in, Represents the embedded sequence of text input. Indicates the number of text tokens. and These represent the start and end markers, respectively. Indicates the input text Embedding of individual tokens; S2: Obtain video-text pair samples, and use the visual encoder and text encoder to encode and aggregate the video-text pair samples to obtain text semantic features, frame-level visual features and video global representation; In one exemplary embodiment, step S2 specifically includes: S21: Obtain cross-modal video-text pair samples containing video and natural language description, wherein the video comes from multiple scene environments, including dynamic content with different backgrounds, lighting conditions and action changes, and the natural language description is natural language sentence text related to the semantics of the video; S22: Extract the original features of the video and natural language description using the visual encoder and text encoder respectively, including text semantic features, global video-level cues and frame-level visual features; S23: Based on the original features of the video and natural language description, a multi-level cue learning mechanism is used to construct a cue structure containing local frame-level cue words and global video-level cue words. The cue words are then embedded into the input sequence of the visual encoder and participate in attention calculation together with the video frame features. The cue vectors in the shared semantic space are used to enhance the semantic consistency between text features and video features, thereby obtaining a task-adaptive cross-modal semantic representation and a global video representation. S3: Based on the text semantic features, frame-level visual features and video global representation, the video representation features and text representation features are enhanced using a multi-level cueing learning module to obtain enhanced video representation features and text representation features. In one exemplary embodiment, the multi-level cue learning module is configured to: jointly model the frame-level visual features output by the visual encoder using local frame-level cues and global video-level cues, and optimize the parameter utilization efficiency of the attention mechanism using a parameter sharing strategy to obtain enhanced video representation features; and synchronously learn the text semantic features output by the text encoder to obtain enhanced text representation features. As an exemplary embodiment, step S3 specifically includes: S31: In Vit's Within the layer, along the time dimension Category embedding Embedded prompts in each frame and the token embedding in each frame By concatenating the vectors, we obtain the query vector. Meanwhile, to ensure that the information from each frame can perceive the global information of the video at the same stage, the global embedding should be added to the query vector based on the number of times the splicing operation is repeated according to the dimension of the global embedding. In the process, the key vector is obtained. Finally, the query vector Key vector Sum value vector Input to the multi-head attention mechanism (MHA), output As the query vector for the next layer of local frame-level hints, iterative updates of local frame-level features are completed; S32: For global video-level cue modeling, first, the global cue vector... as query vector Then the global hint vector Input text sequence Video frame prompts embedded and token embedding By combining them, we obtain the key vector. Finally, the query vector Key vector Sum value vector Input to the multi-head attention mechanism (MHA), output As an updated global video cue vector, it enables global video semantic modeling across frames; S33: For parameter sharing strategies, in the feature dimension Define local frame-level cue vectors and global video-level cue vectors Simultaneously, shared multi-head attention linear projection parameters are set in each encoding layer of both the visual encoder and the text encoder. , , Finally, linear projections are performed on the local frame-level cue vector and the global video-level cue vector respectively to obtain the local query input. and global query input The local attention and global attention mechanisms are differentiated functionally only at the query vector level.

[0018] S4: Based on the enhanced video representation features and text representation features, perform cross-modal supervision constraints to obtain positive sample pairs, negative sample pairs, imperfect sample pairs, and single-modal supervision loss; In one exemplary embodiment, step S4 specifically includes: S41: Based on the enhanced video representation features and text representation features, calculate the similarity matrix between text-retrieval video and the similarity matrix between video-retrieval text, as shown in the formula:

[0019]

[0020] in, and These represent the similarity matrices for text-to-video retrieval and video-to-text retrieval, respectively. and To enhance the video representation features, and For enhanced text representation features, Normalized cosine similarity; S42: Based on the similarity matrix, obtain positive sample pairs, negative sample pairs, and imperfect sample pairs; As an exemplary embodiment, specifically: for the first For a sample, if the following conditions are met If the text modality is considered to have reliable discriminative ability on this sample, then similarly, if the following conditions are met... If the video modality has a reliable discrimination ability on the sample, then the matching pairs are divided into three types based on this result: when both discrimination criteria are satisfied, they are called positive sample pairs; when neither discrimination criterion is satisfied, they are called negative sample pairs; when one discrimination criterion is satisfied and the other discrimination criterion is not satisfied, they are called imperfect sample pairs. S43: Based on the obtained positive sample pairs, negative sample pairs, and imperfect sample pairs, calculate the single-modal supervision loss, as shown in the formula:

[0021] in, For single-modal supervision loss; The number of modes; Number of categories; The category label is based on the sample pair, i.e., positive sample pair, negative sample pair, or imperfect sample pair; The predicted probability distribution of the modality; S5: Based on the positive sample pairs, negative sample pairs, and imperfect sample pairs, calculate the multimodal contrastive loss using multimodal contrastive learning; concatenate the video representation features and text representation features to obtain fusion features, supervise multimodal contrastive learning using the fusion features, and calculate the multimodal fusion loss; In one exemplary embodiment, the multimodal contrast loss is calculated using the following formula:

[0022] in, For multimodal contrast loss, Let P represent the confidence level in the unimodal discrimination of the i-th sample, where P is the set of positive sample pairs and I is the set of imperfect sample pairs. Represents the set of all sample pairs. Represents the normalized cosine similarity. This indicates temperature hyperparameters. Indicates the first Text modality embedding of each sample, Indicates the first Visual modality embedding of each sample, Indicates in set The Middle Text modality embedding of each sample, Indicates in set The Middle Visual modal embedding of each sample; In one exemplary embodiment, the multimodal fusion loss is calculated using the following formula:

[0023] in, For multimodal fusion loss; Number of categories; and These represent the category labels and the predicted probability distributions based on the fusion features, respectively. S6: Construct a total loss function based on the single-modal supervision loss, multimodal contrast loss, and multimodal fusion loss. Use the total loss function to train the text video retrieval model end-to-end to obtain the trained retrieval model. Use the trained retrieval model to perform text video retrieval. In one exemplary embodiment, the total loss function is calculated as follows:

[0024] in, For the total loss function, For hyperparameters; For single-modal supervision loss; For multimodal contrast loss, This represents the loss for multimodal fusion.

[0025] In some embodiments, the above-described text video retrieval method based on multi-level cue learning and cross-modal supervision can also be implemented in the following ways.

[0026] In this embodiment, the text video retrieval method based on multi-level prompting learning and cross-modal supervision includes the following steps: Step S100: Preprocess the video data and text description data to obtain standardized video frame features and text sequence features, and input them into the CLIP-based pre-trained visual encoder and text encoder; In one exemplary embodiment, in step S100, a text-to-video retrieval model framework is built on the pre-trained CLIP model, such as... Figure 2 As shown, the framework uses the encoder of a pre-trained CLIP model as a feature extractor. The specific steps are: prompting the learning framework... S101: Both the visual encoder and text encoder are built based on the pre-trained CLIP ViT-B / 32 architecture. The visual encoder uses a Vision Transformer (ViT) to extract features from the input video frame sequence, representing the video as a feature set composed of multiple frames. For the frame sequence obtained after frame sampling of the input video, its corresponding visual features are represented as follows:

[0027] in Represents the set of feature vectors of a video. This indicates the number of video frames obtained from sampling. Indicates the feature embedding dimension. The dimension of the feature matrix. Represents the global visual features obtained from [CLS] Indicates obtained through sampling Frame characteristics of video.

[0028] S102: The text encoder uses a Transformer structure to perform sequence modeling on the input natural language description, and combines a cue learning mechanism to enhance the text's semantic expressive ability. Its text feature representation is as follows:

[0029] in Represents the embedded sequence of text input. Indicates the number of text tokens. Indicates the input text Embedding of a token and They represent [SOS] and [EOS] respectively, and at the same time Used as a global semantic representation of the entire sentence for cross-modal alignment.

[0030] Step S200: Obtain video-text pair samples, perform frame sampling on the input video to obtain a video frame sequence, extract frame-level visual features using a frozen visual encoder, and perform fine-grained semantic modeling on each frame using local frame-level cues; at the same time, extract text semantic features using a text encoder and perform temporal aggregation on all frame features using global video-level cues to obtain a global video representation. In one exemplary embodiment, in step S200, video-text pair samples are acquired, and these features are processed to adapt to subsequent task requirements. The specific steps are as follows: S201: Obtain cross-modal data pairs containing video and natural language descriptions, wherein the video data comes from multiple scene environments and includes dynamic content with different backgrounds, lighting conditions and action changes, and the text descriptions are natural language statements related to the semantics of the video; S202: The CLIP pre-trained model is used to extract the original features of the video modality and the text modality respectively. The video is encoded by a visual encoder to generate a frame-level token sequence representation, and then a video-level feature representation is constructed. The text is generated by a Transformer encoder to generate a token-level embedding representation, and a start token [SOS] and an end token [EOS] are introduced into the text sequence to enhance the integrity of the semantic structure. S203: In the visual encoding process, a multi-level cue learning mechanism is introduced to construct a cue structure that includes local frame-level cue tokens and global video-level cue tokens. The cue tokens are then embedded into the input sequence of the visual encoder and participate in attention calculation together with video frame features, thereby achieving collaborative modeling of local video detail information and global temporal semantics. At the same time, in the text encoding process, cue vectors with a shared semantic space are introduced to enhance the semantic consistency between text features and video features, ultimately forming a task-adaptive cross-modal semantic representation.

[0031] Step S300: Construct a multi-level cue learning module to jointly model the video frame features output by the visual encoder using local frame-level cues and global video-level cues, and use a parameter sharing strategy to optimize the parameter utilization efficiency of the attention mechanism to obtain enhanced video representation features; perform synchronous cue learning on the text sequence features output by the text encoder to obtain enhanced text representation features. In one exemplary embodiment, in step S300, a multi-level cue learning module is constructed to obtain enhanced video representation features and text representation features, respectively. The specific steps are as follows: S301: In ViT's... Within the layer, along the time dimension Category embedding Embedded prompts in each frame and the token embedding in each frame By concatenating the vectors, we obtain the query vector. Meanwhile, to ensure that the information from each frame can perceive the global information of the video at the same stage, the global embedding should be added to the query vector based on the number of times the splicing operation is repeated according to the dimension of the global embedding. In the process, the key vector is obtained. Finally, the query vector Key vector Sum value vector Input to the multi-head attention mechanism (MHA), output As the query vector for the next layer of local frame-level hints, iterative updates of local frame-level features are completed; S302: For global video-level cue modeling, first, the global cue vector... as query vector Then the global hint vector Input text sequence Video frame prompts embedded and token embedding By combining them, we obtain the key vector. Finally, the query vector Key vector Sum value vector Input to the multi-head attention mechanism (MHA), output As an updated global video cue vector, it enables global video semantic modeling across frames; S303: For parameter sharing strategies, in the feature dimension Define local frame-level cue vectors and global video-level cue vectors Simultaneously, shared multi-head attention linear projection parameters are set in each encoding layer of both the visual encoder and the text encoder. , , Finally, linear projections are performed on the local frame-level cue vector and the global video-level cue vector respectively to obtain the local query input. and global query input The local attention and global attention mechanisms are differentiated functionally only at the query vector level.

[0032] Step S400: Build a cross-modal supervision and constraint module, evaluate the independent discrimination ability of video representation features and text representation features in each sample through single-modal prediction supervision, divide the sample pairs into positive sample pairs, negative sample pairs and imperfect sample pairs according to the discrimination results, and calculate the single-modal supervision loss; In one exemplary embodiment, in step S400, a cross-modal supervision and constraint module is built to evaluate the independent discriminative ability of each sample through single-modal prediction supervision, such as... Figure 3 As shown. The implementation steps are as follows: S401: In the feature dimension In the example, let the enhanced video representation features be... The enhanced text representation features are Construct similarity matrices for text-to-video retrieval respectively. Similarity matrix between video and text retrieval :

[0033]

[0034] in Normalized cosine similarity; S402: For the first For a sample, if the following conditions are met If the text modality is considered to have reliable discriminative ability on this sample, then similarly, if the following conditions are met... If the video modality is considered to have reliable discrimination ability on the sample, then the matching pairs are divided into three types based on this result: when both discrimination criteria are satisfied, they are called positive sample pairs; when neither discrimination criterion is satisfied, they are called negative sample pairs; when one discrimination criterion is satisfied and the other discrimination criterion is not satisfied, they are called imperfect pairs. S403: Class Labels Based on Sample Pairs Predicted probability distribution of modes Then there is a single-modal supervision loss:

[0035] in For the number of modes, This represents the number of categories.

[0036] Step S500: Construct a multimodal contrastive learning model with modal reliability perception capability based on the sample pair partitioning results, and calculate the multimodal contrastive loss; at the same time, concatenate the video representation features and text representation features to obtain fusion features, supervise the multimodal contrastive learning process through the fusion features, and calculate the multimodal fusion loss; In one exemplary embodiment, in step S500, a cross-modal alignment loss is constructed based on the single-modal supervision results, such as... Figure 3 As shown. The steps are as follows: S501: Let P be the set of positive sample pairs and I be the set of imperfect sample pairs. For the i-th sample, the corresponding multimodal contrastive learning loss is:

[0037] in This indicates the confidence level in single-mode discrimination. Represents the set of all sample pairs. Represents the normalized cosine similarity. This indicates temperature hyperparameters. Indicates the first Text modality embedding of each sample, Indicates the first Visual modality embedding of each sample, Indicates in set The Middle Text modality embedding of each sample, Indicates in set The Middle Visual modal embedding of each sample; S502: Concatenate the unimodal visual features and textual features to construct a fused feature representation:

[0038] Where ⊕ represents feature concatenation; S503: Category Labels Based on Fusion Features Predicted probability distribution of fusion features The multimodal contrast loss is calculated as follows:

[0039] in This represents the number of categories.

[0040] like Figure 4 The diagram shown is a schematic of a modular retrieval system.

[0041] Step S600: Construct the total loss function, which integrates the single-modal supervision loss, multimodal fusion loss, and multimodal contrast loss with hyperparameters, and trains the entire text video retrieval model end-to-end to obtain the trained retrieval model.

[0042] In one exemplary embodiment, in step S600, a total loss function for the reliability constraint strategy is constructed by fusing the single-modal supervision loss, multi-modal fusion loss, and multi-modal contrast loss. The steps are as follows: The overall loss function is mainly composed of the single-mode supervision loss of formula (5). Multimodal contrast loss of formula (6) The multimodal fusion loss of formula (8) Therefore, the loss function is calculated as follows:

[0043] in is a hyperparameter used to adjust the weights of multimodal contrastive learning during model training.

[0044] Figure 5 The image shown is a partial schematic diagram of the attention visualization for retrieving video frames.

[0045] The following is a detailed explanation of the relevant details of the invention: (1) Multi-level prompting learning modeling This invention aims to enhance the model's dual modeling capabilities for video temporal semantics and local frame-level details, particularly in video retrieval scenarios involving long videos, complex scenes, and uneven semantic distribution, thereby strengthening the robustness of cross-modal semantic alignment. The module first extracts preprocessed video frame features and text sequence features using CLIP-pretrained visual encoders (ViT) and text encoders (Transformer) to obtain initial visual-linguistic embedding representations. Subsequently, it designs global video-level and local frame-level learnable cue vectors, combining them with a multi-head attention mechanism to achieve joint modeling of multi-level cueing: local frame-level cueing is achieved by concatenating category embedding, frame cue embedding, and token embedding along the temporal dimension, enabling the interconnected perception of single-frame details and global video information; while global video-level cueing uses the global vector as the core query, fusing text sequence and frame-level features to achieve cross-frame overall video semantic modeling. Meanwhile, an attention parameter sharing strategy is introduced in each layer of the visual and text encoders, sharing multi-head attention linear projection parameters. Local and global attention are distinguished only at the query vector level. Through a lightweight cue learning mechanism, the encoder is guided to learn semantic features that are highly relevant to the video retrieval task without destroying the original input structure of the pre-trained model. This constructs a visual-linguistic feature representation with temporal correlation, thereby improving the model's ability to capture dynamic video behavior and cross-frame semantic correlation, and thus enhancing the robustness of the model in cross-modal retrieval.

[0046] (2) Cross-modal supervision and feature fusion This invention introduces a weighted mechanism of unimodal supervision and modality reliability perception, utilizing the independent discriminative confidence of visual and linguistic unimodal features to construct a highly reliable cross-modal training sample set. During training, the model evaluates the independent discriminative ability of visual and textual modalities in each sample by calculating the cosine similarity matrix between text-to-video and video-to-text retrieval. Based on the discriminative results, sample pairs are divided into positive, negative, and imperfect pairs to dynamically adjust the contribution weight of samples with different confidence levels to model training. Simultaneously, a unimodal constraint-multimodal fusion feature interaction mechanism is employed. Unimodal prediction supervision constrains the alignment quality of visual and linguistic features, filtering out interference from low-confidence modalities. Subsequently, late-stage multimodal fusion is completed through feature concatenation, generating a cross-modal joint representation that combines visual temporal features and textual semantic features. Finally, the fused features are used as a supervision signal to provide fine-grained constraints on the multimodal contrastive learning process. This method, which combines unimodal reliability weighting with multimodal feature fusion, can fully explore the potential semantic information in imperfect sample pairs, effectively reduce the impact of modality gap and noisy samples on training, improve the cross-modal matching accuracy between text and video, and enhance the model's adaptability to complex semantic expressions and imperfect matching pairs.

[0047] To verify the effectiveness of this invention, experiments were conducted on the mainstream text-based video retrieval public dataset MSR-VTT-9k: MSR-VTT, proposed by Microsoft researchers, is a large-scale dataset widely used for text-to-video retrieval tasks. This dataset includes 10,000 short videos covering 20 semantic categories such as music and sports news. Each video corresponds to approximately 20 manually annotated text descriptions, primarily concerning the characters, actions, and overall scene semantics within the video. According to the official dataset partitioning scheme, the training set contains 6,513 video clips, the test set contains 2,990 video clips, and the validation set contains the remaining 497 distinct video clips.

[0048] Evaluation Indicators: This method uses two evaluation indicators: (1) Rank-k accuracy. Rank-k is one of the most commonly used metrics in cross-modal retrieval tasks. It measures the probability that at least one of the first k retrieved images is a correct match. Specifically, given a query set... Let the first The rank of the sample matched by each query in the matching queue is Then the definition of Rank-k is:

[0049] in The value is 1 when the condition is met, and 0 otherwise. Therefore, the higher the Rank-k index, the better the retrieval performance. In addition, the most commonly used Rank-k accuracies are Rank-1, Rank-5, and Rank-10.

[0050] (2) Average Rank (MnR). In cross-modal retrieval tasks such as text-based video retrieval, besides the common Rank-k metric, MnR is also one of the important metrics for measuring the retrieval performance of a model. MnR is mainly used to reflect the average rank of correctly matched samples in the ranking queue, thus reflecting the model's overall ability to rank samples. Specifically, the model needs to rank all candidate samples according to similarity. Let the query... The ranking of the corresponding matched sample in the sorting results is Then MnR can be defined as:

[0051] in Indicates the number of query samples. Representing the The ranking of the matching samples in each query. Therefore, MnR does not concern itself with whether the target sample is ranked highly. Instead of considering the individual sample, the model considers its position in the complete ranking queue. Therefore, the smaller the value of MnR, the more the model tends to place correctly matched samples at the beginning of the matching queue, thus demonstrating the model's stronger ability to align across modalities.

[0052] Experimental indicators: Table 1: Evaluation of experimental performance on the MSR-VTT-9k dataset

[0053] Table 2: Comparison of Parameter Quantity and Memory Usage Performance

[0054] The experimental results in Tables 1 and 2 show that our proposed method is significantly superior to existing methods, improving the detection performance of text-to-video tasks and achieving a lightweight model design, thus demonstrating the superiority of our method.

[0055] Example 2 A text-to-video retrieval device 401, such as Figure 6 As shown, it includes: a processor 402 and a storage device 403; the processor 402 loads and executes the instructions and data in the storage device 403 to implement the text video retrieval method based on multi-level prompting learning and cross-modal supervision.

[0056] Example 3 A storage device that stores instructions and data for implementing the text video retrieval method based on multi-level cue learning and cross-modal supervision.

[0057] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A text video retrieval method based on multi-level cue learning and cross-modal supervision, characterized in that, Includes the following steps: S1: Construct a visual encoder and a text encoder based on the pre-trained CLIP model. The visual encoder is used to extract features from the video frame sequence to obtain visual feature representations, and the text encoder is used to perform sequence modeling of natural language descriptions to obtain text feature representations. S2: Obtain video-text pair samples, and use the visual encoder and text encoder to encode and aggregate the video-text pair samples to obtain text semantic features, frame-level visual features and video global representation; S3: Based on the text semantic features, frame-level visual features and video global representation, the video representation features and text representation features are enhanced using a multi-level cueing learning module to obtain enhanced video representation features and text representation features. S4: Based on the enhanced video representation features and text representation features, perform cross-modal supervision constraints to obtain positive sample pairs, negative sample pairs, imperfect sample pairs, and single-modal supervision loss; S5: Based on the positive sample pairs, negative sample pairs, and imperfect sample pairs, calculate the multimodal contrastive loss using multimodal contrastive learning; concatenate the video representation features and text representation features to obtain fusion features, supervise multimodal contrastive learning using the fusion features, and calculate the multimodal fusion loss; S6: Construct a total loss function based on the single-modal supervision loss, multimodal contrast loss, and multimodal fusion loss. Use the total loss function to train the text video retrieval model end-to-end to obtain the trained retrieval model. Use the trained retrieval model to perform text video retrieval.

2. The text video retrieval method based on multi-level prompting learning and cross-modal supervision according to claim 1, characterized in that, The visual features are represented as follows: in, Represents the set of feature vectors of a video. This indicates the number of video frames obtained from sampling. Indicates the feature embedding dimension. The dimension of the feature matrix. This represents the obtained global visual features. Indicates obtained through sampling Frame characteristics of video; The text features are represented as follows: in, Represents the embedded sequence of text input. Indicates the number of text tokens. and These represent the start and end markers, respectively. Indicates the input text Embedding of individual tokens.

3. The text video retrieval method based on multi-level prompting learning and cross-modal supervision according to claim 1, characterized in that, Step S2 specifically includes: S21: Obtain cross-modal video-text pair samples containing video and natural language description, wherein the video comes from multiple scene environments, including dynamic content with different backgrounds, lighting conditions and action changes, and the natural language description is natural language sentence text related to the semantics of the video; S22: Extract the original features of the video and natural language description using the visual encoder and text encoder respectively, including text semantic features, global video-level cues and frame-level visual features; S23: Based on the original features of the video and natural language description, a multi-level cue learning mechanism is used to construct a cue structure containing local frame-level cue words and global video-level cue words. The cue words are then embedded into the input sequence of the visual encoder and participate in attention calculation together with the video frame features. The cue vectors in the shared semantic space are used to enhance the semantic consistency between text features and video features, thereby obtaining a task-adaptive cross-modal semantic representation and a global video representation.

4. The text video retrieval method based on multi-level prompting learning and cross-modal supervision according to claim 1, characterized in that, The multi-level cue learning module is configured to: jointly model the frame-level visual features output by the visual encoder using local frame-level cues and global video-level cues, and optimize the parameter utilization efficiency of the attention mechanism using a parameter sharing strategy to obtain enhanced video representation features. Synchronous cue learning is performed on the semantic features of the text output by the text encoder to obtain enhanced text representation features.

5. The text video retrieval method based on multi-level prompting learning and cross-modal supervision according to claim 1, characterized in that, Step S4 specifically includes: S41: Based on the enhanced video representation features and text representation features, calculate the similarity matrix between text-retrieval video and the similarity matrix between video-retrieval text, as shown in the formula: in, and These represent the similarity matrices for text-to-video retrieval and video-to-text retrieval, respectively. and To enhance the video representation features, and For enhanced text representation features, Normalized cosine similarity; S42: Based on the similarity matrix, obtain positive sample pairs, negative sample pairs, and imperfect sample pairs; S43: Based on the obtained positive sample pairs, negative sample pairs, and imperfect sample pairs, calculate the single-modal supervision loss, as shown in the formula: in, For single-modal supervision loss; The number of modes; Number of categories; The category label is based on the sample pair, i.e., positive sample pair, negative sample pair, or imperfect sample pair; Let be the predicted probability distribution of the modality.

6. The text video retrieval method based on multi-level prompting learning and cross-modal supervision according to claim 1, characterized in that, The formula for calculating the multimodal contrast loss is as follows: in, For multimodal contrast loss, Indicates the first In the single-modal discrimination of a sample, P represents the set of positive sample pairs, and I represents the set of imperfect sample pairs. Represents the set of all sample pairs. Represents the normalized cosine similarity. This indicates temperature hyperparameters. Indicates the first Text modality embedding of each sample, Indicates the first Visual modality embedding of each sample, Indicates in set The Middle Text modality embedding of each sample, Indicates in set The Middle Visual modal embedding of each sample.

7. The text video retrieval method based on multi-level prompting learning and cross-modal supervision according to claim 1, characterized in that, The formula for calculating the multimodal fusion loss is as follows: in, For multimodal fusion loss; Number of categories; and These represent the predicted probability distributions of the category labels and the fusion features, respectively.

8. The text video retrieval method based on multi-level prompting learning and cross-modal supervision according to claim 1, characterized in that, The formula for calculating the total loss function is as follows: in, For the total loss function, For hyperparameters; For single-modal supervision loss; For multimodal contrast loss, This represents the loss for multimodal fusion.

9. A storage device, characterized in that: The storage device stores instructions and data for implementing the text video retrieval method based on multi-level prompting learning and cross-modal supervision as described in any one of claims 1 to 8.

10. A retrieval device, characterized in that: include: A processor and a storage device; the processor loads and executes instructions and data in the storage device to implement the text video retrieval method based on multi-level cue learning and cross-modal supervision as described in any one of claims 1 to 8.

Images