A multimodal video retrieval method and system based on low-rank fine-grained hints

Abstract

The present invention discloses a multimodal video retrieval method and system based on low-rank fine-grained prompts, which belongs to the field of multimodal video retrieval. A multimodal model containing video and subtitle representation generation modules is pre-trained; a prompt update module is introduced into each layer of the N-1 layers of encoders before the video representation generation module, and fine-grained prompts of each modality of each layer are generated after fine-tuning. During fine-tuning, the video features of each modality and the corresponding prompts are spliced into a multimodal input and enter the current encoder layer. At the same time, the prompts of each modality are spliced into the prompt update module for update, and the updated prompts replace the corresponding prompts in the output of the current encoder layer; the prompts of each modality obtained by fine-tuning are fixed, and the video features of each modality output by the previous encoder and the prompts of each modality of the current layer are spliced in the video representation generation module as the multimodal input of the current layer. The final video representation is used to match the subtitle representation. The present invention can realize multimodal prompt learning with scalable modality number and type.

Description

Technical Field

[0001] The present invention relates to the field of multimodal video retrieval, and in particular to a multimodal video retrieval method and system based on low-rank fine-grained prompts. Background Art

[0002] Multimodal tasks involve information from multiple modalities (such as text, images, and speech), leveraging technologies from various fields, including computer vision, natural language processing, and audio processing. For AI to better solve real-world problems, it is necessary to enhance its capabilities in multimodal tasks—that is, to understand, process, and relate information from multiple modalities. In recent years, multimodal learning has gradually demonstrated its enormous potential and importance, playing an indispensable role in fields such as gaming, robotics, and education.

[0003] Existing research on multimodal learning technology can be divided into the construction of large multimodal language models and the research on lightweight methods.

[0004] The emergence of large language models and the limitations of traditional large language models in processing non-text data have made the construction of multimodal large language models a research hotspot. Such methods typically solve specific multimodal tasks by designing sophisticated but large and complex model structures. However, they are often difficult to adapt to different tasks, have high computational costs, and are difficult to fine-tune. Research on lightweight methods is exemplified by prompt learning, which adapts downstream tasks to pre-trained models by adjusting a small number of learnable parameters. The lightweight and efficient advantages of prompt learning have led to its generalization to the multimodal field. However, current multimodal prompt learning methods generally only perform shallow interactions between two specific modalities and, due to the limitations of the model structure, are difficult to generalize to other numbers and types of modalities. Summary of the Invention

[0005] In order to overcome the problems that existing technologies are limited to a fixed number and type of modalities and do not conduct deep interaction on multimodal information, the present invention provides a multimodal video retrieval method and system based on low-rank fine-grained prompts to achieve multimodal prompt learning with scalable number and type of modalities.

[0006] The specific technical solution adopted in the present invention is:

[0007] In a first aspect, the present invention proposes a multimodal video retrieval method based on low-rank fine-grained hints for matching videos and subtitles. The multimodal video retrieval method comprises:

[0008] (1) Pre-train a multimodal model that includes a video representation generation module and a subtitle representation generation module;

[0009] (2) In the video representation generation module, a prompt update module is introduced into each layer of the first N-1 layers of the encoder to generate the corresponding modal prompts in the form of fine-grained prompts; where N represents the total number of encoder layers;

[0010] During the fine-tuning and training of the prompt update module, the video features of each modality are concatenated with the corresponding modal prompts to obtain a multimodal input. The multimodal input enters the current encoder layer for processing. At the same time, the modal prompts input to the current encoder layer are concatenated with each other and enter the prompt update module of the same layer to update the modal prompts. The updated modal prompts replace the corresponding modal prompts in the output of the current encoder layer and are partially concatenated with the video features of each modality in the output result of the current encoder layer to form the multimodal input of the next encoder layer. The modal prompts entering the last encoder layer do not need to be updated again, and the last token of the output result of the last encoder layer is converted into a video representation for matching the subtitle representation.

[0011] (3) Fix the fine-tuned modal prompts and concatenate the modal video feature parts of the previous encoder output results with the modal prompts of the current layer in the video representation generation module as the multimodal input of the current layer; the video representation finally obtained by the video representation generation module is used to match the subtitle representation to achieve the multimodal video retrieval task.

[0012] Furthermore, in step (1), the pre-training of the multimodal model includes:

[0013] Obtaining video representations of candidate videos using a video representation generation module;

[0014] Obtaining subtitle representations of candidate subtitles using a subtitle representation generation module;

[0015] Based on the similarity between subtitle representation and video representation, the contrastive learning method is used to pre-train the video representation generation module and the subtitle representation generation module.

[0016] Furthermore, the video representation generation module includes a multimodal feature extraction network, a dimension adjustment layer, a backbone network composed of N layers of encoders, and a fully connected layer. During the pre-training process of the multimodal model, the multimodal initial video features generated by the multimodal feature extraction network are unified in dimension by the dimension adjustment layer, and then spliced and input into the backbone network. The last token of the backbone network output result is passed through the fully connected layer to obtain the video representation.

[0017] Furthermore, the multimodal initial video features include at least two modality features of an audio modality, a visual modality, and an optical character recognition modality.

[0018] Furthermore, during the fine-tuning training prompt update module, the prompts of each modality input to the first encoder layer are randomly initialized.

[0019] Furthermore, the multimodal input is represented as:

[0020] ;

[0021] ;

[0022] ;

[0023] in, represents the multimodal input of the i+1th encoder layer, represents the mode of the i-th encoder layer Tips, is the total number of modes, represents the mode of the i-th encoder layer Tips, Represents the mode of input to the i-th encoder layer and the i+1-th encoder layer respectively Video features, represents the i-th encoder layer, Indicates a prompt to update the module.

[0024] Furthermore, the prompt update module generates each modal prompt of the corresponding layer in a fine-grained prompt manner, which is expressed as:

[0025] ;

[0026] ;

[0027] ;

[0028] in, represents the i-th encoder layer Token update weight, express No. tokens, represents element-wise multiplication, express Middle The relationship between tokens and the different tokens in the remaining n-1 modal prompts, Representation Importance rank matrix, represents tensor multiplication of multiple matrices, Indicates the The modality in the input of the encoder layer Tips, It means summing the dimensions of the high-order matrix except e.

[0029] Further, Decompose into A rank-1 tensor implements an approximation of fine-grained hints and updates weights Simplified to:

[0030] ;

[0031] in, represents the number of ranks, Indicates the corresponding mode No. The dimensions are Tensor of Indicates the The dimensions are Tensor of and Indicates dimension, superscript T indicates transposition, represents the sum of dimension L, represents the mode of the i-th encoder layer Tips.

[0032] Furthermore, the subtitle representation generation module adopts a pre-trained BERT-based encoder structure.

[0033] In a second aspect, the present invention proposes a multimodal video retrieval system based on low-rank fine-grained prompts, which is used to implement the above-mentioned multimodal video retrieval method based on low-rank fine-grained prompts.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The multimodal video retrieval method based on low-rank fine-grained prompts proposed in the present invention introduces fine-grained prompts at the encoder layer to achieve deep interaction with modal prompts. The prompt update module is independent of the model backbone. The multimodal model is first pre-trained, and then the prompt update module is fine-tuned under the condition of fixed parameters of the pre-trained multimodal model. The number and type of modalities are not restricted. The modal prompts obtained after fine-tuning training are directly used for multimodal video retrieval tasks, realizing efficient and flexible expansion of multimodal learning in the number and type of modalities, and improving the effect of multimodal video retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 is a framework diagram of the multimodal video retrieval method based on low-rank fine-grained hints of the present invention;

[0037] Figure 2 is a schematic diagram of the fine-grained prompt generation process of the present invention;

[0038] Figure 3 Schematic diagram of the low-rank fine-grained prompt generation process of the present invention. DETAILED DESCRIPTION

[0039] The present invention will be further described and illustrated below in conjunction with specific embodiments. The embodiments are merely illustrative of the present disclosure and do not limit its scope. The technical features of the various embodiments of the present invention may be combined accordingly, provided that there is no conflict between them.

[0040] The accompanying drawings are merely schematic illustrations of the present invention and are not necessarily drawn to scale. Some of the blocks shown in the accompanying drawings are functional entities that do not necessarily correspond to physically or logically separate entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0041] like Figure 1 As shown in FIG, the present invention proposes a multimodal video retrieval method based on low-rank fine-grained hints, which mainly includes the following steps:

[0042] S1, pre-training stage

[0043] Pre-train a multimodal model that includes a video representation generation module and a subtitle representation generation module;

[0044] S2, fine-tuning training phase

[0045] In the video representation generation module, a prompt update module is introduced for each of the first N-1 encoder layers to generate the corresponding modality prompts in a fine-grained manner; where N represents the total number of encoder layers;

[0046] S3, inference stage

[0047] The modal prompts obtained by fixed fine-tuning are concatenated in the video representation generation module with the modal video feature parts of the previous encoder output results and the modal prompts of the current layer as the multimodal input of the current layer; the video representation finally obtained by the video representation generation module is used to match the subtitle representation to achieve the multimodal video retrieval task.

[0048] In a specific implementation of the present invention, an optional manner of the multimodal model is as follows:

[0049] The video representation generation module in the multimodal model includes a multimodal feature extraction network, a dimensionality adjustment layer, a backbone network consisting of N layers of encoders, and a fully connected layer. Preferably, the backbone network consisting of N layers of encoders adopts an N-layer Transformer encoder structure. The backbone model composed of Transformer encoders is a general model with strong understanding and generalization capabilities, capable of extracting rich semantic information from the input. During the pre-training process of the multimodal model, the multimodal initial video features generated by the multimodal feature extraction network are unified in dimension by the dimensionality adjustment layer, then spliced and input into the backbone network. The last token of the backbone network output result is passed through the fully connected layer to obtain the video representation.

[0050] For example, a multimodal feature extraction network is a feature extractor used to extract initial features from different video modalities. For example, for videos containing people, a ResNet50 model pre-trained on the VGGFace2 dataset is used as a facial feature extractor to extract facial features from the original video sample as initial features for the visual modality. A VGGish model pre-trained on the YouTube-8m dataset is used as an audio feature extractor to extract audio features from the original video sample, generating initial features for the audio modality. A I3D model pre-trained on the Kinetics dataset is used as an action feature extractor to extract action features from the original video sample as initial features for the action modality. However, these pre-trained models and modal features are not limited to the aforementioned ones; commonly used modalities include optical character recognition, visual (e.g., action, scene, appearance, face, etc.), and audio. The definitions and extraction processes of each modality are common knowledge in the field and will not be elaborated here.

[0051] The multimodal video features obtained by the feature extractor above need to be unified in feature dimension to facilitate subsequent processing. An optional way to obtain them is as follows:

[0052] The initial features of each mode Input a one-dimensional convolutional layer, where and Represents the mode Initial features The sequence length and feature dimension of Convert to a unified target dimension In this embodiment, the size of the convolution kernel is set to 1 to ensure that only the feature dimension is transformed without changing the sequence length. The expression is , the feature matrix size of the final output of the one-dimensional convolutional layer is , thereby completing the dimensional alignment of each modal feature. The structure of unifying feature dimensions is not limited to the above-mentioned one-dimensional convolutional layer, but can also be a fully connected layer or other linear projection methods.

[0053] The subtitle representation generation module in the multimodal model can adopt a pre-trained BERT-based encoder structure. For example, the subtitles are first encoded through BERT, and then the encoded features are mapped through a fully connected layer to obtain a subtitle representation with the same dimension as the video representation for subsequent similarity calculation.

[0054] In the above S1, the original video sample and text annotation are obtained. The text annotation refers to the subtitle of the original video sample. In this embodiment, pre-training is performed using contrastive learning. The loss function is calculated as follows:

[0055]

[0056] in, is the bidirectional maximum margin sorting loss function; is the sample size; is a video representation and subtitle representation Similarity score between (negative sample pairs); is a video representation Corresponding correct subtitle representation Similarity score between (positive sample pairs); Representation interval is a hyperparameter used to control the minimum gap between the similarity of positive sample pairs and the similarity of negative sample pairs. Refers to the result generated by the above video representation generation module, subtitle representation This refers to the result of subtitle text being generated by the aforementioned subtitle representation generation module, as described above. The principles of contrastive learning are common knowledge in the field and will not be elaborated on here. Those skilled in the art may also introduce other training methods and loss functions suitable for multimodal video retrieval tasks.

[0057] In one embodiment of the present invention, a prompt update module is introduced into each of the first N-1 encoder layers in the video representation generation module. The prompt update module is independent of the backbone model to enhance flexibility. During fine-tuning and training of the prompt update module, the parameters of the multimodal feature extraction network pre-trained in the first stage, the backbone network composed of N encoder layers, and the subtitle representation generation module are fixed and no longer updated. The prompt update module, as well as the dimension adjustment layer and fully connected layer in the video representation generation module, are fine-tuned and trained. Each modal video feature is concatenated with the corresponding modal prompt to obtain a multimodal input. The multimodal input enters the current encoder layer for processing. At the same time, the modal prompts input to the current encoder layer are concatenated with each other and enter the prompt update module of the same layer to update the modal prompts. The updated modal prompts replace the corresponding modal prompts in the output of the current encoder layer and are concatenated with the modal video features in the output of the current encoder layer to form the multimodal input of the next encoder layer. The modal prompts entering the last encoder layer do not need to be updated again. The last token of the output of the last encoder layer is converted into a video representation for matching the subtitle representation.

[0058] When fine-tuning the prompt update module, the same loss function as the first stage is used, and the contrastive learning method is used to fine-tune the prompt update module to generate each modal prompt of the corresponding layer in the form of fine-grained prompts.

[0059] An optional way to obtain multimodal input is as follows:

[0060] Each modal feature is concatenated with the hint of the modality, and then concatenated with each other to obtain a multimodal input, which is expressed as:

[0061]

[0062] in, Represents multimodal input; is the total number of modalities of the original video sample; Represents a splicing operation, Indicates the prompt of modal n, and Respectively The sequence length and feature dimension of each modality input to the first encoder layer are randomly initialized.

[0063] Furthermore, the present invention generates each modal prompt of the corresponding layer in a fine-grained prompt manner. An optional way for the multimodal input to enter the backbone network encoder layer for processing is as follows:

[0064] The calculation formula is:

[0065]

[0066]

[0067]

[0068] in, Indicates the modality in the multimodal input of the encoder layer Tips; is the total number of modalities of the original video sample; Indicates a prompt to update the module; represents the i-th Transformer encoder layer; Indicates the Multimodal input to the encoder layers.

[0069] An alternative way to prompt for module updates is as follows:

[0070] The fine-grained prompt method captures the interaction between each modal prompt and supports the flexible expansion of modal types and quantities; Figure 2 As shown, an alternative approach to fine-grained hinting is as follows:

[0071] Each modal prompt is updated using update weights within the prompt update module. Each update weight is consistent with the prompt size, ensuring that each element in the prompt is updated with the corresponding update weight value, achieving fine-grained prompt updates.

[0072] The calculation formula is:

[0073]

[0074]

[0075] in, express No. tokens; yes The target update weight is yes Token update weight; Represents element-wise multiplication; yes The sequence length.

[0076] An optional way to update the weight acquisition process is as follows:

[0077] First, construct a high-order matrix that integrates the prompt information of each modality , the calculation formula is:

[0078]

[0079]

[0080] in, Defined as the element-wise multiplication of a sequence of tensors; and Defined as "tensor multiplication" of multiple matrices.

[0081] Next, use express ,have:

[0082]

[0083] in, express Each token in Element-wise multiplication of ; Indicates only Time and conduct Operation, express Middle The relationship between the token and the different tokens in the remaining n-1 modal prompts.

[0084] at last, The target update weight can be obtained by the following formula:

[0085]

[0086] in, Representation Importance rank matrix; It means summing the dimensions of the high-order matrix except e.

[0087] In a specific implementation of the present invention, a low-rank fine-grained cueing method is used to approximate fine-grained cues using low-rank decomposition, reducing the complexity of fine-grained cues, which grows exponentially with the number of modalities, to a linear level.

[0088] like Figure 3 As shown, an alternative approach to the low-rank fine-grained hinting method is as follows:

[0089] Apply low-rank decomposition to reduce the fine-grained hinting method The complexity brought about. Can be decomposed into A rank-1 tensor:

[0090]

[0091] in, express tensor of ; express Tensor of Represents the transposed Tensor of Indicates the number of ranks.

[0092] Therefore, the update weight formula in the fine-grained hint method can be rewritten as follows:

[0093]

[0094]

[0095] The present invention uses a high-resource dataset to pre-train the entire framework, then freezes most of the framework parameters, and uses a low-resource dataset to fine-tune the prompt update module. During the fine-tuning training, the corresponding loss function and the gradient descent learning method are used to fine-tune the trainable parameters in the framework (including the dimension adjustment layer, the prompt update module, and the fully connected layer), so that the model can complete the multimodal video retrieval task.

[0096] The above method is applied to the following examples to demonstrate the technical effects of the present invention, and the specific steps in the examples are not repeated here.

[0097] The present invention conducts experiments on two multimodal datasets, MSR-VTT and MSVD. MSR-VTT (a high-resource dataset, used for pre-training) and MSVD are used to evaluate the multimodal video retrieval task.

[0098] In order to objectively evaluate the performance of the present invention, the present invention uses the Rank@K (R@K), Mean Rank (MeanR) and Median Rank (MedR) evaluation criteria to evaluate the video retrieval effect in the selected dataset. Among them, except for R@K, the lower the index value, the better the performance of the model.

[0099] To verify the effectiveness of this invention, we conducted comparative experiments with the MMT model in 2 to 5 modalities. The tasks were divided into two categories: (1) given a text query, retrieve the most relevant video from a video collection (Text to Video); (2) given a video content, retrieve the most relevant text description (Video to Text).

[0100] To verify the effectiveness of the present invention in reducing complexity, this implementation counted the actual floating-point operations (FLOPs) of the hint update module and the memory space requirements of the entire model for the fine-grained hinting method and the low-rank fine-grained hinting method in the present invention under 4 to 5 modes on the MSVD dataset.

[0101] According to the steps described in the specific implementation method, the experimental results obtained are shown in Tables 1 to 3. In the present invention, the fine-grained prompt method is expressed as Fine-grained Prompt (FP), and the low-rank fine-grained prompt method is expressed as Low-rank Fine-grained Prompt (LFP).

[0102] Table 1: Pre-training results of multimodal video retrieval obtained by the present invention for the MSR-VTT dataset

[0103]

[0104] Table 2: Evaluation results of multimodal video retrieval obtained by the present invention for the MSVD dataset

[0105]

[0106] Table 3: Statistics of computing resources for the MSVD dataset of the present invention

[0107]

[0108] It can be seen from Tables 1 and 2 that the proposed method outperforms the baseline model in all indicators under all modal configurations, highlighting the effectiveness of the proposed method in multimodal video retrieval tasks and its adaptability to different numbers of modalities.

[0109] As can be seen from Table 3, the actual number of floating-point operations and storage space requirements of the fine-grained hinting method are significantly higher than those of the low-rank fine-grained hinting method. The low-rank fine-grained hinting method in this invention has obvious advantages in terms of computational cost.

[0110] Combining Tables 2 and 3, it can be found that although the low-rank fine-grained hinting method in the present invention is slightly inferior to the fine-grained hinting method in the present invention in some indicators, it has obvious advantages in terms of computational cost, which demonstrates the low-rank fine-grained hinting method in the present invention's truly feasible modal scalability in practical applications.

[0111] This embodiment also provides a modality-scalable, low-rank, fine-grained prompting system for implementing the above-mentioned embodiments. The terms "module," "unit," and the like used below may refer to a combination of software and / or hardware that implements a predetermined function. Although the systems described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible.

[0112] The above system includes:

[0113] A multimodal model, which includes a video representation generation module and a subtitle representation generation module;

[0114] A hint update module, which is introduced into each layer of the first N-1 encoder layers in the video representation generation module;

[0115] A fine-tuning training module is used to fine-tune the training prompt update module based on the pre-trained multimodal model to generate prompts for each modality of the corresponding layer in a fine-grained prompt manner;

[0116] The multimodal video retrieval module fixes and fine-tunes the modal prompts obtained. In the video representation generation module, the video feature parts of each modality in the previous encoder output result are spliced with the modal prompts of the current layer as the multimodal input of the current layer; the video representation finally obtained by the video representation generation module is used to match the subtitle representation to achieve the multimodal video retrieval task.

[0117] As for the system embodiment, since it basically corresponds to the method embodiment, the relevant parts can be referred to the partial description of the method embodiment, and the implementation methods of the remaining modules will not be repeated here. The system embodiment described above is only illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of the present invention. Ordinary technicians in this field can understand and implement it without paying any creative work.

[0118] Embodiments of the system of the present invention can be applied to any device with data processing capabilities, such as a computer or other device. System embodiments can be implemented through software, hardware, or a combination of software and hardware. For example, a software implementation, as a logical device, is implemented by a processor of any device with data processing capabilities, reading corresponding computer program instructions from non-volatile memory into internal memory and executing them.

[0119] The above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and many variations are possible. All variations that can be directly derived or imagined by a person skilled in the art from the disclosure of the present invention should be considered to be within the scope of protection of the present invention.

Claims

1. A multimodal video retrieval method based on low-rank fine-grained cues for matching videos and subtitles, characterized by: The multimodal video retrieval method includes: (1) Pre-train a multimodal model that includes a video representation generation module and a subtitle representation generation module; (2) In the video representation generation module, a prompt update module is introduced into each layer of the first N-1 layers of the encoder to generate the corresponding modal prompts in the form of fine-grained prompts; where N represents the total number of encoder layers; During the fine-tuning and training of the prompt update module, the video features of each modality are concatenated with the corresponding modal prompts to obtain a multimodal input. The multimodal input enters the current encoder layer for processing. At the same time, the modal prompts input to the current encoder layer are concatenated with each other and enter the prompt update module of the same layer to update the modal prompts. The updated modal prompts replace the corresponding modal prompts in the output of the current encoder layer and are partially concatenated with the video features of each modality in the output result of the current encoder layer to form the multimodal input of the next encoder layer. The modal prompts entering the last encoder layer do not need to be updated again, and the last token of the output result of the last encoder layer is converted into a video representation for matching the subtitle representation. (3) Fix the fine-tuned modal prompts and concatenate the modal video feature parts of the previous encoder output results with the modal prompts of the current layer in the video representation generation module as the multimodal input of the current layer; the video representation finally obtained by the video representation generation module is used to match the subtitle representation to achieve the multimodal video retrieval task.

2. The multimodal video retrieval method based on low-rank fine-grained hints according to claim 1, characterized in that In step (1), the pre-training of the multimodal model includes: Obtaining video representations of candidate videos using a video representation generation module; Obtaining subtitle representations of candidate subtitles using a subtitle representation generation module; Based on the similarity between subtitle representation and video representation, the contrastive learning method is used to pre-train the video representation generation module and the subtitle representation generation module.

3. The multimodal video retrieval method based on low-rank fine-grained hints according to claim 1 or 2, characterized in that: The video representation generation module includes a multimodal feature extraction network, a dimension adjustment layer, a backbone network composed of N layers of encoders, and a fully connected layer. During the pre-training process of the multimodal model, the multimodal initial video features generated by the multimodal feature extraction network are unified in dimension by the dimension adjustment layer, and then spliced and input into the backbone network. The last token of the backbone network output result is passed through the fully connected layer to obtain the video representation.

4. The multimodal video retrieval method based on low-rank fine-grained hints according to claim 3, characterized in that The multimodal initial video features include at least two modality features of an audio modality, a visual modality, and an optical character recognition modality.

5. The multimodal video retrieval method based on low-rank fine-grained hints according to claim 1, characterized in that During the fine-tuning training of the prompt update module, the prompts of each modality input to the first encoder layer are randomly initialized.

6. The multimodal video retrieval method based on low-rank fine-grained hints according to claim 1, characterized in that Multimodal input is represented as: ； ； ； in, represents the multimodal input of the i+1th encoder layer, represents the mode of the i-th encoder layer Tips, is the total number of modes, represents the mode of the i-th encoder layer Tips, Represents the mode of input to the i-th encoder layer and the i+1-th encoder layer respectively Video features, represents the i-th encoder layer, Indicates a prompt to update the module. Represents a splicing operation.

7. The multimodal video retrieval method based on low-rank fine-grained hints according to claim 6, characterized in that The prompt update module generates each modal prompt of the corresponding layer in a fine-grained prompt manner, which is expressed as: ； ； ； in, represents the i-th encoder layer Token update weight, express No. tokens, represents element-wise multiplication, express Middle The relationship between tokens and the different tokens in the remaining n-1 modal prompts, Representation Importance rank matrix, represents tensor multiplication of multiple matrices, Indicates the The modality in the input of the encoder layer Tips, It means summing the dimensions of the high-order matrix except e. and Represents dimension.

8. The multimodal video retrieval method based on low-rank fine-grained hints according to claim 7, characterized in that: Will Decompose into A rank-1 tensor implements an approximation of fine-grained hints and updates weights Simplified to: ； in, represents the number of ranks, Indicates the corresponding mode No. The dimensions are Tensor of Indicates the The dimensions are Tensor of and Indicates dimension, superscript T indicates transposition, represents the sum of dimension L, represents the mode of the i-th encoder layer Tips.

9. The multimodal video retrieval method based on low-rank fine-grained hints according to claim 1, characterized in that The subtitle representation generation module adopts a pre-trained BERT-based encoder structure.

10. A multimodal video retrieval system based on low-rank fine-grained hints, used to implement the method of claim 1, characterized in that: The system comprises: A multimodal model, which includes a video representation generation module and a subtitle representation generation module; A hint update module, which is introduced into each layer of the first N-1 encoder layers in the video representation generation module; A fine-tuning training module is used to fine-tune the training prompt update module based on the pre-trained multimodal model to generate prompts for each modality of the corresponding layer in a fine-grained prompt manner; The multimodal video retrieval module fixes and fine-tunes the modal prompts obtained. In the video representation generation module, the video feature parts of each modality in the previous encoder output result are spliced with the modal prompts of the current layer as the multimodal input of the current layer; the video representation finally obtained by the video representation generation module is used to match the subtitle representation to achieve the multimodal video retrieval task.

Images