Text-video retrieval method and system based on multi-granularity learnable interaction

Abstract

The invention discloses a text-video retrieval method and system based on multi-granularity learnable interaction, and relates to the field of data retrieval, and the method comprises the following steps: encoding a query text and an unpaired video in to-be-retrieved data, and extracting query text features and video features; obtaining a plurality of learnable vectors, carrying out cross-modal interaction on the learnable vectors and fine-grained video features, carrying out cross-grained alignment on query text features and learnable variables, carrying out fine-grained alignment on each word feature and the learnable vector, and aligning the query text features and video features; and capturing fine-granularity alignment information and coarse-granularity alignment information, capturing weights of different similar vectors among modals, summing multiple similarity scores based on the weights to obtain final multi-granularity similarity scores, and sorting the final multi-granularity similarity scores to obtain a final retrieval result. According to the method, cross-modal interaction and other operations are carried out through the plurality of learnable vectors and the fine-grained visual features, the fine-grained visual information is utilized, and the retrieval accuracy is improved.

Description

Technical Field

[0001] The present invention relates to the field of data retrieval, and in particular to a text-video retrieval method and system based on multi-granularity learnable interaction. Background Art

[0002] Significant advances in acquisition technology and device capabilities have driven the exponential growth of multimodal data on the internet, including images, videos, and text. Text-video pairs are a common combination in multimodal data. Video retrieval using text as a query has become a de facto paradigm for cross-modal access to multimedia content. It not only helps users find specific content of interest within massive amounts of data but also facilitates content classification and personalized recommendations on back-end servers. Therefore, efficient and robust text-video retrieval has become a key area of ​​multimodal research.

[0003] In recent years, CLIP (Contrastive Language-Image Pre-training), a large-scale image-text pre-training model using contrastive learning, has achieved remarkable results in both computer vision and natural language processing. This is mainly due to the fact that CLIP is trained on approximately 400 million pairs of image-text data, which enables the model to fully learn the deep semantic associations and feature mapping capabilities between modalities, thereby demonstrating strong generalization and robustness in a variety of visual and language tasks. Based on this, researchers have transferred the image-text knowledge pre-trained by CLIP to the field of text-video cross-modal retrieval and constructed a general paradigm: (1) feature extraction using CLIP text encoder and visual encoder; (2) cross-modal alignment; (3) training strategy. Recent mainstream methods can be divided into coarse-grained alignment, fine-grained alignment, and multi-grained alignment according to the alignment granularity.

[0004] Coarse-grained alignment typically aligns the query text with the entire video or video frame. For example, CLIP4Clip treats video frames as images, encodes the text and video using the CLIP text and visual encoder, extracts [CLS] tags to represent video frame features, and extracts [EOS] tags to represent text features. It then temporally aggregates the video frame features to obtain a holistic video representation, and uses cosine to calculate the global similarity between the entire video and the query. Finally, it trains using the InfoNCE loss. TS2-Net, after calculating the weights for each video patch within a video frame, captures temporal information by swapping video patches between adjacent frames. It then applies a Top-k method to select a set of video patches from each frame, aggregates them into frame features, and replaces the traditional [CLS] tags with text alignment. However, coarse-grained registration focuses primarily on the overall scene information in the video and lacks the ability to capture local details (objects, actions, etc.).

[0005] Fine-grained alignment utilizes word alignment with video frames or video chunks. For example, DRL (Descriptive Reality Linking) exploits the sequential information of query text and video to achieve more effective word and frame alignment while reducing the impact of feature redundancy. TokenFlow aligns words with video chunks to capture local details in the video. This model achieves impressive performance on mainstream video-text retrieval benchmarks. While fine-grained alignment can capture local details in a video, it ignores information such as the global context.

[0006] Due to the limitations of coarse-grained and fine-grained alignment, some researchers have adopted different levels of alignment to explore similarity at multiple granularities. Existing methods, when utilizing fine-grained visual information (patch), use a multilayer perceptron (MLP) to calculate the weight of each video patch and select the top-k ranked video patches using the Top-k method. While this alleviates the problem of high redundancy in video patches, it inevitably leads to information loss when performing coarse-grained and fine-grained alignment on the retrieval data, resulting in poor retrieval results. Summary of the Invention

[0007] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and provide a text-video retrieval method and system based on multi-granularity learnable interaction to solve the problems in the prior art.

[0008] The present invention specifically provides the following technical solution: a text-video retrieval method based on multi-granularity learnable interaction, comprising: Obtaining the data to be retrieved including query text and unpaired videos; Encode the query text and unpaired video in the data to be retrieved respectively, and extract query text features and video features; the query text features and video features both contain coarse-grained information and fine-grained information; Obtain multiple learnable vectors, perform cross-modal interaction between the multiple learnable vectors and fine-grained video features, and perform cross-granular alignment between the query text features and the learnable variables after cross-modal interaction. After cross-granular alignment, perform fine-grained alignment between each word feature in the query text features and the learnable vector to capture fine-grained alignment information, and align the query text features with the video features to capture coarse-grained alignment information. Aggregate fine-grained and coarse-grained alignment information into their own similarity scores, and calculate the weights of different features within each modality during aggregation to capture the weights of different similarity vectors between modalities; Based on the weights of different similarity vectors between the modalities, multiple similarity scores are summed to obtain a final multi-granularity similarity score, and the final retrieval results are obtained by sorting the final multi-granularity similarity scores.

[0009] Preferably, the cross-granularity alignment of query text features and learnable variables after cross-modal interaction is performed as follows: Cosine similarity is used to calculate the similarity matrix between the query text features and the learnable vector after cross-modal interaction, and cross-granularity alignment is performed based on the similarity matrix; among them, cross-granularity alignment is used to filter out redundant information in the learnable vector.

[0010] Preferably, the fine-grained alignment of each word feature in the query text feature and the learnable vector is performed to capture fine-grained alignment information, and the query text feature is aligned with the video feature to capture coarse-grained alignment information, specifically: Compute the cosine similarity between each word feature in the query text feature and each learnable vector to capture fine-grained alignment information in the video; The cosine similarity between the query text features and each video frame feature in the video features is calculated to obtain the sentence-frame similarity matrix. The cosine similarity between the query text features and the video features is also calculated to obtain the sentence-video similarity score. The sentence-frame similarity matrix and the sentence-video similarity score are used as coarse-grained alignment information.

[0011] Preferably, the weights of different features in each modality are calculated separately during aggregation to capture the weights of different similarity vectors between modalities, specifically: When the coarse-grained alignment information is aggregated into similarity scores, softmax is directly used to obtain the weight of the similarity vector between modalities; When the fine-grained alignment information is aggregated into a similarity score, the weight of each word feature or video frame feature within the modality is calculated, and the weight of the similarity vector between the modalities is obtained.

[0012] Preferably, the specific method for extracting the query text features is: An [EOS] mark is added to the query text and encoded using CLIP's text encoder to obtain word features; the [EOS] mark at the top layer of CLIP's text encoder is used as the query text feature.

[0013] Preferably, the specific method for extracting the video features is: Divide the video frames in the unpaired video into disjoint video blocks and add a [CLS] flag before each frame; The video frame features are constructed by the [CLS] flags in each video frame, and the video features are obtained by temporally aggregating the video frame features.

[0014] The present invention provides a text-video retrieval system based on multi-granularity learnable interaction, comprising: A data acquisition module is used to acquire the data to be retrieved including query text and unpaired videos; An encoding module is used to encode the query text and the unpaired video in the search data, respectively, and extract query text features and video features; the query text features and video features both contain coarse-grained information and fine-grained information; The multi-granularity interactive learning module is used to obtain multiple learnable vectors, perform cross-modal interaction between the multiple learnable vectors and fine-grained video features, and perform cross-granular alignment between the query text features and the learnable variables after cross-modal interaction. After cross-granular alignment, each word feature in the query text features is fine-grainedly aligned with the learnable vector to capture fine-grained alignment information, and the query text features are aligned with the video features to capture coarse-grained alignment information. A dual-weight aggregation module is used to aggregate fine-grained alignment information and coarse-grained alignment information into their own similarity scores. During aggregation, the weights of different features within each modality are calculated separately to capture the weights of different similarity vectors between modalities. The retrieval module is used to sum up multiple similarity scores based on the weights of different similarity vectors between the modalities to obtain a final multi-granularity similarity score, and to obtain a final retrieval result by sorting the final multi-granularity similarity scores.

[0015] The present invention provides a computer device, including a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor performs the steps of the above-mentioned text-video retrieval method based on multi-granularity learnable interaction.

[0016] The present invention provides a storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned text-video retrieval method based on multi-granularity learnable interaction are implemented.

[0017] Compared with the prior art, the present invention has the following significant advantages: The present invention performs cross-modal interaction between multiple initialized learnable vectors and fine-grained video features, and performs cross-granular alignment on the query text features and the learnable variables after cross-modal interaction. It can filter out unimportant information learned by the learnable vector interaction, and then perform fine-grained alignment and coarse-grained alignment operations, making fuller use of fine-grained visual information. The fine-grained alignment information and the coarse-grained alignment information are aggregated into respective similarity scores, and the weights of different features within the modality are calculated to reduce the impact of redundant features on retrieval performance. At the same time, the importance of different information within and between modalities is taken into account, and it can focus on the most relevant information in the text-video pair, further enhancing the accuracy of retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1This is an overall flow chart of a text-video retrieval method based on multi-granularity learnable interaction according to the present invention; Figure 2 1 is a schematic diagram of the search process of the method proposed in the present invention; Figure 3 It is a text retrieval video graph of the present invention; Figure 4 It is the video retrieval text graph of the present invention; Figure 5 This is a specific flow chart of a text-video retrieval method based on multi-granularity learnable interaction of the present invention. DETAILED DESCRIPTION

[0019] The following is a clear and complete description of the technical solutions of the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0020] The overall structure of the model is as follows Figure 1 As shown, the model innovatively provides two new perspectives for studying and improving cross-modal alignment: multi-granularity interactive learning and dual-weight aggregation. First, a multi-granularity interactive learning module is proposed, which randomly initializes multiple learnable vectors to interact with fine-grained visual features (patches) for cross-modal interaction, and then uses cross-granularity alignment to filter out unimportant information learned by the learnable vector interaction, and then captures the fine-grained and coarse-grained alignment information between text and video at the same time. Then, a dual-weight aggregation module is designed, which takes into account the importance of different information within and between modalities when aggregating similarity matrices of different granularities into similarity scores. Finally, the similarity scores of different granularities are summed to obtain the final multi-granularity similarity score. The following is a further detailed description through specific implementation methods: In this implementation method, a text-video retrieval method based on multi-granularity learnable interaction includes the following steps:

[0021] Step S1: Obtain the data to be retrieved including query text and unpaired videos.

[0022] When training the text-video retrieval model, a historical dataset of the text-video cross-modal retrieval task is obtained. In the embodiment, common text retrieval video datasets MSR-VTT, MSVD, and DiDeMo are used for training and testing. The MSR-VTT dataset contains more than 10,000 video clips of 10 to 30 seconds, each with approximately 20 natural language descriptions. There are 180,000 training data pairs and 1,000 test data pairs. The MSVD dataset contains 1,970 video clips, ranging in length from 1 second to 62 seconds. Each video has approximately 40 descriptions, for a total of approximately 85,550 descriptions. Among them, 1,200 pairs of data represent training, 100 represent verification, and 670 represent testing. The DiDeMo dataset consists of 10,464 video clips, each with 3 to 5 text descriptions. All sentence descriptions of each video are concatenated as the final description of the entire video. There are 8,395 training data pairs and 1,004 test data pairs.

[0023] Step S2: Encode the query text and unpaired video in the data to be retrieved respectively, and extract query text features and video features; both query text features and video features contain coarse-grained information and fine-grained information.

[0024] The CLIP text encoder and video encoder are used to encode the query text and video data in the historical dataset to obtain video block features and word features.

[0025] Video feature extraction: The CLIP visual encoder is used to encode the video For encoding, CLIP visual encoder is based on VisionTransformer (ViT) architecture, which contains 12 layers and 12 attention heads. Specifically, given a video containing frames , expressed as, the unpaired video The video frames are divided into disjoint video blocks (Patch), which are expressed as , and add a [CLS] flag before each video frame. The video frame features are constructed by the [CLS] flags in each video frame, expressed as, is the batch size, is the number of video frames, is the feature dimension. By temporally aggregating the video frame features, the video features are obtained and expressed as .

[0026] Text feature extraction: The query text is encoded using CLIP's text encoder, which uses a Transformer architecture consisting of 12 layers and 8 attention heads. First, given the query text and append an [EOS] flag, then encode using the CLIP text encoder. , get word features , expressed as, Indicates the number of word features. Finally, the [EOS] flag at the top layer of CLIP's text encoder Transformer is used as the entire query text. The characteristics are expressed as .

[0027] Step S3: Obtain multiple learnable vectors, perform cross-modal interaction between the multiple learnable vectors and fine-grained video features, and perform cross-granular alignment between the query text features and the learnable variables after cross-modal interaction. After cross-granular alignment, perform fine-grained alignment between each word feature in the query text features and the learnable vector to capture fine-grained alignment information, and align the query text features with the video features to capture coarse-grained alignment information.

[0028] By initializing multiple learnable vectors to interact cross-modally with video block features, and adopting cross-granularity alignment of query text and learnable variables, redundant information obtained by the interaction of learnable vectors is filtered out, word features and learnable vectors are aligned to capture fine-grained alignment information, and query text is aligned with the entire video to capture coarse-grained alignment information.

[0029] Step 3-1: Learning vector-video block interaction. Randomly initialize several learnable vectors, represented as ,in is the number of learnable vectors. With video block features Interaction via cross-modal attention mechanism.

[0030] Step 3-2: Cross-granularity alignment. Use cosine similarity to calculate the learnable vector after query text features and cross-modal interaction The similarity matrix between them is used to perform cross-granularity alignment based on the similarity matrix; wherein, the cross-granularity alignment is used to filter out redundant information in the learnable vector.

[0031] Step 3-3: Fine-grained alignment. Calculate the alignment between each word feature in the query text feature and each learnable vector The cosine similarity between them is used to capture the detailed information in the video, that is, the fine-grained alignment information.

[0032] Steps 3-4: Coarse-grained alignment. To fully capture the global information of the video, we first calculate the cosine similarity between the query text features and each video frame feature in the video features to obtain a sentence-frame similarity matrix. We then calculate the cosine similarity between the query text features and the video features to obtain a sentence-video similarity score. The sentence-frame similarity matrix and sentence-video similarity score serve as coarse-grained alignment information.

[0033] Step S4: Aggregate the fine-grained alignment information and the coarse-grained alignment information into their respective similarity scores, and calculate the weights of different features within each modality separately during aggregation to capture the weights of different similarity vectors between modalities.

[0034] When aggregating alignment information at different granularities into their own similarity scores, a fully connected layer calculates weights for each feature within the modality to mitigate the impact of redundant features on retrieval performance. For learnable vectors, information is filtered through cross-granular alignment, so their weights are no longer calculated separately. Finally, a softmax is used to capture the weights of different similarity vectors in the inter-modality similarity matrix to mitigate the impact of unimportant similarity information.

[0035] When the coarse-grained alignment information is aggregated into a similarity score, softmax is directly used to obtain the weight of the inter-modal similarity vector. Specifically, during the aggregation process of the sentence-learnable vector similarity matrix, the learnable vectors already have weights because they interact with the video chunks through a cross-modal attention mechanism. Therefore, when the coarse-grained alignment information is aggregated into a similarity score, softmax is directly used to obtain the weight of the inter-modal similarity vector.

[0036] When aggregating fine-grained alignment information into similarity scores, the weights of each word feature or video frame feature within the modality are calculated, and the weights of the inter-modality similarity vectors are obtained. Specifically, for the word-to-learnable vector similarity matrix, a fully connected layer is used to first calculate the weights of each word within the modality, and then a softmax is used to obtain the weights of the inter-modality similarity vectors. For the sentence-to-frame similarity matrix, a fully connected layer first calculates and weights the frame features within the modality, and then a softmax is used to obtain the weights of the inter-modality similarity vectors.

[0037] Step S5: Based on the weights of different similarity vectors between modalities, multiple similarity scores are summed to obtain a final multi-granularity similarity score, and the final retrieval results are obtained by sorting the final multi-granularity similarity scores.

[0038] When training the text-video retrieval model, the InfoNCE loss of the multi-granularity similarity score is also calculated to optimize the model to obtain the optimized text-video retrieval model. The data to be retrieved is then input into the optimized text-video retrieval model to obtain the retrieval results.

[0039] ; ; ; in, is the final multi-granularity similarity score, represents the batch size, Indicates the query text, Indicates the Videos, and Represent the retrieval losses from text to video and video to text, respectively. Represents the final loss function.

[0040] This paper considers that current text-to-video retrieval models underutilize fine-grained visual information and ignore the weights of intra-modal features, resulting in inaccurate model similarity calculations. Based on this, the present invention uses multiple randomly initialized learnable vectors to interact with video patch features. Furthermore, it constructs multi-granularity alignment based on the full utilization of fine-grained visual information. It also considers the importance of both intra-modal features and inter-modal vectors, further enhancing retrieval accuracy.

[0041] This paper uses the standard recall rate (Recall@rank-k, R@k) as an evaluation metric, which is widely used in text and video retrieval. R@k represents the proportion of results related to the query sample among the top K similarity-ranked results returned by the query. A higher R@k value indicates better retrieval performance.

[0042] The present invention conducts a large number of experiments on three video benchmarks, and The index reached 51.6%, which is 7.1% higher than the baseline model Clip4Clip and 2.2% higher than the current advanced model UCoFIA. The indicator reached 48.9% in the Activity-Net dataset The index reached 47.6%, which was 1.5% and 1.9% higher than the current advanced model UCoFIA.

[0043] like Figure 3 and Figure 4As shown in Figure 2, examples of text-to-video and video-to-text retrieval are demonstrated on the MSR-VTT dataset. It can be observed that the present invention assigns higher similarity scores to the correct videos and texts, which indicates that the present invention accurately understands the query sentence and video content. This is mainly attributed to two factors. (1) The present invention further leverages the advantages of multi-granularity alignment on the basis of fully utilizing fine-grained visual information, enabling the model to have a richer and deeper understanding of text-video pairs. (2) The present invention takes into account the importance of different information within and between modalities, enabling the model to focus on the most relevant information in the text-video pair.

[0044] The verification process includes: Step 1: Obtain the query text and unpaired videos entered by the user as the data to be retrieved.

[0045] Step 2: Input the data to be retrieved into the trained cross-modal retrieval model to obtain its feature vector; Step 3: Calculate the cosine similarity based on the feature vectors and sort them to obtain the final retrieval results.

[0046] The present invention proposes a text-video retrieval system based on multi-granularity learnable interaction, which includes: a data acquisition module, an encoding module, a multi-granularity interactive learning module, a dual-weight aggregation module and a retrieval module.

[0047] Among them, the data acquisition module is used to obtain the data to be retrieved containing query text and unpaired videos; the encoding module is used to encode the query text and unpaired videos in the query data respectively, and extract the query text features and video features; the query text features and video features both contain coarse-grained information and fine-grained information; the query text features and video features both contain coarse-grained and fine-grained information; the multi-granularity interactive learning module is used to obtain multiple learnable vectors, perform cross-modal interaction between the multiple learnable vectors and the fine-grained video features, and perform cross-granular alignment on the query text features and the learnable variables after cross-modal interaction. Finally, each word feature and learnable vector in the query text feature is fine-grainedly aligned to capture fine-grained alignment information, and the query text feature is aligned with the video feature to capture coarse-grained alignment information; the dual-weight aggregation module is used to aggregate the fine-grained alignment information and the coarse-grained alignment information into their respective similarity scores, and calculate the weights of different features in each modality during aggregation to capture the weights of different similarity vectors between modalities; the retrieval module is used to sum up multiple similarity scores based on the weights of different similarity vectors between modalities to obtain the final multi-granularity similarity score, and obtain the final retrieval result by sorting the final multi-granularity similarity score.

[0048] The present invention provides a computer device, including a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor performs the steps of the above-mentioned text-video retrieval method based on multi-granularity learnable interaction.

[0049] According to the disclosed embodiments, a computing device may communicate with one or more external devices (e.g., keyboards, pointing devices, Bluetooth communications, etc.), or any device (e.g., routers, modems, etc.) that enables a computing device to communicate with one or more other computing devices.

[0050] The present invention provides a storage medium having a computer program stored thereon, characterized in that when the computer program is executed by a processor, the steps of the above-mentioned text-video retrieval method based on multi-granularity learnable interaction are implemented.

[0051] According to the disclosed embodiments, the storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. For purposes of the present invention, the storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0052] The above content further explains the present invention in detail in combination with specific preferred embodiments. For those skilled in the art of the technical field to which the present invention belongs, several simple deductions or replacements can be made without departing from the concept of the present invention, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. A text-video retrieval method based on multi-granularity learnable interaction, characterized in that: The steps include: Obtaining the data to be retrieved including query text and unpaired videos; Encode the query text and unpaired video in the data to be retrieved respectively, and extract query text features and video features; The query text features and video features both contain coarse-grained information and fine-grained information; Obtain multiple learnable vectors, perform cross-modal interaction between the multiple learnable vectors and fine-grained video features, and perform cross-granular alignment between the query text features and the learnable variables after cross-modal interaction. After cross-granular alignment, perform fine-grained alignment between each word feature in the query text features and the learnable vector to capture fine-grained alignment information, and align the query text features with the video features to capture coarse-grained alignment information. Aggregate fine-grained and coarse-grained alignment information into their own similarity scores, and calculate the weights of different features within each modality during aggregation to capture the weights of different similarity vectors between modalities; Based on the weights of different similarity vectors between the modalities, multiple similarity scores are summed to obtain a final multi-granularity similarity score, and the final retrieval results are obtained by sorting the final multi-granularity similarity scores.

2. A text-video retrieval method based on multi-granularity learnable interaction as claimed in claim 1, characterized in that: The cross-granularity alignment of query text features and learnable variables after cross-modal interaction is specifically as follows: Cosine similarity is used to calculate the similarity matrix between the query text features and the learnable vector after cross-modal interaction, and cross-granularity alignment is performed based on the similarity matrix; among them, cross-granularity alignment is used to filter out redundant information in the learnable vector.

3. A text-video retrieval method based on multi-granularity learnable interaction as claimed in claim 2, characterized in that: The fine-grained alignment of each word feature and the learnable vector in the query text feature is performed to capture fine-grained alignment information, and the query text feature is aligned with the video feature to capture coarse-grained alignment information, specifically: Compute the cosine similarity between each word feature in the query text feature and each learnable vector to capture fine-grained alignment information in the video; The cosine similarity between the query text features and each video frame feature in the video features is calculated to obtain the sentence-frame similarity matrix. The cosine similarity between the query text features and the video features is also calculated to obtain the sentence-video similarity score. The sentence-frame similarity matrix and the sentence-video similarity score are used as coarse-grained alignment information.

4. The text-video retrieval method based on multi-granularity learnable interaction according to claim 3, characterized in that: During aggregation, the weights of different features within each modality are calculated separately to capture the weights of different similarity vectors between modalities. Specifically, When the coarse-grained alignment information is aggregated into similarity scores, softmax is directly used to obtain the weight of the similarity vector between modalities; When the fine-grained alignment information is aggregated into a similarity score, the weight of each word feature or video frame feature within the modality is calculated, and the weight of the similarity vector between the modalities is obtained.

5. The text-video retrieval method based on multi-granularity learnable interaction according to claim 1, characterized in that: The specific extraction method of the query text feature is: An [EOS] mark is added to the query text and encoded using CLIP's text encoder to obtain word features; the [EOS] mark at the top layer of CLIP's text encoder is used as the query text feature.

6. The text-video retrieval method based on multi-granularity learnable interaction according to claim 1, characterized in that: The specific method for extracting the video features is as follows: Divide the video frames in the unpaired video into disjoint video blocks and add a [CLS] flag before each frame; The video frame features are constructed by the [CLS] flags in each video frame, and the video features are obtained by temporally aggregating the video frame features.

7. A text-video retrieval system based on multi-granularity learnable interaction, characterized in that: include: A data acquisition module is used to acquire the data to be retrieved including query text and unpaired videos; An encoding module is used to encode the query text and unpaired video in the search data respectively, and extract query text features and video features; The query text features and video features both contain coarse-grained information and fine-grained information; The multi-granularity interactive learning module is used to obtain multiple learnable vectors, perform cross-modal interaction between the multiple learnable vectors and fine-grained video features, and perform cross-granular alignment between the query text features and the learnable variables after cross-modal interaction. After cross-granular alignment, each word feature in the query text features is fine-grainedly aligned with the learnable vector to capture fine-grained alignment information, and the query text features are aligned with the video features to capture coarse-grained alignment information. A dual-weight aggregation module is used to aggregate fine-grained alignment information and coarse-grained alignment information into their own similarity scores. During aggregation, the weights of different features within each modality are calculated separately to capture the weights of different similarity vectors between modalities. The retrieval module is used to sum up multiple similarity scores based on the weights of different similarity vectors between the modalities to obtain a final multi-granularity similarity score, and to obtain a final retrieval result by sorting the final multi-granularity similarity scores.

8. A computer device, characterized in that: The invention comprises a memory and a processor, wherein a program is stored in the memory, and when the program is executed by the processor, the processor executes the steps of a text-video retrieval method based on multi-granularity learnable interaction as described in any one of claims 1 to 6.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a text-video retrieval method based on multi-granularity learnable interaction according to any one of claims 1 to 6 are implemented.

Images