Video text similarity measurement method and system

Abstract

This invention proposes a video-text similarity measurement method and system, comprising: extracting multimodal features of a video using a pretrained feature extractor to obtain an initial video feature representation; inputting the initial features into a coarse-grained and fine-grained parallel attention network to obtain intermediate representation features; inputting the intermediate feature representation into a feature fusion network to obtain a final video multimodal feature representation; extracting text features from the input text using a pretrained model to obtain a text feature representation; generating text feature representation vectors corresponding to the different modal features of the video through gated embedding; and performing similarity measurement on the video multimodal feature representation and the text feature representation. This invention combines a coarse-grained attention network, a fine-grained attention network, and a feature fusion module, and integrates a multi-head attention network into the fine-grained attention network to alleviate computer memory pressure, thereby accelerating the processing of the video's multimodal features.

Description

Technical Field

[0001] The present invention belongs to the technical field of video-text cross-modal retrieval, and in particular relates to a method and system for measuring similarity between video and text. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, rapid progress in video analysis and natural language processing has propelled video-text cross-modal research to a climax. However, significant semantic differences exist between video and text modalities, and narrowing these gaps between modalities remains a challenging task. Video-text cross-modal retrieval aims to retrieve target videos (texts) given a text (video) query. The retrieved videos or texts are then ranked based on their similarity scores. Compared to image-text cross-modal retrieval, video-text cross-modal retrieval presents challenges in effectively representing video features due to the multimodal information inherent in videos and the temporal information between video features. Existing video-text cross-modal retrieval methods primarily focus on leveraging unimodal video features, failing to fully exploit the multimodal information contained within the video modality. Even though some research methods have extracted multimodal information from videos, they simply overlay this information. This simple overlay of multimodal features in videos tends to generate a large amount of noisy data, significantly hindering cross-modal retrieval between video and text. Most existing video-text cross-modal retrieval methods map extracted unimodal video and text features into a common space for similarity measurement. This is an effective similarity measurement method in current cross-modal retrieval research. Another effective method is to directly measure the similarity of video and text features, which can reduce the similarity measurement time. A common similarity measurement method between different modalities is to measure the Euclidean distance between the features of the two modalities.

[0004] The network structure of video-text cross-modal retrieval uses a dual network structure. In the dual network structure, video feature extraction and text feature extraction use two independent neural networks, and then the extracted features are encoded. Finally, the semantic gap between different modal features is reduced by designing a loss function. The widely used loss is a bidirectional ranking loss for video (text) embedding, positive text (video) embedding, and negative text (video) embedding. This maximizes the similarity between the video (text) embedding and the corresponding text (video) embedding, and reduces the similarity with all other mismatches. Although existing work has steadily improved the performance of video-text cross-modal retrieval, some work still needs to be improved. How to effectively reduce the semantic gap between different modal features contained in the video and learn more effective feature representations has become the focus of current research. In summary, existing methods face the following problems:

[0005] In cross-modal retrieval, simply using a video's multimodal features increases the semantic differences between different modalities. Furthermore, the consistency between different feature points within the same modality needs to be strengthened. Furthermore, there is no better way to semantically align features from different modalities within a video. Processing multimodal features in videos increases the burden on computer hardware. Summary of the Invention

[0006] To overcome the above-mentioned deficiencies of the prior art, the present invention provides a video-text similarity measurement method, which utilizes a multi-head attention mechanism to enhance the connection between feature points in the same modal features in the video, further reducing the memory pressure of the computer.

[0007] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0008] In a first aspect, a method for measuring similarity between video and text is disclosed, including:

[0009] Use the pre-trained feature extractor to extract the multimodal features of the video and obtain the initial video feature representation;

[0010] Input the initial features into the coarse-fine parallel attention network to obtain the intermediate representation features;

[0011] Input the intermediate feature representation into the feature fusion network to obtain the final video multimodal feature representation;

[0012] Use the pre-trained model to extract features from the input text and obtain text feature representation;

[0013] Generate text feature representation vectors corresponding to different modal features of the video through gate embedding;

[0014] Perform similarity measurement on video multimodal feature representation and text feature representation.

[0015] In some implementation examples, the process of extracting initial features of different video modalities is as follows:

[0016] Use different networks to extract multi-modal features of the video and obtain initial feature representations of different modalities in the video;

[0017] Align different modal features in the video and incorporate temporal information from feature extraction into different features;

[0018] Embedding the initial feature representations and temporal features of different modalities;

[0019] The coarse-grained and fine-grained parallel attention network is used to process the embedded features from local region features and global region features.

[0020] In some implementation examples, the process of the coarse-grained attention network in the coarse-grained and fine-grained parallel attention network is:

[0021] The different modal features of each video are divided through the regional pooling strategy;

[0022] The features of each divided part are input into the convolution layer, and the connection between each part area is realized through the regional attention mechanism;

[0023] By performing inner product calculation on the calculated partial feature similarity matrix and the embedded partial features, a partial feature enhancement matrix is ​​obtained;

[0024] Aggregate features are obtained by weighted combination of partial enhanced features.

[0025] In some implementation examples, the fine-grained attention network processing process in the coarse-grained and fine-grained parallel attention network is:

[0026] Convert the initial features of each video modality into a global feature map;

[0027] Calculate the connections between the feature points in the graph to obtain a complete connection graph;

[0028] According to the attention mechanism of the graph, the attention coefficient is calculated to obtain the node features;

[0029] The node features are batch normalized to obtain the features generated by the fine-grained attention network.

[0030] In some implementation examples, the similarity measurement is specifically: evaluating the similarity between the video and the text by calculating multiple layers of similarity, and then performing a weighted summation of the multiple layers of similarity.

[0031] In some implementation examples, the feature fusion network fuses features of different modalities in the video to ultimately obtain a video representation, thereby reducing the semantic differences between different modalities in the video.

[0032] In some implementation examples, a pre-trained BERT model is used for text feature extraction, and then a maximum pooling operation and a fully connected layer are connected to finally output the text feature representation.

[0033] Secondly, a video text similarity measurement system is disclosed, including:

[0034] The video initial feature extraction module is configured to: extract multimodal features of the video using a pre-trained feature extractor to obtain an initial video feature representation;

[0035] The temporal embedding module is configured to: embed the time when each feature extractor extracts features with the features of different video modalities to enhance the alignment between video features;

[0036] The coarse-fine parallel attention module is configured to: input the initial features into the coarse-fine parallel attention network to obtain intermediate representation features;

[0037] The feature fusion module is configured to: input the intermediate feature representation into the feature fusion network to obtain the final video multimodal feature representation;

[0038] The text feature extraction module is configured to: extract text features from the input text using a pre-trained model to obtain a text feature representation;

[0039] Generate text feature representation vectors corresponding to different modal features of the video through gate embedding;

[0040] The multi-layer similarity measurement module is configured to perform similarity measurement on the video multimodal feature representation and the text feature representation.

[0041] One or more of the above technical solutions have the following beneficial effects:

[0042] This paper combines a coarse-grained attention network, a fine-grained attention network, and a feature fusion module. It also incorporates a multi-head attention network into the fine-grained attention network to alleviate computer memory pressure, thereby accelerating the processing of multimodal features in videos. Furthermore, it narrows the semantic gap between different modal features in a video, learns a robust multimodal feature representation for videos, and significantly improves the accuracy of cross-modal retrieval.

[0043] Advantages of additional aspects of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The accompanying drawings, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0045] Figure 1 1 is a flowchart of a video-text cross-modal retrieval method based on a coarse-fine granularity parallel attention mechanism in an embodiment;

[0046] Figure 2 Schematic diagram of the process of video-text cross-modal retrieval method based on coarse-fine granularity parallel attention mechanism in an embodiment;

[0047] Figure 3 This is the overall architecture diagram of the coarse-fine-grained parallel attention network in the embodiment;

[0048] Figure 4 Schematic diagram of the overall architecture of the coarse-grained attention network of the embodiment;

[0049] Figure 5 Schematic diagram of the overall framework of the fine-grained attention network of the embodiment;

[0050] Figure 6 Schematic diagram of the overall architecture of the feature fusion network of the embodiment. DETAILED DESCRIPTION

[0051] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0052] It should be noted that the terms used herein are for describing particular embodiments only and are not intended to limit the exemplary embodiments according to the present invention.

[0053] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0054] Example 1

[0055] See attached Figure 1 、 2 As shown, this embodiment discloses a method for measuring similarity of video text, including:

[0056] Video feature processing network:

[0057] First, this example uses a pre-trained feature extractor to extract multimodal features from a video, generating an initial video feature representation. Second, the initial features are fed into a coarse-fine parallel attention network to generate an intermediate representation. Third, the intermediate representation is fed into a feature fusion module to generate the final multimodal feature representation.

[0058] Text feature processing network:

[0059] The pre-trained BERT model processes the input text to obtain text feature representations. The gated embedding module then generates text feature representation vectors corresponding to different modal features of the video.

[0060] Video-text cross-modal retrieval

[0061] In a specific implementation example, the overall process of video-text cross-modal retrieval is divided into: video retrieval of text, and text retrieval of video.

[0062] Video retrieval text:

[0063] The video sequences in the test set are fed into the network to retrieve text sequences related to the video content. The retrieval accuracy is calculated based on the retrieved text sequences.

[0064] Text Retrieval Video:

[0065] The text sequences in the test set are input into the network to retrieve video sequences related to the text content. The retrieval accuracy is calculated based on the retrieved video sequences.

[0066] In a specific implementation example, the process of extracting initial features of different video modes is as follows:

[0067] The audio feature extractor uses the VGGish model; the record feature extractor uses the Google Cloud Speech-to-Text API extraction system to convert the record into English text; the appearance feature extractor is composed of the SENet-154 model.

[0068] The multi-modal features of the video are extracted using different networks. Thus, the initial feature representations of different modalities in the video are obtained: Among them, F n Represents the feature representation extracted by the nth feature extractor. F(v) represents the set of overall video features extracted by multiple video feature extractors. N represents the number of video feature extractors. In order to align the features of different modalities in the video, the time information during feature extraction is incorporated into different features. The time information is represented as: Among them, T nIt represents the time information when the nth feature extractor extracts features. T(v) represents the time set when multiple feature extractors extract features. The form after embedding different modal features and time features of the video is:

[0069]

[0070] in, Represents the feature representation extracted by the i-th feature extractor of the embedding. The initial feature I(v) is input into the coarse-fine parallel attention network to process the initial feature.

[0071] The coarse-fine parallel attention network includes a coarse-grained attention network and a fine-grained attention network, which process features from the perspective of local region features and global region features respectively. The overall architecture of the coarse-fine parallel attention network is shown in the figure below. Figure 3 shown.

[0072] Coarse-grained attention mechanism:

[0073] The overall architecture of the coarse-grained attention network is as follows Figure 4 shown. Represents the i-th video modality feature. Where B represents the batch size and C represents the number of channels of the feature. In the experiment, the different modality features of each video are divided into p parts through the regional pooling strategy, and each video feature of different modalities is divided into Each partial feature is input into three 1*1 convolutional layers v1(x), v2(x) and v3(x). This structure divides the feature representation of each modality into p parts, and there is a certain connection between each partial area. This connection is achieved through the regional attention mechanism. The formula of the regional attention mechanism is expressed as:

[0074]

[0075] in, is the similarity between the features of part i and part j. In order to make the closeness of the relationship between the parts more distinctive, the present invention introduces an exponential function in the similarity calculation. The similarity function is:

[0076]

[0077]

[0078]

[0079] in, and is the weight parameter learned by functions v1(.) and v2(.) from the neural network. Through similarity calculation, the attention matrix is ​​expressed as This matrix fully reflects the similarity between the i-th feature and the j-th feature. By calculating the similarity of partial features, the interference of noise on the data is reduced. and some features of the embedding By performing inner product calculation, we can obtain some feature enhancement matrices Finally, the aggregated features are obtained through the weighted combination of the partially enhanced features.

[0080]

[0081]

[0082]

[0083] Among them, w p is the weight vector learned from the partial enhancement features to represent the difference between different partial features. The final output of the coarse-grained attention network is Among them, B represents the batch number and C is the feature dimension. The different features of the final video are represented as Where N represents the number of different modalities in the video. The advantages of coarse-grained attention networks are as follows:

[0084] The coarse-grained attention network divides the initial modality features into p equal parts, each of which is equivalent to a region.

[0085] By dividing the initial modal features, the large area feature calculation is converted into small area feature calculation, saving memory space and time cost.

[0086] Fine-grained Attention Network:

[0087] In order to further explore the relationship between fine-grained features, this paper proposes a fine-grained attention network. The implementation of the fine-grained attention network is based on the graph attention network. The overall framework of the fine-grained attention network is as follows Figure 5 As shown. The initial features of each video modality Convert to a global feature map G = (V, E), where V represents the feature point set, E represents the edge set, and each edge represents two feature points v i and v j The close relationship between them. i and v j The close relationship between them is expressed by E(v i ,v j )=v i T .vj Representation. Use the graph attention module to calculate the connection between feature points in the graph. The essence of graph attention is to calculate the connection between two nodes in the graph. The closer the relationship between two feature nodes, the greater the weight of the edge connecting the two nodes. The complete connection graph G = (V, E) is obtained, where V = {v1, v2, ..., v M-1 ,v M} is the node feature. M represents the number of nodes in the graph. According to the attention mechanism of the graph, the attention coefficient is calculated. The calculation formula of the attention coefficient is:

[0088] χ i,j =softmax(e i,j ) (9)

[0089] e i,j =b(W1v i ,W2v j ) (10)

[0090]

[0091] W1 and W2 are parameters learned by the network. The amount of graph data composed of video features is relatively large. Training the original graph attention network is relatively expensive. In the experiment, multiple attention heads are used to calculate the attention coefficients to alleviate memory pressure and improve computational efficiency. In the experiment, the multi-head attention is represented as follows:

[0092]

[0093]

[0094] Among them, || represents connection, W o ∈R D×D 、 and Denotes the mapping parameter matrix. The present invention uses 8 parallel attention heads, and d is equal to D / 8. In the experiment, the present invention uses the ReLU function as the activation function of the attention module output. These node features are represented as:

[0095]

[0096] Among them, N i Represents node v i In order to optimize and improve the training speed of the neural network and make the neural network converge quickly, the present invention performs node feature v' i Batch normalization is performed, and the normalization is expressed as: v' i =BN(v' i). Among them, BN is the batch normalization layer. The feature representation generated by the fine-grained attention network is: V'={v'1,v'2,......,v' M-1 ,v' M},v' i ∈R d V' is finally generated through the average pooling layer and the fully connected layer B represents the batch size. Then, the different modality features of the video are represented as Where N represents the number of different modality features contained in the video. The advantages of fine-grained attention networks are summarized as follows:

[0097] The fine-grained attention network considers the relationship between each feature and enhances the relationship between small regions, for example, whether the faces of two people in a video are the same person.

[0098] The present invention incorporates a multi-head attention network, which not only saves memory space but also improves computing speed.

[0099] The parallel coarse-grained and fine-grained attention networks complement each other: the coarse-grained attention network strengthens connections between large regions, while the fine-grained attention network increases connections between smaller regions. Consequently, important object features are highlighted. The features M(v) = {M1(v), M2(v)} generated by the coarse-fine parallel attention module are input into the feature fusion module, thereby strengthening the relationship between features from different video modalities and narrowing the semantic gap between them.

[0100] Feature fusion network:

[0101] In order to reduce the semantic differences between different modalities in the video, this paper uses a feature fusion network to fuse the features of different modalities in the video, and finally obtains the video representation: Where N represents the number of different modes in the video. The overall architecture of the feature fusion network is as follows Figure 6 shown.

[0102] Text feature extraction:

[0103] In the text feature representation module, the present invention uses a pre-trained BERT model, then connects the maximum pooling operation and the fully connected layer, and finally outputs the text feature representation h'(c). In order to facilitate the similarity measurement between the text feature representation and the different modal feature representations contained in the video, a gate embedding module is designed to embed the text feature representation h'(c) into the multi-layer text feature representation: Φ(c) = [h1(c),h2(c),......,h N (c),h N+1 (c)].

[0104] Multi-layer similarity measure:

[0105] A multi-level similarity measurement is performed on the text feature representation Φ(c) and the different modal feature representations Ω(v) of the video. This measurement method not only fully utilizes the multimodal characteristics of the video, but also fully exploits the semantic complementarity between the different modal features of the video. The similarity between the video and text is evaluated by calculating multi-layer similarity, and then performing a weighted summation of the multi-layer similarities. The calculation formula is as follows:

[0106]

[0107]

[0108] Among them, (w'1,w'2,......,w' N ,w' N+1 ) represents the parameters of the fully connected layer.

[0109] This application incorporates a multi-head attention mechanism into the model, reducing computer memory pressure. Secondly, the multimodal features in the video modality are fused through a feature fusion network to improve semantic consistency between different modalities. Thirdly, to enhance the alignment between the different modal features of the video, the present invention embeds the temporal information of the feature extraction into the different modal features of the video. Finally, the model evaluates the similarity between text and video using a multi-layer similarity metric.

[0110] This paper utilizes a coarse-fine parallel attention mechanism to process multimodal global and local features in videos. This strengthens the connections between feature points within the same modality within the video. Furthermore, integrating a multi-head attention mechanism into the coarse-fine parallel attention network further reduces computer memory pressure.

[0111] The present invention uses a feature fusion network to fuse features of different modalities in a video to reduce the heterogeneity between features of different modalities. This method not only strengthens the relationship between features of the same modality, but also reduces the heterogeneity between features of different modalities in a video.

[0112] experiment:

[0113] Experiments were conducted on two datasets and compared with 11 other popular methods (JSFusion, HT, CE, MMT, MEE, JPose, MEE-COCO, FSE, HSE, CT-SAN, CCA, DE). To fully demonstrate the superiority of the present disclosure, it can be seen from Tables 1 and 2 that the method provided in this embodiment shows better retrieval performance than other methods on different datasets.

[0114] Table 1

[0115]

[0116]

[0117] Table 2

[0118]

[0119] Example 2

[0120] The purpose of this embodiment is to provide a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0121] Example 3

[0122] The purpose of this embodiment is to provide a computer-readable storage medium.

[0123] A computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps of the above method.

[0124] Example 4

[0125] Video initial feature extraction module: extracts the initial features of different modalities contained in the video.

[0126] Temporal embedding module: embeds the time when each feature extractor extracts features with the features of different video modalities, thereby enhancing the alignment between video features;

[0127] Coarse-fine parallel attention module: The raw features of different video modalities are processed using a coarse-fine parallel attention network to enhance the connection between different feature points in the same modality from both coarse-grained and fine-grained perspectives;

[0128] Feature fusion module: The feature fusion module is used to fuse the different modal features contained in the video, thereby reducing the semantic gap between different modal features.

[0129] Text feature extraction module: The pre-trained BERT model is used to extract text features, and then a text feature set is generated through the gate embedding module.

[0130] Multi-layer similarity measurement module: performs similarity measurement on different modal features of text feature video, and then calculates the weighted average to obtain the final video text similarity measurement.

[0131] In order to improve the robustness of video feature representation, the present invention uses a coarse-fine parallel attention model and a feature fusion module to learn more effective video feature representation. Among them, coarse-grained attention learns the relationship between different feature blocks under the same modality features, and fine-grained attention applies attention to global features to strengthen the connection between feature points. Coarse-grained attention and fine-grained attention complement each other. The present invention integrates a multi-head attention network into the model to reduce the pressure on computer memory, and uses a feature fusion module to further reduce the semantic gap between different video modalities. The model architecture proposed in the present invention not only strengthens the relationship between global features and local features, but also compensates for the differences between different modality features in the video. The effectiveness of this method has been demonstrated on the widely used datasets of Acitivity and LSMDC.

[0132] The steps involved in the apparatuses of Examples 2, 3, and 4 above correspond to those of Method Example 1. For detailed implementations, please refer to the relevant description of Example 1. The term "computer-readable storage medium" should be understood to mean a single medium or multiple media containing one or more instruction sets; it should also be understood to include any medium capable of storing, encoding, or carrying an instruction set for execution by a processor and causing the processor to perform any method of the present invention.

[0133] Those skilled in the art will appreciate that the modules or steps of the present invention described above can be implemented using a general-purpose computer device. Alternatively, they can be implemented using program code executable by a computing device, which can then be stored in a storage device and executed by the computing device. Alternatively, they can be fabricated into separate integrated circuit modules, or multiple modules or steps can be fabricated into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0134] Although the above describes the specific embodiments of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without any creative work are still within the scope of protection of the present invention.

Claims

1. The similarity measurement method of video text is characterized by: include: Use the pre-trained feature extractor to extract the multimodal features of the video and obtain the initial video feature representation; Input the initial features into the coarse-fine parallel attention network to obtain the intermediate representation features; Input the intermediate feature representation into the feature fusion network to obtain the final video multimodal feature representation; Use the pre-trained model to extract text features from the input text to obtain text feature representation; Generate text feature representation vectors corresponding to different modal features of the video through gate embedding; Perform similarity measurement on video multimodal feature representation and text feature representation; The process of coarse-grained attention network processing in coarse-grained and fine-grained parallel attention network is as follows: The different modal features of each video are divided through the regional pooling strategy; The features of each divided part are input into the convolution layer, and the connection between each part area is realized through the regional attention mechanism; The formula of the regional attention mechanism is expressed as: in, It is Part and similarities between the features of the parts; By performing inner product calculation on the calculated partial feature similarity matrix and the embedded partial features, a partial feature enhancement matrix is ​​obtained; in, is the partial feature enhancement matrix, is the partial feature similarity matrix, is part of the embedded features; Aggregate features are obtained by weighted combination of partial enhanced features.

2. The video text similarity measurement method according to claim 1, wherein: The process of extracting initial features of different modes of video is as follows: Use different networks to extract multi-modal features of the video and obtain initial feature representations of different modalities in the video; Align different modal features in the video and incorporate temporal information from feature extraction into different features; Embed the initial feature representations and temporal features of different modalities; The coarse-grained and fine-grained parallel attention network is used to process the embedded features from local region features and global region features.

3. The video text similarity measurement method according to claim 2, wherein: The processing process of the fine-grained attention network in the coarse-grained and fine-grained parallel attention network is as follows: Convert the initial features of each video modality into a global feature map; Calculate the connections between the feature points in the graph to obtain a complete connection graph; According to the attention mechanism of the graph, the attention coefficient is calculated to obtain the node features; The node features are batch normalized to obtain the features generated by the fine-grained attention network.

4. The video text similarity measurement method according to claim 1, wherein: The similarity measurement specifically includes: evaluating the similarity between the video and the text by calculating multi-layer similarities, and then performing a weighted summation of the multi-layer similarities.

5. The video text similarity measurement method according to claim 1, wherein the feature fusion The network fuses the features of different modalities in the video and finally obtains the video representation, reducing the semantic differences between different modalities in the video.

6. The video text similarity measurement method according to claim 1, wherein: The pre-trained BERT model is used for text feature extraction, and then the maximum pooling operation and the fully connected layer are connected to finally output the text feature representation.

7. A video text similarity measurement system, comprising: a video text similarity measurement method according to any one of claims 1 to 6, wherein: include: The video initial feature extraction module is configured to: extract multimodal features of the video using a pre-trained feature extractor to obtain an initial video feature representation; The temporal embedding module is configured to: embed the time when each feature extractor extracts features with the features of different video modalities to enhance the alignment between video features; The coarse-fine parallel attention module is configured to: input the initial features into the coarse-fine parallel attention network to obtain intermediate representation features; The feature fusion module is configured to: input the intermediate feature representation into the feature fusion network to obtain the final video multimodal feature representation; The text feature extraction module is configured to: extract text features from the input text using a pre-trained model to obtain a text feature representation; Generate text feature representation vectors corresponding to different modal features of the video through gate embedding; The multi-layer similarity measurement module is configured to perform similarity measurement on the video multimodal feature representation and the text feature representation.

8. A computing device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method described in any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 6 are performed.

Images