A Knowledge-Enhanced Video-Text Retrieval Method

Abstract

The present invention discloses a knowledge-enhanced video-text retrieval method. By encoding video-text through the video-text deep learning pre-training model, namely the CLIP model, the graphic-text association knowledge is transferred to obtain a good video-text representation. By introducing the extra-domain commonsense knowledge graph Conceptnet to enhance the knowledge of video-text encoding, first, the graph embedding vector of the knowledge graph is obtained through a graph convolutional neural network, and then the Cross-attention module is used to enhance the knowledge of text encoding; the text encoding after knowledge enhancement is used to assist the temporal feature aggregation of video frames through the Xpool operation and enhance the global video representation; the video-text representation after knowledge enhancement is subjected to contrastive learning to improve the video-text retrieval performance; through multiple groups of experiments, it is proved that the present invention significantly improves the recall rate of video-text retrieval.

Description

Technical Field

[0001] The present invention relates to a knowledge-enhanced video-text retrieval method, belonging to the technical fields of computer vision and multimodality. Background Art

[0002] With the rapid development of communication technology and Internet technology, and the improvement of network bandwidth, the dissemination of video information on the Internet has become more convenient. The rise of major short video platforms has led to the advent of the video era, and people's demand for watching and publishing videos has increased rapidly. At the same time, the resulting problems are the fragmentation and flooding of information. Therefore, it has become particularly important to screen and retrieve valuable and interesting videos from a large number of videos. Video-text retrieval technology needs to consider the processing of two different modal signals. Due to the heterogeneity between different modalities, the key technical point that video-text retrieval needs to solve is to achieve the alignment of different modalities and bridge the semantic gap. Since the development of deep learning, video-text retrieval technology has also achieved a qualitative leap. By training complex neural networks, signals of different modalities are mapped into the same semantic space, making the alignment between modalities more feasible.

[0003] Existing video-text retrieval methods usually rely on deep learning and neural networks to train encoders of different modalities and modal interaction modules to achieve mapping. When inputting a set of video-text data, video features and text features are extracted through the corresponding video encoder and text encoder respectively, and at the same time, the vector dimension is reduced to make the information more densely concentrated in the feature vector. Then, the two vectors are mapped into the same space through the modal interaction module, and the video-text vectors to be aligned are made closer in this space, so as to achieve the purpose of calculating the distance between the two for retrieval.

[0004] Generally, deep learning methods use datasets labeled with video-text pairs for supervised training, which have certain problems. In a video and its corresponding descriptive text, the video content usually contains more detailed information and requires the combination of certain knowledge to be fully understood; while the text content is more concise and often only covers a small part of the video content. The video-text representations trained only with the in-domain knowledge in the dataset lack the filling of additional information, which causes difficulties in the alignment between different modal representations, and the recall rate of video-text retrieval needs to be improved. Summary of the Invention

[0005] The main objective of the present invention is to provide a knowledge-enhanced video-text retrieval method. Different from the video-text retrieval methods commonly used in the field of video retrieval that rely on in-domain knowledge in the training set, the specific innovation of the present invention lies in the adoption of out-of-domain knowledge, including knowledge in the video-text dataset and commonsense knowledge, to enhance the text encoding vector and the video encoding vector. Due to the refinement of text content and the richness of video content, in some video-text pairs, the text often only covers part of the video content; moreover, due to the long-tail distribution of words in the text, it is difficult to associate this part of the content for video-text using only in-domain knowledge. The enhancement by out-of-domain knowledge can assist in video-text alignment and improve the retrieval performance.

[0006] The technical solution of the present invention is a knowledge-enhanced video-text retrieval method. Specifically, the main steps include the following technical contents:

[0007] Step 1: Use a pre-trained model for contrasting video-text pairs, namely the CLIP (Contrastive Language-Image Pre-Training) model, to perform preliminary encoding on the video-text to obtain a text encoding vector and a video encoding vector; the CLIP model consists of a CLIP text encoder and a CLIP image encoder;

[0008] Step 2: Obtain the knowledge graph embedding of the commonsense knowledge graph Conceptnet. The specific acquisition method is to use a graph convolutional neural network to perform knowledge aggregation on the commonsense knowledge graph Conceptnet to obtain the knowledge graph embedding after knowledge aggregation.

[0009] Step 3: Enhance the text encoding vector obtained in Step 1 based on the knowledge graph embedding of Step 2. The specific enhancement method is as follows: First, retrieve relevant nodes in the Conceptnet knowledge graph by string matching according to the text tokenization of the text encoding vector in Step 1, and then select the embedding of the retrieved nodes in the knowledge graph embedding of Step 2, and perform cross-modal attention interaction with the text encoding vector in Step 1 to obtain a knowledge-enhanced text feature vector.

[0010] Step 4: Enhance the video encoding vectors obtained in Step 1. Perform cross-modal attention interaction between the text feature vectors enhanced with knowledge obtained in Step 3 and the video frame encoding vectors. The interaction method is to assign attention values to different video frame encoding vectors by calculating the similarity with the text feature vectors, and the video frames more relevant to the text feature vectors obtain larger attention values. Convert the attention values into normalized weights through the Softmax function, and perform weighted summation on all video frame encoding vectors to obtain the video feature vectors enhanced with knowledge.

[0011] Step 5: In the training stage of the CLIP model, perform contrastive learning on the video feature vectors enhanced with knowledge obtained in Step 4 and the text feature vectors enhanced with knowledge obtained in Step 3, so that the video feature vectors and text feature vectors belonging to the same video-text pair have a closer distance, that is, a greater similarity, in the projection space. In the inference stage, project the text feature vectors and video feature vectors obtained from the video-text through Steps 1 to 4 into a common semantic space, calculate the cosine distance between the video feature vectors and text feature vectors as the video-text similarity, and reorder them from largest to smallest according to the similarity size to achieve video-text retrieval.

[0012] Further, in Step 1, the specific encoding method of the CLIP model is as follows:

[0013] For text, first tokenize the text into discrete words, then add the start symbol [SOS] at the beginning of the text and the end symbol [EOS] at the end of the text, and send them into the CLIP text encoder for encoding together; the CLIP text encoder output of each word and symbol is a vector of fixed length, and finally take the encoding output of [EOS] as the global encoding vector of the text, which is regarded as aggregating the information of the entire sentence.

[0014] For video, since the CLIP model is encoded for images, when inputting a video, it is first necessary to uniformly downsample the video into a fixed number of video frames, then crop all video frames to a fixed size, and then use the CLIP image encoder to encode all video frames to obtain video frame encoding vectors, and the vector length is the same as the text encoding vector. All video frame encoding vectors are collectively referred to as video encoding vectors.

[0015] Further, in step 4, extraterritorial knowledge is used to enhance the video-text encoding, including the video-text association knowledge in the CLIP model and the knowledge in the commonsense knowledge base. The video-text association knowledge in the CLIP model is used to enhance the text encoding vector knowledge. This includes the CLIP model and the subsequent video-text similarity calculation module. The text is encoded by the CLIP model, and then the text encoding vector is trained and fine-tuned to transfer the image-text association knowledge in the CLIP model to the text encoding vector. The commonsense knowledge graph ConceptNet is used to further enhance the knowledge of the text encoding vector. First, the graph convolutional neural network extracts the graph embedding of the Conceptnet knowledge graph, and the cross-attention method in the Transformer is used to interact with the text encoding vector to obtain the knowledge-enhanced text feature vector.

[0016] Further, for the knowledge enhancement method of the video encoding vector, extraterritorial knowledge is used to enhance the video-text encoding. The extraterritorial knowledge includes the video-text association knowledge in the CLIP model and the knowledge in the commonsense knowledge base. The video-text association knowledge in the CLIP model is used to enhance the knowledge of the video encoding vector; the video sampling frames are encoded by the CLIP model, and the video encoding vector is trained and fine-tuned to transfer the video-text association knowledge in the CLIP model to the video encoding vector. The commonsense knowledge base is used to enhance the knowledge of the video encoding vector; the knowledge-enhanced text feature vector is used to interact with the video encoding vector, different attention weights are assigned to different video frame encoding vectors according to the relevance with the knowledge-enhanced text feature vector, and then weighted summation is performed to obtain the knowledge-enhanced video feature vector, realizing the knowledge enhancement of the video encoding vector.

[0017] Compared with the existing technologies, the present invention has the following advantages:

[0018] Different from the usual video-text retrieval methods, the innovation of the present invention lies in introducing extraterritorial knowledge, combining the image-text association knowledge in the CLIP model with the common sense knowledge in the extraterritorial knowledge graph to enhance the video-text representation. Since the information in the video is rich, it usually requires the combination of additional knowledge to be fully understood, while the content of the text often only covers a small part of the video content, making it difficult to effectively achieve cross-modal alignment of video-text representation by only learning the in-domain knowledge of the training dataset. The knowledge graph representation that extracts and converges structural and adjacent node information is enhanced through cross-attention interaction with the text encoding, and then the knowledge-enhanced text features and the video frame encoding are used for attention-based X-pooling to converge temporal features and enhance the video representation, enabling more effective cross-modal alignment. Multiple sets of experiments have proved that the present invention significantly improves the recall rate of video-text retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a specific flowchart of the present invention.

[0020] Figure 2 It is a retrieval framework diagram proposed by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0021] The present invention will be described in detail below with reference to the drawings and embodiments.

[0022] The flowchart of the present invention is as Figure 1 shown, which is a specific flowchart, and the implementation process is as follows:

[0023] Step 1: Encode the video-text using the CLIP model. First, prepare the CLIP-VIT-Base-32Patch version of the CLIP model and import it into a program written in the Python language. Then, input the text and video into the CLIP model. It should be noted that both the text and video need to be preprocessed before input: for the text, the tokenizer in the CLIP-VIT-Base-32Patch model is used to tokenize the text to obtain discrete words, and the start symbol [SOS] is added at the beginning of the text, and the end symbol [EOS] is added at the end of the text. After being encoded by the CLIP text encoder, an initial text encoding vector is obtained. The encoding vectors output for each word and symbol have the same length. The encoding vector corresponding to the [EOS] position is regarded as the global encoding vector of the text, which records the overall information of the text; for the video, the third-party library Opencv4.6.0 in Python is used to uniformly sample the video, and 12 frames are sampled for each video. For the sampled video frames, the following processing also needs to be carried out: each frame is cropped to a fixed size of 224 pixels x 224 pixels and normalized. The purpose is to make the neural network pay more attention to the differences between different images rather than the absolute values of the image pixels. The normalization formula is as follows:

[0024]

[0025] In formula (1), frame(i) is the value of the i-th pixel in the video frame, and μ and σ are respectively the pixel average value and variance of a randomly sampled frame in the video dataset. The processed video frames are encoded by the CLIP image encoder to obtain an initial video frame encoding vector, which has the same length as the text encoding vector. All the video frame encoding vectors are collectively referred to as the video encoding vector.

[0026] Step 2: Obtain the graph embedding of the commonsense knowledge graph Conceptnet.

[0027] The commonsense knowledge graph contains a large amount of knowledge related to life scenarios and common nouns. The video-text association knowledge contained in the CLIP model often refers to the co-occurrence knowledge between objects and scenes in images and different words in text. For the rich content contained in videos, it usually requires combining extra-domain knowledge to be fully understood. As a supplement to in-domain knowledge, commonsense knowledge can assist in improving the performance of video-text retrieval. ConceptNet originated from the crowdsourcing project OpenMindCommonSense and is a freely available commonsense knowledge graph aimed at helping computers understand the meanings of words used by people. The essence of a knowledge graph is a collection of triples <starting node, relationship label, ending node>, but such a structure cannot be directly input and needs to be transformed into the corresponding graph embedding for further processing. This method uses a graph convolutional neural network to perform graph embedding operations on the knowledge graph Conceptnet, and the specific steps are as follows:

[0028] (1) For a knowledge graph G(N, V), N represents the set of nodes and V represents the set of relationships, where e i ∈N represents a specific node, and (e1, rel, e2) ∈ V represents the relationship of a certain triple. First, use the word embedding technology Glove to encode the node e i into an initial vector of D dimensions where represents the vector space of the real number field.

[0029] (2) Use a graph convolutional neural network to further encode the node vectors to aggregate the information of adjacent nodes and relationships. The information aggregation formula from the l-th layer node to its next-layer node is as follows:

[0030]

[0031] In formula (1), J i represents the set of nodes adjacent to the h i node in the graph, and h j is the adjacent node in the set. R represents the set of all relationships connected to the h i node. W and W o are the weights of the linear layer, and the superscript l represents the layer number of information aggregation. is the relationship vector, indicating the importance of the j-th adjacent node to the current h i node relationship in the l-th layer information aggregation layer. α r is the learnable weight of the r-th relationship vector. Tanh is the hyperbolic tangent function and serves as the activation function of the neural network in formula (1).

[0032] (3) The relationship vector of the l-th layer The acquisition and information update method is given by the following formula.

[0033]

[0034]

[0035] (4) After information aggregation and information iterative update based on neighboring nodes and relationship weights through L layers, the output result of the last layer is the graph embedding embedding that accumulates the structural information of the knowledge graph and triple knowledge.

[0036] Step 3: Perform knowledge enhancement on the initial text encoding vector. Let the initial encoding vector of the text be where n is the number of words after text tokenization (plus the start symbol and end symbol), and d is the dimension of the text encoding vector. Use text tokenization to perform matching retrieval among the nodes of the Conceptnet knowledge graph to obtain relevant nodes, and then extract the graph embedding embedding of the relevant nodes. For text tokenizations with empty retrieval results, fill them with zero vectors of dimension D. The final retrieval result is that is, the graph embedding embedding of the knowledge graph related to the text. Then use h to perform knowledge enhancement on t. The specific enhancement method is as follows: First, map the retrieved graph embedding embedding of the knowledge graph through a linear layer to the common semantic space with the same dimension as the text, and then use the cross-attention method in Transformer to perform knowledge enhancement on the text encoding vector. The specific calculation formula is as follows:

[0037]

[0038] Q t =W q ·t (5)

[0039]

[0040]

[0041]

[0042] (4), (5), (6), (7) In the formula, W h , W q , W k , W v are all trainable linear layer weight parameters. The text encoding vector t h after multi-layer cross-attention knowledge interaction is averaged and pooled and then combined with the initial text global encoding vector tEOS Perform residual connection to obtain a knowledge-enhanced text feature vector Considering that for different video-text understandings, the amount of knowledge required is also different, a gating mechanism is designed to control the degree of knowledge enhancement. The specific formula is as follows:

[0043] Gate_score = sigmoid(W2(ReLU(W , ·t EOS )) (9)

[0044]

[0045] In Equation (10), W1 and W2 are trainable linear layer weight parameters, the ReLU function is the non-linear activation function of the gating mechanism, and the sigmoid function is the gating function.

[0046] Step 4. Perform knowledge enhancement on the initial video encoding vector, as Figure 2 shown in. Let the text feature vector after knowledge enhancement be The initial video encoding vector is Where f is the number of sampled frames of the video, and d is the vector dimension of the text and the video. Since the video has been uniformly sampled and each frame is encoded by the CLIP image encoder, the video encoding at this time only contains the spatial features of different frames and does not converge the unique temporal features of the video. Traditional methods for converging temporal features include Max-pooling and Mean-pooling. The Max-pooling method takes the maximum eigenvalue on different frames as the global feature of the video; Mean-pooling takes the arithmetic mean of the eigenvalues of different frames as the global feature of the video. Both pooling methods simply process the frames: Mean-pooling treats all frames as equally important. However, in the content described by the text, some specific frames are more relevant to the text content and are therefore more important for the association between the text and the video; Max-pooling only takes specific frames and ignores the importance of other frames. Considering this problem, this method uses the X-pool method for converging the temporal features of the video. The specific implementation method is to interact the text feature vector with the features of each frame of the video, calculate a set of weights, which represent the degree of association between a certain frame of video feature and the text. For frames with a stronger association with the text, a larger weight is assigned, while frames with a weaker association are assigned a smaller weight, and the weights are learned by iteratively updating the parameters during the training process. Weighted summation of all video frame features according to the weights gives the global feature of the video that converges the temporal features. The present invention uses a knowledge-enhanced text feature vector to perform the X-pool operation on the video encoding vector, achieving knowledge enhancement of the video encoding vector while converging the video temporal information. The specific formula for the X-pool operation is as follows:

[0047]

[0048] K v = LN(v)W K (12)

[0049] V v = LN(v)W V (13)

[0050]

[0051] r v|t = LN(Attn(Q t , K v , V v )·W O ) (15)

[0052]

[0053] In formulas (11), (12), (13), and (15), W Q 、W K 、W v 、W O are all trainable linear layer weight parameters. The result of formula (14) represents the weight of the degree of association between each frame of the video and the text, where FC is the fully connected layer. LN in formulas (11), (12), (13), (15), and (16) all represent LayerNorm operations. The result of formula (16) represents the global video feature after X-pool interaction for a knowledge-enhanced text feature vector, that is, the knowledge-enhanced video feature vector.

[0054] Step 5. Train the CLIP model. The training strategy is to perform cross-modal comparative learning between the knowledge-enhanced video feature vector and the text feature vector. The cross-modal comparative learning strategy means that in a mini-batch of training, the video and text from the same video-text pair are regarded as positive sample pairs, and other videos or texts from different video-text pairs in the mini-batch are regarded as negative sample pairs. By optimizing the objective function, the distance between positive sample pairs is shortened in the common semantic space, while the distance between negative sample pairs is increased. This allows the matching video-text pairs to obtain a higher similarity score when calculating similarity, thus achieving the purpose of video retrieval. The objective function used in training is InfoNCE, and the specific formula is as follows:

[0055]

[0056]

[0057] L infoNCE =L t2v +L r2t (19)

[0058] In formulas (17) and (18), L t2v represents the contrastive loss from text to video, L v2t represents the contrast loss from video to text, B represents the size of the mini-batch, and λ is the temperature parameter of the Softmax function, which is used to control the smoothness of the Softmax calculation result. s is the similarity calculation function between video and text. In this paper, the cosine distance is used to measure the similarity between the two. The specific calculation formula is as follows:

[0059] z t =||t|| 2 (20)

[0060] z v =||v|| 2 (twenty one)

[0061] s(t, v) = z t ·z v T (22)

[0062] In formulas (20) and (21), ||·|| 2 is the L2 norm normalization operation, and the specific calculation formula is as follows:

[0063]

[0064] In the present invention, the hyperparameter settings for CLIP model training are as follows: the learning rate of the CLIP model is 1e-6, the learning rate of the non-CLIP module is 1e-5, the training optimizer is AdamW, the weight decay is 0.2, the learning rate decay strategy is cosine annealing, the batch-size is 32, and the number of training epochs is 5. After the training loss of the CLIP model converges on the objective function through the video-text dataset, the gradient update of the parameters is stopped, and then the video-text pairs outside the training dataset can be inferred and retrieved. The specific inference and retrieval process is as follows: Input a video or text as the query for retrieval. In the trained CLIP model, after encoding in step 1, knowledge enhancement and interaction are performed with the text or video in the retrieval library gallery using the methods proposed in steps 2 and 3. Finally, the knowledge-enhanced video feature vector and text feature vector are obtained, the similarity is obtained by calculating the cosine distance between the two, and the K samples with the largest similarity are taken as the retrieval results after re-ranking according to the similarity magnitude.

[0065] Embodiment

[0066] The basic process of the maximum disparity acquisition method of the present invention specifically includes the following steps:

[0067] 1) Use the CLIP model to preliminarily encode text and video: First, load the downloaded CLIP-VIT-Base-32Patch version of the CLIP model into a program written in the Python language, and the deep learning framework used is pytorch1.8.0. Preprocess the video-text data before inputting it into the CLIP model: Tokenize the text and add a start tag and an end tag at the beginning and end of the text; for video data, uniform sampling, cropping to a fixed size, and normalization are required. Then input the preprocessed video-text data into the model for preliminary encoding to obtain a text encoding vector and a video encoding vector.

[0068] 2) Obtain the graph embedding of the commonsense knowledge graph Conceptnet as the out-of-domain knowledge used for knowledge enhancement. The graph embedding of Conceptnet is extracted using the method of graph convolutional neural network, and its specific calculation formula is as follows:

[0069]

[0070]

[0071]

[0072] For the nodes in the knowledge graph, first use the Glove word embedding method to initially encode the nodes as Then, through l layers of iterative aggregation of the information of adjacent nodes and the structural information of the knowledge graph. is the relationship vector, calculated from the head and tail node vectors connected by the relationship, representing the importance of the relationship. After L layers of information aggregation and information update, take the node vector of the last layer as the graph embedding of the knowledge graph.

[0073] 3) Use the Conceptnet knowledge graph embedding obtained in 2) to enhance the initial text encoding vector. First, use text tokenization to retrieve the corresponding nodes in the Conceptnet knowledge graph, and take out the embedding of the corresponding nodes from the graph embedding of the knowledge graph. For the results with empty retrieval, fill them with zero vectors of the same dimension. The specific enhancement method is the cross-attention module proposed in Transformer, that is, when calculating the attention, project the text embedding as Q (query), project the knowledge graph embedding as K (key) and V (value), and calculate the attention score. The specific calculation formula is as follows:

[0074]

[0075] Q t =W q ·t

[0076]

[0077]

[0078]

[0079] After the text feature vectors after multi-layer cross-attention knowledge interaction are averaged and pooled, they are residually connected with the globally encoded text vectors of the initial encoding to obtain knowledge-enhanced text feature vectors. Considering that different texts require different amounts of knowledge, this method also designs a gating mechanism to control the degree of knowledge enhancement. The formula is as follows:

[0080] Gate_score=sigmoid(W2(ReLU(W1·t EOS ))

[0081]

[0082] In this method, the dimension of the initial text encoding vector is set to 512, and the dimension of the Conceptnet knowledge graph embedding is 300. Before the interaction, it is necessary to project the knowledge graph embedding into the common vector space where the text encoding vector is located through a learnable linear layer and uniformly map the dimension to 512.

[0083] 4) Use the knowledge-enhanced text feature vectors obtained in 3) to enhance the video encoding vectors. The X-pool method is used to converge the temporal features of the video encoding, that is, the encoding vector of each frame is interacted with the enhanced text feature vector to obtain the attention weights, and then weighted and summed to obtain the enhanced video encoding vector. When calculating the attention, the text feature vector is normalized by Layernorm and then projected into Q (query), and the video frame encoding vector is normalized by Layernorm and projected into K (key) and V (value) to calculate the attention score. The specific calculation formula is as follows:

[0084]

[0085] K v =LN(v)W K

[0086] V v =LN(v)W V

[0087]

[0088] r v|t =LN(Attn(Q t ,K v ,V v )·W O )

[0089]

[0090] In the present invention, the initial dimension of the video encoding vector is set to 512, and the weight parameters in the enhancement network are initialized as the identity matrix, and the bias parameters are initialized to 0.

[0091] 5) Train the model. The training strategy is to perform contrastive learning between the video feature vectors and text feature vectors obtained in 3) and 4). In a mini-batch during training, the video and text from the same video-text pair are regarded as a positive sample pair, and the other videos or texts from different video-text pairs in the mini-batch are regarded as negative sample pairs. By optimizing the objective function, the distance between positive sample pairs is reduced in the common semantic space, while the distance between negative sample pairs is increased, so that when calculating the similarity, the matching video-text pairs can obtain a larger similarity score, achieving the purpose of video retrieval. The objective function used in training is InfoNCE, and the specific formula is as follows:

[0092]

[0093]

[0094] L infoNCE = L t2v + L v2t

[0095] where s is the similarity calculation function, and the calculation formula is as follows.

[0096] z t = ||t|| 2

[0097] z v = ||v||

[0098] s(t, v) = z t · z v T

[0099] For the retrieval and inference stage, input the video-text data into the model to obtain the knowledge-enhanced video feature vectors and text feature vectors. After calculating the similarity between the two through the similarity calculation function and re-ranking according to the similarity magnitude, the obtained result is the retrieval result.

Claims

1. A knowledge-enhanced video-text retrieval method, characterized in that It includes the following steps: Step 1: Use the pre-trained model of the contrastive video-text pair, i.e., the CLIP model, to perform preliminary encoding on the video-text to obtain a text encoding vector and a video encoding vector; the CLIP model is composed of a CLIP text encoder and a CLIP image encoder; Step 2: Obtain the knowledge graph embedding of the commonsense knowledge graph Conceptnet; the specific obtaining method is to use a graph convolutional neural network to perform knowledge aggregation on the commonsense knowledge graph Conceptnet to obtain the knowledge graph embedding after knowledge aggregation; Step 3: Enhance the text encoding vector obtained in Step 1 based on the knowledge graph embedding of Step 2; the specific enhancement method is as follows: first, retrieve relevant nodes in the Conceptnet knowledge graph by string matching according to the text segmentation of the text encoding vector in Step 1, and then select the embedding of the retrieved nodes in the knowledge graph embedding of Step 2, and perform cross-modal attention interaction with the text encoding vector in Step 1 to obtain a text feature vector after knowledge enhancement; Step 4: Enhance the video encoding vector obtained in Step 1; perform cross-modal attention interaction between the text feature vector after knowledge enhancement obtained in Step 3 and the video frame encoding vector. The interaction method is to assign attention values to different video frame encoding vectors by calculating the similarity with the text feature vector, and the video frames more relevant to the text feature vector obtain larger attention values; convert the attention values into normalized weights through the Softmax function, and weighted sum all the video frame encoding vectors to obtain a video feature vector after knowledge enhancement; In Step 4, for the knowledge enhancement method of the video encoding vector, external knowledge is used to enhance the video-text encoding, and the external knowledge includes the video-text association knowledge in the CLIP model and the knowledge in the commonsense knowledge base; Use the video-text association knowledge in the CLIP model to enhance the video encoding vector; sample and encode the video frames through the CLIP model, train and fine-tune the video encoding vector, and transfer the video-text association knowledge in the CLIP model to the video encoding vector; Use the commonsense knowledge base to enhance the video encoding vector; perform interaction between the text feature vector after knowledge enhancement and the video encoding vector, assign different attention weights to different video frame encoding vectors according to the relevance with the text feature vector after knowledge enhancement, and then perform weighted sum to obtain a video feature vector after knowledge enhancement, so as to realize the knowledge enhancement of the video encoding vector; Step 5: Perform contrastive learning on the knowledge-enhanced video feature vectors obtained in Step 4 and the knowledge-enhanced text feature vectors obtained in Step 3, so that the video feature vectors and text feature vectors belonging to the same video-text pair have a closer distance, that is, a greater similarity, in the projection space; in the inference stage of the CLIP model, project the text feature vectors and video feature vectors obtained for the video-text through Steps 1 to 4 into a common semantic space, calculate the cosine distance between the video feature vectors and text feature vectors as the video-text similarity, and reorder the video-text similarities from largest to smallest to achieve video-text retrieval.

2. The knowledge-enhanced video-text retrieval method according to claim 1, wherein In Step 1, the specific encoding method of the CLIP model is as follows: For text, first tokenize the text into discrete words, then add a start-of-sequence symbol [SOS] at the beginning of the text and an end-of-sequence symbol [EOS] at the end of the text, and send them together into the CLIP text encoder for encoding; the CLIP text encoder output for each word and symbol is a vector of fixed length, and finally take the encoding output of [EOS] as the global encoding vector of the text; For video, uniformly downsample the video to a fixed number of video frames, then crop all video frames to a fixed size, and then use the CLIP image encoder to encode all video frames to obtain video frame encoding vectors, where the length of the video frame encoding vectors is the same as that of the text encoding vectors; all video frame encoding vectors are collectively referred to as video encoding vectors.

Images