Short video retrieval method combining pre-training model and dependency syntax tree

Abstract

The invention discloses a short video retrieval method combining a pre-training model and a dependency syntax tree, and relates to the technical field of multi-modal learning, computer vision and information retrieval cross fusion. The semantic information in the context is captured by using the advantages of BERT, and the deep semantic representation is generated, so that the meaning of the sentence is more accurately understood. A clear syntactic relationship is provided by using the dependency syntactic tree, and the structured dependency relationship between words can be revealed. The relationship can help the model to better understand logic semantics in sentences and extract text features related to tasks in a more targeted manner. Through the dependency syntax tree and the model, primary and secondary information in the long sentences can be distinguished more accurately, the problems of word order ambiguity and the like are eliminated, feature extraction is more stable, and robustness is improved. The dependency syntactic tree is explicit grammatical representation, and structured information of the dependency syntactic tree can assist in analyzing BERT implicit features, so that more explainable basis is provided for the decision process of the model, and the interpretability is remarkably enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of the cross - integration of multi - modal learning, computer vision and information retrieval, and particularly relates to a short - video retrieval method combining a pre - trained model and a dependency syntax tree. Background Art

[0002] The rapid development of the Internet has promoted the prosperity of short - video platforms. The scale of Internet users continues to grow, and users of online videos and short - videos have become the largest in terms of user scale except for instant messaging. However, the development of the Internet has also brought challenges to the management of short - video data. Short - video platforms have brought a user - generated content model, resulting in an explosive growth of short - video data, which presents the characteristics of being massive, heterogeneous, and unstructured, thus leading to the problem of information overload. In addition, short - videos often contain multi - modal information such as text, audio, and images, increasing the complexity of content retrieval.

[0003] In this context, the importance of video retrieval is highlighted. Through efficient video retrieval technology, relevant videos can be quickly matched using the text description provided by users, enabling the screening of valuable content from massive data. The development of video retrieval technology not only improves the user experience but also provides technical support for short - video platforms to optimize the recommendation system and improve the content distribution efficiency, and is one of the key means to solve the problem of short - video data management.

[0004] The background technology of text - video retrieval covers the combination of natural language processing, computer vision, and multi - modal learning. Semantic features of text and video are extracted through deep - learning models and aligned in a unified embedding space. Relying on pre - trained models and large - scale labeled data sets, attention mechanisms and contrast learning are used to achieve efficient text - video matching, and the ranking is optimized through vector similarity retrieval. This technology has wide applications in fields such as short - video recommendation, surveillance analysis, and content management, promoting the intelligent development of human - computer interaction and information retrieval.

[0005] Currently, the research in the field of text - video retrieval mainly focuses on the improvement of the video side, while relatively less exploration has been done on the text side. Therefore, there is still a large room for improvement in the methods on the text side. Currently, in the field of text - video retrieval, the methods for extracting text features mainly include: traditional text feature extraction methods, word - vector - based methods, and pre - trained - model - based methods:

[0006] (1) Traditional text feature extraction methods

[0007] The Bag of Words (BoW) is a simple yet effective method for text representation that represents a document as a feature vector. Its core idea is to view text as an unordered collection of words, disregarding the order and syntactic structure of the words. The occurrence of each word in the document is independent and does not depend on whether other words appear. Specifically, it creates a vocabulary containing all the words by counting all the words in the text, and then represents each document as a vector, where each dimension corresponds to a word in the vocabulary and the value is the frequency of the word's occurrence in the document (or after weight processing, such as TF-IDF value).

[0008] However, as a traditional text feature extraction method, it has many deficiencies. It only focuses on word frequency or the presence of words and completely ignores the order of words in the text, which may lead to the loss of semantic information. For large corpora, the bag-of-words model may generate a very high-dimensional feature vector space, and this high-dimensional data increases computational complexity and is prone to the problem of sparse matrices. It cannot reflect the semantic relationships between words and is based on statistical information of word frequency or the existence of words, unable to capture context information and difficult to understand the meaning of polysemous words in different contexts.

[0009] (2) Methods based on word vectors

[0010] Word2Vec is a neural network-based word vector generation model. Its core idea is to map words to a low-dimensional continuous vector space so that semantically similar words are closer in this space. It learns distributed word representations that can capture the semantic relationships between words by training the model to predict the target word in a given context or to predict the context from the target word. Different from the traditional bag-of-words model, Word2Vec can retain the semantic information and context relationships of words and is widely used in natural language processing tasks.

[0011] Word2Vec generates a unique vector representation for each word and cannot distinguish the polysemy of the same word in different contexts. It is trained based on a local context window (e.g., 5 words) and can only capture local semantic relationships, unable to understand longer-range dependencies or global semantics. Its performance depends to a large extent on the size and quality of the training corpus. If the corpus is insufficient, the generated embedding vectors may not accurately represent the semantics of words.

[0012] (3) Methods based on pre-trained models

[0013] Using a bidirectional Transformer architecture, the self-attention mechanism is utilized to capture global context information from the text. It learns dynamic, context-sensitive word representations through pre-training tasks such as masked language modeling and next sentence prediction, thereby generating high-quality semantic features that can adapt to different natural language processing tasks. These features contain both local details of words and reflect the global structure and relationships of sentences.

[0014] The core idea of BERT is to consider the context before and after a word simultaneously from the context through a bidirectional Transformer architecture to capture the deep semantic relationships of words in a sentence. Different from traditional unidirectional language models, BERT uses masked language modeling and next sentence prediction as training objectives to learn the context representations of words and sentence-level relationships respectively. By obtaining general language representations through pre-training on a large amount of corpus, BERT can be fine-tuned to adapt to specific requirements in downstream tasks, thus efficiently extracting rich features of the text.

[0015] However, the parameter scale of the BERT model is relatively large (for example, BERT-base has 110 million parameters and BERT-large has 340 million parameters), which requires more computing resources and memory for training or inference. BERT's bidirectional self-attention mechanism needs to process the global information of the entire sentence, so the inference speed is relatively slow and it is not suitable for real-time or low-latency tasks. The input length limit of BERT is usually 512 words, and the excess part needs to be truncated or segmented, which may lead to the loss of long text information or require additional splicing methods, increasing the processing complexity. Although BERT can handle certain long-distance dependencies, in extremely long texts, its performance may decline because the computational complexity increases significantly with the increase of text length. Summary of the Invention

[0016] Aiming at the deficiencies of the prior art, the present invention provides a short video retrieval method combining a pre-trained model and a dependency syntax tree, which combines the context semantic understanding ability of a deep learning model and the structured information of a linguistic method by using a method that combines syntactic structure and deep semantic representation, so as to solve the problem that the existing text video retrieval method based on deep learning uses a pre-trained language model to extract text features and insufficiently utilizes the grammatical structure information of sentences, resulting in limited effects when dealing with complex language structures or long texts.

[0017] The technical solution of the present invention is as follows:

[0018] Step 1: Receive the text of the query input by the user and preprocess the text to obtain the preprocessed text;

[0019] The preprocessing first cleans the text, removes redundant spaces, then unifies the case, and performs spelling check and correction;

[0020] Step 2: Perform syntactic analysis on the preprocessed text and generate a dependency syntactic tree;

[0021] Step 2-1: Segment the sentences in the preprocessed text and perform part-of-speech tagging to determine the part of speech of each word;

[0022] Step 2-2: Use a dependency syntactic analysis model to establish the dependency relationships between the words in the text, and find the predicate verb from the text as the root node of the dependency syntactic tree;

[0023] Step 2-3: Establish a weighted directed graph based on the dependency relationships between the words, and use the maximum spanning tree algorithm based on the weighted directed graph and the root node determined in Step 2.2 to obtain the dependency syntactic tree;

[0024] The weighted directed graph is G=(V,E), where V represents the set of nodes, and the nodes include the root node and ordinary nodes. Each ordinary node corresponds to a word other than the predicate verb. E represents the set of edges, including all possible edges. Two nodes with a dependency relationship are connected by an edge. At the same time, the direction of the edge is towards the dependent word, and the weight of the edge is s(h,m), where h is the parent node and m is the child node;

[0025] Step 3: Use a pre-trained BERT model to extract semantic features from the preprocessed text to obtain word vector features;

[0026] Step 3-1: Use a tokenizer to segment the preprocessed text into words;

[0027] Step 3-2: Add special tokens to each sentence in the preprocessed text, and then map each word to a vector representation of a fixed dimension to obtain word embedding vectors;

[0028] The specific operation of adding special tokens to each sentence is: add a token for representing the overall semantics at the beginning of the sentence, and add a token for indicating the end of the sentence at the end of the sentence;

[0029] Step 3-3: Use the sine-cosine function to generate position encodings for each word, and the dimension of the position encoding is the same as the dimension of the word embedding vector;

[0030] Step 3-4: Add the word embedding vector representation of each word to the position encoding to obtain the input vector;

[0031] Step 3-5: Pass the obtained input vector into several Transformer layers for forward propagation to output word vector features;

[0032] Step 4: Parse the dependency syntactic tree, and construct a graph data structure and its adjacency matrix according to the parsing result and the word vector features obtained in Step 3;

[0033] The parsing result of the dependency syntactic tree includes: the word segmentation result of the sentence, that is, words; the part-of-speech tag of each word, the dependency relationship between words, the direction of the dependency relationship, and the root node;

[0034] The method for constructing the graph data structure is specifically as follows: Each word is used as a node in the graph data structure, and a part-of-speech tag is attached to each node. Then, the word vector feature corresponding to each word is used as the node feature. According to the dependency relationship between words in the dependency syntactic tree, two nodes with a dependency relationship are connected by an edge. The direction of the edge points to the dependent word, and the weight of the edge is s(h,m);

[0035] The adjacency matrix is an n*n matrix, and the values in it are used to represent the existence of an edge and the direction of the edge. If word i depends on word j, the matrix element value a ij is the weight of the edge; if word i does not depend on word j, the matrix element value is 0;

[0036] Step 5: Input the graph data structure into the graph convolutional neural network. The node features of the graph data structure are propagated and aggregated through the edges to obtain the word vector features that combine the dependency relationship and the context semantics;

[0037] Step 6: Perform a pooling operation on the word vector features that combine the dependency relationship and the context semantics to generate the text feature vector;

[0038] Step 7: Obtain the video feature vectors of all videos, perform a linear projection on the video feature vectors and the text feature vectors to map the video feature vectors and the text feature vectors to the same dimension, and then normalize them to obtain the normalized video feature vectors and text feature vectors;

[0039] B1: Obtain the videos and perform multi-modal feature extraction on each video; the multi-modal features include the visual representation vector of the video, the MFCC features, and the comprehensive text feature vector;

[0040] B1.1: Extract features from the images in the video to obtain the visual representation vector;

[0041] B1.2: Use the MFCC method to extract features from the audio in the video to obtain the MFCC features;

[0042] B1.3: Extract features from the text in all videos according to the methods in Step 2 - Step 6 to obtain the text feature vector corresponding to each video, and calculate the mean value to obtain the comprehensive text feature vector;

[0043] B2: Align the visual representation vector, the MFCC features, and the comprehensive text feature vector of the video;

[0044] B3: Fuse the visual representation vector, MFCC features, and integrated text feature vector of the aligned video to obtain a video feature vector;

[0045] Step 8: Calculate the cosine similarity between all the normalized video feature vectors and text feature vectors, and display the one or more videos with the maximum cosine similarity to the user.

[0046] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0047] The present invention proposes a new text feature extraction method based on the combination of BERT and dependency syntax trees, which is an innovative method that combines the context semantic understanding ability of deep learning models and the structured information of linguistic methods.

[0048] In terms of enhancing the depth of semantic understanding, utilize the advantages of BERT: Through the bidirectional Transformer mechanism, BERT can capture semantic information in the context, generate deep semantic representations, and thus more accurately understand the meaning of sentences. Utilize the dependency syntax tree to provide explicit syntactic relationships, which can reveal the structured dependencies between words. Such relationships can help the model better understand the logical semantics in the sentence.

[0049] Provide structured support for text feature extraction: The dependency syntax tree clearly decomposes the structure of a sentence into relationships between words, such as subject-predicate-object, modifiers, etc. This information is very helpful for analyzing the sentence logic, rhetorical structure, and even implicit meanings. The combination of BERT and the dependency tree: Based on the context features generated by BERT, combining the dependency syntax tree can focus on key dependency relationships (such as the subject-predicate-object core path) and more specifically extract text features relevant to the task.

[0050] Enhance the robustness and interpretability of the model. Through the dependency syntax tree, the model can more accurately distinguish the primary and secondary information in long sentences, eliminate problems such as word order ambiguity, etc., making the feature extraction more robust and the robustness improved. The dependency syntax tree is an explicit grammatical representation, and its structured information can assist in analyzing the implicit features of BERT, providing more interpretable basis for the decision-making process of the model, and significantly enhancing the interpretability. Description of the Drawings

[0051] Figure 1 It is a flowchart of a short video retrieval method combining a pre-trained model and a dependency syntax tree in an embodiment of the present invention;

[0052] Figure 2 It is a flowchart of determining a dependency syntax tree according to the maximum spanning tree algorithm in an embodiment of the present invention. Detailed Embodiments

[0053] In this embodiment, the retrieval object is short videos; the original videos are from the Microsoft Video Description (MSVD) dataset collected and released by Microsoft Research. It is widely used in the research of multimodal learning, video understanding, and generation tasks. The video content of this dataset comes from YouTube. The research institute selected 1,970 videos from it. The average duration of the videos is about 30 seconds, and they are accompanied by about 41 English descriptions, with a total of 80,827 descriptions, all manually annotated, which can accurately reflect the people, scenes, actions, etc. in the videos.

[0054] A short video retrieval method combining a pre-trained model and a dependency syntax tree, as Figure 1 shown, includes:

[0055] Step 1: Receive the text of the query input by the user and preprocess the text to obtain the preprocessed text;

[0056] The preprocessing first cleans the text to remove extra spaces, then unifies the case, and performs spelling check and correction;

[0057] Step 2: Perform syntactic analysis on the preprocessed text and generate a dependency syntax tree. The specific steps are as follows:

[0058] Step 2-1: Tokenize and perform part-of-speech tagging on the sentences in the preprocessed text to determine the part of speech of each word;

[0059] In this embodiment, the sapCy dependency parsing tool is used to tag the part of speech of each word;

[0060] Step 2-2: Use a dependency syntax analysis model to establish the dependency relationship between the words in the text and find the predicate verb from the text as the root node of the dependency syntax tree;

[0061] In this embodiment, the query input is "A person is playing with sand at the beach.". First, the sentence is decomposed into independent lexical units, and each word will be tagged with its part-of-speech label. A: adjective determiner (DT), person: noun (NN), is: verb (VBZ), playing: verb (VBG), with: preposition (IN), sand: noun (NN), at: preposition (IN), the: determiner (DT), beach: noun (NN).

[0062] The dependency parsing model in this embodiment is a probability distribution model. Using the probability distribution model for dependency parsing, the relationships between various words in a sentence are extracted. The dependency relationships in this query input are: subject-predicate relationship, where "playing" is the predicate verb and "person" is the subject, and the dependency relationship is "nsubj"; verb-object relationship, where "sand" is the object of "playing", and the dependency relationship is "dobj"; prepositional phrase, the phrase "with sand" led by the preposition "with" is the manner of the action "playing", and the dependency relationship is "prep"; locative phrase, the phrase "at the beach" led by the preposition "at" indicates the location where the action occurs, and the dependency relationship is "prep".

[0063] Step 2-3: Establish a weighted directed graph based on the dependency relationships between words, and use the maximum spanning tree algorithm to obtain a dependency syntax tree based on the weighted directed graph and the root node determined in Step 2.2;

[0064] The weighted directed graph is G=(V, E), where V represents the set of nodes, and the nodes include the root node and ordinary nodes. Each ordinary node corresponds to a word other than the predicate verb. E represents the set of edges, including all possible edges. Two nodes with a dependency relationship are connected by an edge, and the direction of the edge is towards the dependent word. The weight of the edge is s(h, m), where h is the parent node and m is the child node;

[0065] The maximum spanning tree algorithm, as Figure 2 shown, specifically includes the following steps:

[0066] A1: Initialize the weighted directed graph;

[0067] A2: Select the incoming edge with the maximum weight for each ordinary node except the root node from the weighted directed graph G. The incoming edge represents the possible dependency relationship of other nodes as the parent node of this node;

[0068] A3: Determine whether there is a cycle. If there is, execute A4; otherwise, execute A5;

[0069] A4: Merge the nodes within the cycle to form a super node, adjust the weights of the edges, and return to A3 to ensure that the maximum dependency syntax tree can still be found after resolving the cycle;

[0070] A5: Expand the merged super node to restore it to the original nodes, and restore the edges to their original weights to obtain the dependency syntax tree;

[0071] After generating the dependency syntax tree, use the annotated dependency syntax tree bank to train the dependency parsing model to improve the accuracy and generalization ability of the dependency parsing model. For the input sentence ω={ω 0 ,ω 1 ,ω 2,…, ω q ,…, ω n} , where ω q is the q-th word, n is the number of words, and the defined dependency relationship y is a dependency syntax tree. The goal of the dependency syntax analysis model is:

[0072]

[0073] where is the maximum dependency syntax tree, arg max determines which parent node h and the current node m have the strongest dependency relationship, Y(ω) is the set of all possible dependency syntax trees, and (h, m) represents an edge;

[0074] Step 3: Use the pre-trained BERT model to extract semantic features from the preprocessed text to obtain word vector features. The specific steps are as follows:

[0075] Step 3-1: Use the WordPiece tokenizer to split the preprocessed text into words;

[0076] Step 3-2: To meet the requirements of the BERT model input structure and provide semantically important identifiers for specific tasks, special tokens are added to each sentence in the preprocessed text, and then each word is mapped to a vector representation of a fixed dimension to obtain word embedding vectors;

[0077] The specific operation of adding special tokens to each sentence is as follows: Add a token representing the overall semantics at the beginning of the sentence and add a token indicating the end of the sentence at the end of the sentence;

[0078] Step 3-3: Use the sine-cosine function to generate position encodings for each word, retaining the position information of the word in the sentence. The dimension of the position encoding is the same as the dimension of the word embedding vector;

[0079] Step 3-4: Add the word embedding vector representation of each word to the position encoding to obtain the input vector;

[0080] In this embodiment, the pre-trained BERT model is used to extract the semantic features of the text, and the query input is "A person is playing with sand at the beach.". First, use the WordPiece tokenizer to break the sentence into independent words and assign a unique ID to each word; then, BERT maps each word into a high-dimensional vector with a dimension of 768. For example, A may be mapped to a vector: [0.23, -0.54, 0.10,...], and person may be mapped to another vector: [0.78, -0.22, 0.65,...].

[0081] Step 3-5: Input the obtained input vectors into several Transformer layers for forward propagation to output word vector features;

[0082] BERT processes these input vectors through multiple Transformer layers (usually 12 layers). Each layer of the Transformer contains two main parts: Self-Attention and Feed-forward Neural Network. Through the self-attention mechanism, the model models the relationships between each word in the input sentence and other words. This is the key to the bidirectionality of BERT. Through the self-attention mechanism, the vector of each word is affected by other words in the sentence. For example, the representation of "person" is not only affected by itself but also by words like "playing" and "beach". The output of each Transformer layer also undergoes a non-linear transformation through a feed-forward neural network to further enhance the model's expressive power.

[0083] Each layer generates a new word vector, which contains more information from the sentence context. The final representation of each word is the comprehensive result of the information of the word in all contexts. For example, after the first layer of self-attention, the vector originally representing "person" may become richer. Considering the relationships with "playing" and "sand", "person" may have more information about actions:

[0084] Initial vector (word vector without attention mechanism): "person" → [0.78, -0.22, 0.65,...];

[0085] Vector after passing through the Transformer layer (with context information added): "person" → [0.85, -0.10, 0.72,...].

[0086] Step 4: Parse the dependency syntactic tree and construct a graph data structure and its adjacency matrix based on the parsing result and the word vector features obtained in Step 3; The parsing result of the dependency syntactic tree includes: the tokenization result of the sentence, i.e., words; the part-of-speech tag of each word, the dependency relationship between words, the direction of the dependency relationship, and the root node;

[0087] The specific method for constructing the graph data structure is as follows: Take each word as a node in the graph data structure, attach a part-of-speech tag to each node, and then use the word vector features corresponding to each word as node features; According to the dependency relationships between words in the dependency syntactic tree, connect two nodes with a dependency relationship with an edge. The direction of the edge points to the dependent word, and the weight of the edge is s(h, m);

[0088] The adjacency matrix is an n*n matrix, and the values therein are used to represent the existence of edges and the directions of edges. If word i depends on word j, the value of matrix element a ij is the weight of the edge; if word i does not depend on word j, the value of the matrix element is 0;

[0089] In this embodiment, the query input is "A person is playing with sand at the beach.". The adjacency matrix representation of this text input is as follows:

[0090]

[0091] Step 5: Input the graph data structure into the graph convolutional neural network. The node features of the graph data structure perform information propagation and aggregation through edges to obtain word vector features that combine dependency relationships and context semantics;

[0092] In this embodiment, the initial features of each node are composed of word vector features output by the pre-trained BERT model. For example, the initial feature of "person" is [0.85, -0.10, 0.72,...] output by BERT. Since the edges of the graph data structure represent the dependency relationships after dependency parsing and represent the structural information between words. Therefore, in the graph convolutional layer, the features of each node will be updated according to the features of its neighbor nodes and the features of the edges. The update formula is as follows:

[0093]

[0094] Among them, is the feature vector of node i in the l-th layer of the graph convolutional layer, l represents the number of the graph convolutional layer, a ij is the element value of the adjacency matrix, d i and d j are the degrees of node i and node j respectively, W l is the weight matrix of the l-th layer of the graph convolutional layer, b is a learnable parameter, ReLU is the activation function, and N(i) is the set of neighbor nodes;

[0095] Step 6: Perform a pooling operation on the word vector features that combine dependency relationships and context semantics to generate a text feature vector;

[0096] Step 7: Obtain the video feature vectors of all videos, perform a linear projection on the video feature vectors and the text feature vectors to map the video feature vectors and the text feature vectors to the same dimension, and then normalize them to obtain the normalized video feature vectors and text feature vectors;

[0097] The method for obtaining the video feature vectors is as follows:

[0098] B1: Obtain videos and perform multi-modal feature extraction on each video; the multi-modal features include the visual representation vector, MFCC features, and comprehensive text feature vector of the video.

[0099] B1.1: Extract features from the images in the video to obtain the visual representation vector.

[0100] Read videos in batches. For each video, select key frames according to the rule of sampling at fixed intervals to reduce redundancy and form a frame sequence. Adjust the resolution of the key frames and normalize the pixel values in the key frames to ensure the quality of the key frames and the consistency of feature extraction. Select a suitable pre-trained model (such as ResNet) and use the weights pre-trained on a large-scale image dataset (such as ImageNet), and retain the feature output of the final connection layer. Extract the features of the key frames in the frame sequence in turn, and aggregate the inter-frame features through a temporal attention mechanism to generate the visual representation vector of the video for subsequent feature fusion.

[0101] B1.2: Use the MFCC method to extract features from the audio in the video to obtain MFCC features.

[0102] Before using the MFCC method to extract audio features, preprocessing is required. Use tools to separate the audio data from the video and save it in a standard format (wav). Use a filter to denoise the audio data, divide it into short frames of a set length to improve the audio quality; apply a Hamming window function to each frame to make the frame transition smoother. After the preprocessing is completed, calculate the Mel frequency spectrum for each preprocessed frame and convert it into a logarithmic Mel frequency spectrum, and then use the discrete cosine transform to extract the MFCC features and save them as feature vectors for subsequent feature fusion.

[0103] B1.3: Extract features from the text in all videos according to the method in steps 2 - step 6 to obtain the text feature vector corresponding to each video, and calculate the mean to obtain the comprehensive text feature vector.

[0104] According to the proposed text feature extraction method combining the pre-trained model and the dependency syntax tree, for each short video in the dataset, perform feature extraction for each text annotation and unify the length of the extracted vectors, and calculate the mean of the feature vectors of all texts to obtain the comprehensive text feature vector for subsequent use.

[0105] B2: Align the visual representation vector, MFCC features, and comprehensive text feature vector of the video.

[0106] After the feature information of the three modalities of the video is extracted, the feature vectors of each modality are inconsistent in dimension, time, etc., so feature alignment is required.

[0107] (1) Temporal alignment;

[0108] Select one modality as the temporal reference (vision). Since the annotation texts of the dataset used for text data cover a relatively large time granularity (the time of the entire video clip), the annotation features need to be copied to all frames in the corresponding time period.

[0109] (2) Dimension alignment;

[0110] Map the low-dimensional modality features to the high-dimensional modality space through a fully connected network to ensure that the feature vectors of the three modalities are in the same dimensional space.

[0111] (3) Normalization;

[0112] After temporal alignment and dimension alignment, normalize the features of each modality to ensure the consistency of the feature value ranges between modalities.

[0113] B3: Fuse the visual representation vector, MFCC features, and integrated text feature vector of the aligned video to obtain the video feature vector;

[0114] In this embodiment, the feature representations of different modalities, the visual representation vector V vision , the integrated text feature vector V text , and the MFCC features V audio are used as inputs, and the attention mechanism is used to generate a weighted video feature vector V for subsequent text-video retrieval tasks.

[0115] V = Attention(V vision , V text , V audio )

[0116] Step 8: Calculate the cosine similarity between all normalized video feature vectors and text feature vectors, and display the one or more videos with the maximum cosine similarity to the user;

[0117] In this embodiment, calculate the dot product of the text feature vector and the video feature vector and the Euclidean norms of the two feature vectors, and divide the dot product by the product of the two norms to obtain the cosine similarity.

Claims

1. A short video retrieval method combining a pre-trained model and a dependency syntax tree, characterized in that: The specific steps include: Step 1: receiving the query text input by the user, and preprocessing the text to obtain the preprocessed text; Step 2: Perform syntactic analysis on the preprocessed text and generate a dependency syntax tree; Step 3: Use the pre-trained BERT model to extract semantic features from the pre-processed text to obtain word vector features; Step 4: Parse the dependency syntax tree and construct a graph data structure and its adjacency matrix based on the parsing results and the word vector features obtained in step 3; Step 5: Input the graph data structure into the graph convolutional neural network. The node features of the graph data structure are propagated and aggregated through the edges to obtain word vector features that combine dependency relationships and contextual semantics. Step 6: Pool the word vector features that combine dependency and context semantics to generate a text feature vector; Step 7: Obtain video feature vectors of all videos, perform linear projection on the video feature vectors and the text feature vectors to map the video feature vectors and the text feature vectors to the same dimension, and then normalize them to obtain normalized video feature vectors and text feature vectors; Step 8: Calculate the cosine similarity between all normalized video feature vectors and text feature vectors, and display one or more videos with the largest cosine similarity to the user.

2. According to claim 1, a short video retrieval method combining a pre-training model and a dependency syntax tree is characterized in that: The preprocessing described in step 1 first cleans the text, removes extra spaces, then unifies the case, and performs spelling checking and correction.

3. According to claim 1, a short video retrieval method combining a pre-training model and a dependency syntax tree is characterized in that: The step 2 comprises: Step 2-1: Segment and tag the sentences in the preprocessed text to determine the part of speech of each word; Step 2-2: Use the dependency syntax analysis model to establish the dependency relationship between words in the text and find the predicate verb from the text as the root node of the dependency syntax tree; Step 2-3: Create a weighted directed graph based on the dependency relationship between words, and use the maximum spanning tree algorithm based on the weighted directed graph and the root node determined in step 2.2 to obtain a dependency syntactic tree.

4. According to claim 3, a short video retrieval method combining a pre-training model and a dependency syntax tree is characterized in that: The weighted directed graph described in step 2-3 is G=(V,E), where V represents a node set, including a root node and ordinary nodes, each ordinary node corresponds to a word other than a predicate verb, and E represents an edge set, including all possible edges. Two nodes with a dependency relationship are connected by an edge, and the direction of the edge is to point to the dependent word. The weight of the edge is s(h,m), h is the parent node, and m is the child node.

5. The short video retrieval method combining a pre-training model and a dependency syntax tree according to claim 1, characterized in that: The step 3 specifically includes: Step 3-1: Use a word segmenter to segment the preprocessed text into words; Step 3-2: Add special tags to each sentence in the preprocessed text, and then map each word to a vector representation of a fixed dimension to obtain a word embedding vector; The adding of special marks to each sentence specifically includes: adding a mark for indicating the overall semantics at the beginning of the sentence, and adding a mark indicating the end of the sentence at the end of the sentence; Step 3-3: Use the sine-cosine function to generate a positional encoding for each word. The dimension of the positional encoding is the same as the dimension of the word embedding vector. Step 3-4: Add the word embedding vector representation of each word to the positional encoding to obtain the input vector; Step 3-5: Pass the obtained input vector into several Transformer layers for forward propagation and output the word vector features.

6. The short video retrieval method combining a pre-training model and a dependency syntax tree according to claim 1, characterized in that: The parsing result of the dependency syntax tree described in step 4 includes: the word segmentation result of the sentence, that is, the words; the part-of-speech tag of each word, the dependency relationship between words, the direction of the dependency relationship, and the root node.

7. The short video retrieval method combining a pre-training model and a dependency syntax tree according to claim 1, characterized in that: The method for constructing the graph data structure described in step 4 is specifically as follows: taking each word as a node in the graph data structure, attaching a part-of-speech tag to each node, and then taking the word vector feature corresponding to each word as the node feature; According to the dependency relationship between words in the dependency syntax tree, two nodes with dependency relationship are connected with an edge, the direction of the edge is pointing to the dependent word, and the weight of the edge is s(h,m); The adjacency matrix is ​​an n*n matrix, in which the values ​​are used to indicate whether an edge exists and the direction of the edge. If word i depends on word j, the matrix element value a ij is the weight of the edge; If word i does not depend on word j, the matrix element value is 0.

8. The short video retrieval method combining a pre-training model and a dependency syntax tree according to claim 1, characterized in that: The method for obtaining the video feature vector in step 7 is: B1: Obtain videos and perform multimodal feature extraction on each video; the multimodal features include a visual representation vector, MFCC features, and a comprehensive text feature vector of the video; B2: Align the visual representation vector, MFCC features and comprehensive text feature vector of the video; B3: Fusion the visual representation vector, MFCC features and comprehensive text feature vector of the aligned video to obtain the video feature vector.

9. The short video retrieval method combining a pre-training model and a dependency syntax tree according to claim 8, characterized in that: B1 specifically includes: B1.1: Extract features from images in the video and obtain visual representation vectors; B1.2: Use the MFCC method to extract features from the audio in the video and obtain MFCC features; B1.3: Extract features from the text in all videos according to the method of steps 2 to 6 to obtain the text feature vector corresponding to each video, and calculate the average to obtain the comprehensive text feature vector.

Images