A video content vectorization retrieval method, device, equipment and storage medium

Abstract

The application provides a video content vectorization retrieval method and device, equipment and a storage medium, and the method comprises the following steps: obtaining key frames and text content of an input video by preprocessing the input video; inputting the key frames and the text content into a pre-trained multi-modal embedding model to obtain a corresponding first high-dimensional vector; inputting user input retrieval content into the multi-modal embedding model to obtain a corresponding second high-dimensional vector, performing vector retrieval on the first high-dimensional vector, and outputting candidate video data corresponding to the retrieval content; and reordering a plurality of video data through time sequence analysis to generate a video data set. Through the implementation of the application scheme, the video key frames and the text content are jointly embedded in multiple modes, the visual information and the text information of the video are mapped to a unified high-dimensional vector space, direct vector retrieval between user input text or images and video content is realized, and the accuracy of the retrieval result is effectively improved.

Description

Technical Field

[0001] This application relates to the field of computer information processing, and in particular to a method, apparatus, device and storage medium for vectorized retrieval of video content. Background Technology

[0002] With the explosive growth of video data, efficiently understanding, organizing, and retrieving video content has become a crucial issue in the field of computer information processing. Most existing video processing methods rely on manual labeling or single-frame image feature extraction, often neglecting the temporal continuity and semantic relationships of videos, leading to insufficient accuracy in retrieval results. Specifically, when users input text descriptions to search for relevant video data, single-frame image features or manual labels often fail to capture the semantic changes between consecutive frames in the video, nor can they effectively match text with video content. This results in limited coverage and low accuracy of retrieval results. Summary of the Invention

[0003] This application provides a video content vectorization retrieval method, apparatus, device, and storage medium to address the problem of insufficient accuracy of retrieval results in related technologies based on single-frame image features or manual tags.

[0004] The first aspect of this application provides a video content vectorization retrieval method, the video content vectorization retrieval method comprising:

[0005] The keyframes and text content of the input video are obtained by preprocessing the input video.

[0006] The keyframes and the text content are input into a pre-trained multimodal embedding model to obtain the corresponding first high-dimensional vector;

[0007] The user-input search content is fed into the multimodal embedding model to obtain the corresponding second high-dimensional vector;

[0008] Based on the second high-dimensional vector, a vector retrieval is performed on the first high-dimensional vector to output candidate video data corresponding to the retrieval content;

[0009] Based on the temporal sequence correlation between the second high-dimensional vector and the first high-dimensional vector, temporal analysis is performed on the candidate video data to generate a set of temporally related nodes; wherein, the candidate video data is a set of video data corresponding to several first high-dimensional vector nodes selected based on similarity ranking after vector retrieval in the vector database according to the second high-dimensional vector;

[0010] By reconstructing the path of the set of temporally associated nodes, a temporal connection relationship between video data is constructed, and a continuous semantic path is generated;

[0011] The candidate video data is scored for context consistency based on the continuous semantic path, and the candidate video data is reordered based on the scoring results to generate the final video data set.

[0012] Optionally, in a first implementation of the first aspect of this application, the step of obtaining the keyframes and text content of the input video by preprocessing the input video includes:

[0013] The input video is segmented into several shot segments based on the continuous frame sequence of the input video.

[0014] By calculating the difference between adjacent frames of the shot segment in the HSL color space, candidate frames representing video semantics in each shot segment are determined, and a set of candidate keyframes is obtained.

[0015] Feature redundancy detection is performed on the candidate keyframe set, and non-repeating image frames are selected as keyframes;

[0016] The keyframes are semantically parsed to generate corresponding semantic text content, and combined with the transcription results obtained from speech recognition of the input video audio track, the text content of the input video is obtained.

[0017] Optionally, in the second implementation of the first aspect of this application, before the step of inputting the keyframe and the text content into a pre-trained multimodal embedding model to obtain the corresponding first high-dimensional vector and storing it in a vector database, the method further includes:

[0018] Keyframes are extracted from each video based on the labeled video data, and the keyframes are input into an image coding network to obtain the corresponding keyframe image feature vectors.

[0019] The semantic text content corresponding to the video is text-encoded to obtain a text feature vector corresponding to the keyframe image feature vector;

[0020] Paired samples are generated by pairing the feature vectors of the keyframe images with the feature vectors of the text.

[0021] By inputting the paired samples into the joint embedding network, the keyframe image feature vector and the text feature vector are trained to be aligned to generate an embedding representation in a shared semantic space.

[0022] The parameters of the joint embedding network are iteratively updated based on the embedding representation to obtain a multimodal embedding model.

[0023] Optionally, in the third implementation of the first aspect of this application, the step of inputting the keyframe and the text content into a pre-trained multimodal embedding model to obtain the corresponding first high-dimensional vector includes:

[0024] The keyframe is input into the image coding subnetwork of the multimodal embedding model to obtain the image embedding vector of the keyframe;

[0025] The text content is input into the text encoding subnetwork of the multimodal embedding model to obtain the text embedding vector of the text content;

[0026] By fusing and mapping the image embedding vector and the text embedding vector in the joint embedding space, a first high-dimensional vector representing the semantics of the input video is generated.

[0027] Optionally, in the fourth implementation of the first aspect of this application, after the step of inputting the keyframe and the text content into a pre-trained multimodal embedding model to obtain the corresponding first high-dimensional vector, the method further includes:

[0028] Generate a set of vector nodes based on the first high-dimensional vector;

[0029] By performing hierarchical processing on the set of vector nodes, the hierarchical structure of each vector node is determined, and a hierarchical linked list of vector nodes is established according to the hierarchical order.

[0030] Neighbor candidate filtering is performed on the vector nodes of each layer, the similarity between nodes is calculated based on the distance metric, and the set of neighbor nodes for each node is determined.

[0031] By sequentially inserting the vector nodes and their neighbor node sets into a hierarchical navigable small-world graph structure, a vector database based on the HNSW index is generated.

[0032] Optionally, in the fifth implementation of the first aspect of this application, the step of inputting the user-inputted search content into the multimodal embedding model to obtain the corresponding second high-dimensional vector includes:

[0033] The search content input by the user is identified to determine the search type of the search content;

[0034] If the search type is an image, then the search content is input into the image encoding subnetwork;

[0035] If the search type is text, then the search content is input into the text encoding subnetwork;

[0036] By standardizing the initial vector output by the corresponding sub-network and mapping it in the joint embedding space using the multimodal fusion module, a second high-dimensional vector is generated.

[0037] Optionally, in the sixth implementation of the first aspect of this application, the step of performing vector retrieval on the first high-dimensional vector based on the second high-dimensional vector and outputting candidate video data corresponding to the retrieval content includes:

[0038] Generate the vector node to be retrieved based on the position of the second high-dimensional vector in the joint embedding space;

[0039] By traversing the neighbor nodes of the vector node to be retrieved in the vector database, the similarity between the vector node to be retrieved and each neighbor node is calculated, and a similarity mapping table is generated.

[0040] The neighbor node set is prioritized according to the similarity mapping table to determine the first high-dimensional vector nodes that are closest to the vector node to be retrieved.

[0041] By indexing and reading the video data corresponding to the plurality of first high-dimensional vector nodes, candidate video data corresponding to the search content is generated.

[0042] A second aspect of this application provides a video content vectorization retrieval device, which is used to implement a video content vectorization retrieval method. The video content vectorization retrieval device includes:

[0043] The processing module is used to obtain the keyframes and text content of the input video by preprocessing the input video;

[0044] The first input module is used to input the keyframe and the text content into a pre-trained multimodal embedding model to obtain the corresponding first high-dimensional vector;

[0045] The second input module is used to input the search content input by the user into the multimodal embedding model to obtain the corresponding second high-dimensional vector;

[0046] The retrieval module is used to perform vector retrieval on the first high-dimensional vector based on the second high-dimensional vector, and output candidate video data corresponding to the retrieval content;

[0047] The analysis module is used to perform time series analysis on candidate video data based on the time series correlation between the second high-dimensional vector and the first high-dimensional vector, and generate a set of time series related nodes; wherein, the candidate video data is a set of video data corresponding to several first high-dimensional vector nodes selected based on similarity sorting after vector retrieval in the vector database based on the second high-dimensional vector;

[0048] The reconstruction module is used to construct the temporal connection relationship between video data and generate continuous semantic paths by reconstructing the path of the set of temporally associated nodes.

[0049] The generation module is used to perform context consistency scoring on candidate video data based on the continuous semantic path, and to reorder the candidate video data based on the scoring results to generate the final video data set.

[0050] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute a computer program stored in the memory, and when the processor executes the computer program, it implements the steps of the video content vectorization retrieval method provided in the first aspect of this application.

[0051] The fourth aspect of this application provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the video content vectorization retrieval method provided in the first aspect of this application.

[0052] In summary, the video content vectorization retrieval method, apparatus, device, and storage medium provided in this application involves preprocessing the input video to obtain keyframes and text content; inputting the keyframes and text content into a pre-trained multimodal embedding model to obtain a corresponding first high-dimensional vector; inputting the user-inputted retrieval content into the multimodal embedding model to obtain a corresponding second high-dimensional vector; performing vector retrieval on the first high-dimensional vector based on the second high-dimensional vector to output candidate video data corresponding to the retrieval content; and reordering several video data sets through temporal analysis to generate a video data set. Through the implementation of this application, multimodal joint embedding of video keyframes and text content maps the visual and textual information of the video to a unified high-dimensional vector space, enabling direct vector retrieval between user-inputted text or images and video content, effectively improving the accuracy of retrieval results. Attached Figure Description

[0053] Figure 1 A flowchart illustrating the video content vectorization retrieval method provided in this application embodiment;

[0054] Figure 2 A schematic diagram of the program modules of the video content vectorization retrieval device provided in the embodiments of this application;

[0055] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0056] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] To address the issue of insufficient accuracy in retrieval results using methods based on single-frame image features or manual tags in related technologies, this application provides a video content vectorization retrieval method, such as... Figure 1 This is a flowchart illustrating the video content vectorization retrieval method provided in this embodiment. The video content vectorization retrieval method includes the following steps:

[0058] Step 110: Obtain the keyframes and text content of the input video by preprocessing the input video.

[0059] Specifically, by preprocessing the input video, shot segmentation and keyframe extraction can be performed to obtain key image frames that represent the overall semantics of the video, as well as related text information. Shot segmentation is generally based on visual feature differences between consecutive frames, such as detection of color space or pixel changes. Combined with inter-frame difference calculations, the position of shot boundaries can be determined. Keyframe selection relies on feature redundancy detection of candidate frames. By removing frames with excessive similarity, representative images are retained, thereby reducing data redundancy and preserving the video's semantic information. Simultaneously, text content can be generated using speech recognition technology from the video audio track and existing text information, enabling the visual and audio information in the video to be transformed into structured data for subsequent multimodal processing. The processed video content possesses both image and text analyzable forms, providing a foundation for vectorized representation.

[0060] In one optional implementation of this embodiment, the step of preprocessing the input video to obtain keyframes and text content includes: segmenting the input video into several shot segments based on a continuous frame sequence; determining candidate frames representing video semantics in each shot segment by calculating the difference between adjacent frames of the shot segments in the HSL color space, thus obtaining a set of candidate keyframes; performing feature redundancy detection on the candidate keyframe set and selecting non-repeating image frames as keyframes; performing semantic parsing on the keyframes to generate corresponding semantic text content, and combining the transcription results obtained from speech recognition of the input video audio track to obtain the text content of the input video.

[0061] In this embodiment, when processing the input video, shot segmentation is performed based on the continuous frame sequence of the video. Shot segmentation refers to dividing the video stream into several segments that maintain visual consistency. Each shot segment corresponds to a continuous camera operation during filming. To achieve this segmentation, visual differences between adjacent frames can be used as a criterion. By analyzing the pixel distribution characteristics of consecutive frames, locations where significant changes occur in the image can be identified. For example, when the camera switches from one scene to another, the overall pixel distribution between frames undergoes abrupt changes, which can serve as markers of shot boundaries. By performing this operation on the entire video, a series of relatively independent shot segments can be obtained, thus providing a basis for subsequent keyframe selection. After completing shot segmentation, representative candidate frames need to be further selected from each shot segment to ensure that the retained frames can express the semantic content of the segment as much as possible. For this purpose, adjacent frames within a segment can be mapped to the HSL color space for difference calculation. The HSL color space is a color representation method designed based on human visual perception. It consists of three parts: hue, saturation, and lightness. Compared to the RGB color space, HSL is closer to the intuitive perception of color by the human eye. In practice, the HSL components of adjacent frames in a clip can be compared pixel by pixel, and the overall difference between frames can be calculated. When the difference exceeds a set threshold, it indicates that the current frame is visually significantly different from the previous frame, and this frame can be marked as a candidate frame. In this way, several potential keyframes can be extracted from a shot clip. These frames can reflect the points of change in the content of the scene, thus forming a set of candidate keyframes. However, in the set of candidate keyframes, there may be a large number of frames with similar or almost identical content. If all of them are directly retained, it will lead to redundancy. Therefore, feature redundancy detection is required for the candidate set. Feature redundancy detection determines which frames express excessive repetition by calculating the similarity between candidate frames. Similarity measurement can be based on image feature vectors, such as using a convolutional neural network to extract deep features of each frame, and then calculating the cosine similarity or Euclidean distance between frames. If the similarity between two frames exceeds a set threshold, one of the frames will be considered redundant and discarded, leaving only the frames with differences as keyframes. For example, in an experimental demonstration video, if several consecutive frames show the appearance of a bottle undergoing the same chemical reaction with almost no change in image quality, only one frame needs to be retained as a representative, thus reducing redundancy and improving data processing efficiency. After obtaining the streamlined set of keyframes, semantic parsing can be further performed. Semantic parsing is the process of understanding and describing image content using a Visual-Language Model (VLM). The model outputs textual information that expresses the semantics of the image.For example, when a keyframe shows a teacher writing "quadratic function" on a blackboard, the semantic parsing output text might be "The teacher is explaining the quadratic function." This parsing process transforms content that originally exists in image form into a semantic text description, allowing computers to understand the visual information of the video in natural language. Visual language models include, but are not limited to, BLIP / BLIP-2 and CLIP. Simultaneously, the language information in the video's audio track also needs to be transcribed, using speech recognition technology to convert the teacher's audio explanation into readable text. Combining the semantic parsing results of the keyframes with the transcribed audio text generates the complete text content of the input video. Thus, during video retrieval, the system can utilize both key information at the image level and multimodal matching using speech and text information. For example, in the aforementioned teaching video scenario, if a user enters the search keyword "quadratic function explanation," the system can locate the segment where the teacher is explaining the quadratic function through the text generated by semantic parsing and the transcribed content obtained from speech recognition, thereby supporting more accurate retrieval.

[0062] Step 120: Input the keyframes and text content into the pre-trained multimodal embedding model to obtain the corresponding first high-dimensional vector.

[0063] Specifically, by inputting keyframes and text content into a pre-trained multimodal embedding model, a high-dimensional vector representation in a unified semantic space can be generated. The multimodal embedding model includes an image encoding sub-network and a text encoding sub-network. The image encoding sub-network uses convolutional neural networks or visual transformers to extract features from keyframes, converting image information into vector representations. The text encoding sub-network uses a pre-trained language model to map text information into vectors. By fusing image vectors and text vectors in a joint embedding space, a high-dimensional vector capable of simultaneously expressing visual and textual semantics can be obtained.

[0064] In an optional implementation of this embodiment, before the step of inputting keyframes and text content into a pre-trained multimodal embedding model to obtain the corresponding first high-dimensional vector and storing it in a vector database, the method further includes: extracting keyframes from each video based on labeled video data, and inputting the keyframes into an image coding network to obtain corresponding keyframe image feature vectors; text encoding the semantic text content corresponding to the video to obtain text feature vectors corresponding to the keyframe image feature vectors; generating paired samples by pairing the keyframe image feature vectors and text feature vectors; training the keyframe image feature vectors and text feature vectors by inputting the paired samples into a joint embedding network to generate an embedding representation in a shared semantic space; and iteratively updating the parameters of the joint embedding network based on the embedding representation to obtain a multimodal embedding model.

[0065] In this embodiment, when constructing a multimodal joint embedding space using labeled video data, keyframes are first extracted from the video. These keyframes, representing the main semantics of the video, are determined after preliminary shot segmentation and candidate frame filtering. The obtained keyframes are then input into an image coding network, which employs a structure based on convolutional neural networks (CNNs) or visual Transformers. CNNs extract low-level features of the image in its spatial neighborhood through local convolutional kernels, then layer-by-layer to form a high-level semantic representation. Visual Transformers, based on a self-attention mechanism, divide the image into small blocks and capture global semantic relationships through attention weights. In this way, keyframes are mapped to high-dimensional image feature vectors. These feature vectors not only contain low-level information such as color, texture, and shape, but also integrate scene semantics and object relationships, thus representing the video content more accurately. Next, the semantic text content corresponding to the video needs to be encoded. The text encoding process employs a pre-trained language model-based text encoder, such as the Bidirectional Encoder Representation Model (BERT). This model models the input text sequence through a multi-layered self-attention mechanism. This self-attention mechanism simultaneously considers information from all other words in the context when representing the current word, thus capturing long-distance dependencies. After inputting the speech recognition results from the video and the semantic descriptions generated from image parsing into the text encoding network, text feature vectors corresponding to the keyframe image feature vectors are obtained. These vectors represent the semantic information of the text in a high-dimensional space. Then, the image feature vectors and text feature vectors are paired. This pairing process essentially establishes a connection between visual and linguistic information within the same semantic unit, forming paired samples. These paired samples include not only positive samples (semantically corresponding image-text pairs) but also negative samples (unrelated image-text pairs). The setting of positive and negative samples helps the network learn to shorten the distance between related modalities and distance unrelated modalities in the semantic space. Through this pairing training, the network gradually masters the correspondences between different modalities. These paired samples are input into a joint embedding network, which has two branches that process image features and text features respectively. At a higher level, a shared projection layer maps both types of features to the same vector space. During training, a contrastive learning approach is used to align the image and text in the shared semantic space by minimizing the vector distance between related samples and maximizing the distance between unrelated samples. The generation of embedding representations is the core of this process; embedding representations refer to the comparable representations of the image and text in the same vector space. For example, if a keyframe containing a "basketball court" scene and a text describing "several people playing basketball" are input, their representations in the embedding space will be close to each other.Finally, by iteratively updating the parameters of the joint embedding network based on the generated embedding representations during training, the network gradually converges to a state capable of stably aligning images and text. The parameter iterative update process relies on optimization algorithms, such as stochastic gradient descent or its improved versions. These algorithms minimize the loss function by continuously adjusting the weight parameters, and the loss function measures the model's error in the alignment task. After training on a large amount of labeled data, the resulting multimodal embedding model can quickly generate its high-dimensional vector representation when a new video is input, enabling the comparison and retrieval of semantic information from different modalities within the same space.

[0066] In one optional implementation of this embodiment, the step of inputting keyframes and text content into a pre-trained multimodal embedding model to obtain the corresponding first high-dimensional vector includes: inputting keyframes into the image encoding subnetwork of the multimodal embedding model to obtain the image embedding vector of the keyframes; inputting text content into the text encoding subnetwork of the multimodal embedding model to obtain the text embedding vector of the text content; and generating a first high-dimensional vector representing the semantics of the input video by fusing and mapping the image embedding vector and the text embedding vector in the joint embedding space.

[0067] In this embodiment, keyframes are first scale-normalized and pixel-value-normalized, and then fed into the image coding sub-network to obtain a representative feature tensor. Next, global pooling or attention pooling is applied to the feature tensor to obtain a fixed-length image embedding vector. The vector is then mapped to a specified joint embedding dimension through a linear projection layer, and finally, layer normalization and L2 normalization are applied to obtain the final image embedding. At the same time, the text content is segmented into a word sequence by a word segmenter and input into the text coding sub-network. After being encoded by a multi-layer transformer, the sequence is represented by CLS tagging (a special symbol added at the beginning of the sequence) or mean pooling to extract a fixed-length text representation. Then, linear projection and normalization are used to make the text embedding and image embedding comparable in dimension and scale. Subsequently, a fusion mapping is performed in the joint embedding space. A cross-modal attention mechanism can be employed, where the text embedding is used as the query vector and the image embeddings of multiple keyframes are used as key-value pairs. The similarity between the query and the key is calculated and normalized using Softmax to obtain attention weights. Image information is then aggregated using a weighted summation method to obtain an image summary vector aligned with the text semantics. Alternatively, the image and text vectors can be fused using a multilayer perceptron after concatenation, and the information flow is balanced through residual connections and layer normalization. To ensure the preservation of temporal information, a Transformer with positional encoding can be applied to the keyframe sequence before image embedding generation to capture inter-frame relationships. The aggregated sequence result is then fused with the text embedding using the aforementioned attention fusion. Finally, the fused vector is subjected to final projection and normalization to obtain a high-dimensional vector representation. This vector contains both visual elements and linguistic semantics, which can be used for subsequent similarity measurement and index retrieval. For example, in a teaching scenario, when the text is "explanation of quadratic functions," the attention weights will assign higher weights to the keyframes on the blackboard and the teacher's writing, making the resulting vector more likely to represent the semantic features of related segments.

[0068] In one optional implementation of this embodiment, after the step of inputting keyframes and text content into a pre-trained multimodal embedding model to obtain the corresponding first high-dimensional vector, the method further includes: generating a set of vector nodes based on the first high-dimensional vector; determining the hierarchical structure of each vector node by performing hierarchical processing on the set of vector nodes, and establishing a hierarchical linked list of vector nodes according to the hierarchical order; performing neighbor candidate screening between vector nodes in each layer, calculating the similarity between nodes based on distance metric, and determining the set of neighbor nodes for each node; and generating a vector database based on HNSW index by sequentially inserting vector nodes and their set of neighbor nodes into a hierarchical navigable small world graph structure.

[0069] In this embodiment, a node structure is constructed for each vector based on the first high-dimensional vector. The node structure includes a unique identifier, vector data, and random level values ​​for subsequent layering. The random level values ​​are generated according to a negative exponential or geometric distribution, thus obtaining a set of vector nodes. Subsequently, an empty layered graph is initialized, and the first vector node is set as the entry point. Next, the set of vector nodes is layered. By reading the random level values ​​of each node, the nodes are inserted into the corresponding multiple levels. A layered linked list is established according to the level order, or the entry point of each level is maintained for layer-by-layer navigation. Thus, the higher levels contain sparse nodes with high level values, while the lower levels contain all nodes, thereby forming a hierarchical topology. Then, neighbor candidate screening is performed on the nodes of each level. In the candidate screening, the similarity between nodes is calculated using a predetermined distance metric function (e.g., an equivalent transformation of Euclidean distance or cosine similarity). A priority queue is used to maintain the set of closest candidates. The construction parameters include the maximum number of neighbors M and the construction probe parameter efConstruction. The former limits the size of the final retained neighbors, and the latter limits the capacity of the candidate pool retained during the search. The candidate set is then pruned using a selection heuristic. This heuristic selects complementary and dispersed neighbors based on a greedy comparison of distances to the current node, ensuring the graph's navigability. Reciprocal connections are added to the neighbor list when necessary, meaning backlinks are added to selected neighbors to maintain undirectedness or quasi-undirectedness. The index is then constructed by sequentially inserting vector nodes and their selected neighbor sets into the hierarchical navigable small-world graph. The insertion process proceeds top-down: starting from the top-level entry point, a greedy search finds the nearest entry position; then, at each lower level, a local search is performed using the built-in candidate pool, and the final neighbors are determined heuristically; finally, at the bottom level, a broader candidate exploration is conducted with a larger efConstruction value, pruning to M neighbors. Therefore, each insertion updates the neighbor list of the affected nodes and triggers neighbor rearrangement when necessary. For example, when a vector representing a teaching segment is inserted, clusters semantically similar are quickly located at higher levels, and then several nearest neighbor vectors are selected at the bottom level using a larger candidate pool. Interconnections are established between these vectors, allowing for rapid descent from higher levels and precise matching at the bottom level during retrieval. Finally, the constructed hierarchical navigable small-world graph is persisted as a vector database index using an adjacency list or compressed adjacency format. Relevant parameters such as M and efConstruction can be configured according to storage and query requirements.

[0070] Step 130: Input the search content entered by the user into the multimodal embedding model to obtain the corresponding second high-dimensional vector.

[0071] Specifically, when inputting user-defined search terms into a multimodal embedding model, the first step is to determine the modality of the input content—whether it's an image or text—and then select the appropriate encoding network for vectorization. Image inputs are processed into image vectors via an image encoding network, while text inputs are processed into text vectors via a text encoding network. The generated vectors are then normalized and multimodal fusion operations mapped to a joint embedding space, allowing user input to be compared with video vectors within the same semantic space. In this way, regardless of whether the user inputs a text description or a diagram, it can be directly matched with video data within the same high-dimensional vector space.

[0072] In one optional implementation of this embodiment, the step of inputting the user-inputted search content into a multimodal embedding model to obtain the corresponding second high-dimensional vector includes: identifying the user-inputted search content and determining the search type; if the search type is an image, inputting the search content into an image encoding subnetwork; if the search type is text, inputting the search content into a text encoding subnetwork; and generating the second high-dimensional vector by standardizing the initial vector output by the corresponding subnetwork and mapping it in the joint embedding space using a multimodal fusion module.

[0073] In this embodiment, during the generation of the second high-dimensional vector, the user-input search content is first modally determined, classifying the input as visual or textual information to determine which processing channel of the multimodal embedding model it enters. Input recognition can be achieved through methods such as data format analysis, pixel feature analysis, or character encoding detection. For example, image input triggers the visual channel, while natural language text triggers the text channel. After recognition, if the input is an image, it is fed into the image encoding sub-network within the model. This sub-network extracts high-dimensional features from the image and maps them to a shared semantic space. If the input is text, it is fed into the text encoding sub-network, which extracts text semantic vectors through a multi-layer self-attention mechanism and positional encoding and maps them to the same semantic space. Although the two sub-networks process different modalities internally, the output vectors maintain consistency with the multimodal joint embedding space in terms of dimension and semantic alignment, ensuring that the generated vector can be directly compared with the first high-dimensional vector in the database. After the sub-network outputs are completed, the initial vectors undergo standardization, including L2 normalization and possible layer normalization operations, to unify the vector length and feature distribution, ensuring they are unaffected by modal differences in subsequent similarity calculations. Subsequently, the vectors enter a multimodal fusion module for mapping. This module adjusts the vectors through linear projection, non-linear activation, and gating mechanisms to maintain a semantic structure consistent with the first high-dimensional vector of the video keyframes and text content in the shared semantic space. The gating mechanism generates weights using the Sigmoid function to balance the influence of the input vectors in the joint embedding space, while residual connections enhance fusion stability while preserving the integrity of the original modal information. Through this process, the final vectors not only retain the modal features of the input but are also mapped to a high-dimensional space aligned with the video library vectors, achieving cross-modal comparability. For example, in an e-commerce video retrieval scenario, a user can upload a product image or enter the text description "red sneakers." Image input is classified as visual modality, and an image coding sub-network extracts embedding vectors containing color, shape, and texture features. Text input is classified as text modality, and a text coding sub-network generates embedding vectors containing semantic and contextual information. After standardization and multimodal fusion, both types of vectors are mapped to a shared semantic space, resulting in a second high-dimensional vector. This vector can be directly compared with the first high-dimensional vector stored in the vector database using cosine similarity or other distance metrics to retrieve the most relevant video data to the user's query. Matching can be achieved within a unified space regardless of whether the input is an image or text.

[0074] Step 140: Perform vector retrieval on the first high-dimensional vector based on the second high-dimensional vector, and output candidate video data corresponding to the retrieved content.

[0075] Specifically, vector retrieval is performed on high-dimensional vectors in the video vector database based on the generated user vectors. Similar content can be quickly located in a large-scale database using an approximate nearest neighbor search algorithm. Common algorithms include index structures based on Hierarchical Navigable Small World (HNSW) graphs, which achieve efficient similarity search by hierarchically classifying vector nodes and filtering neighbor nodes. The retrieval process first calculates the similarity between user vectors and database vectors, for example, using cosine similarity or Euclidean distance. Then, the similarity results are sorted, and the video data corresponding to the closest vectors is selected. Finally, these matched video data are returned to the user, achieving cross-modal, high-precision video content retrieval.

[0076] In one optional implementation of this embodiment, the step of performing vector retrieval on the first high-dimensional vector based on the second high-dimensional vector and outputting candidate video data corresponding to the retrieval content includes: generating a vector node to be retrieved based on the position of the second high-dimensional vector in the joint embedding space; calculating the similarity between the vector node to be retrieved and each neighbor node by traversing the neighbor nodes of the vector node to be retrieved in the vector database, and generating a similarity mapping table; prioritizing the set of neighbor nodes according to the similarity mapping table, and determining several first high-dimensional vector nodes that are closest to the vector node to be retrieved; and generating candidate video data corresponding to the retrieval content by indexing and reading the video data corresponding to the several first high-dimensional vector nodes.

[0077] In this embodiment, during vector retrieval, a corresponding vector node to be retrieved is first generated based on the position of the second high-dimensional vector in the joint embedding space. This node contains a numerical representation of the vector, a unique identifier, and auxiliary information for navigation, such as a hierarchical index or a candidate neighbor list, thereby allowing the queried node to be located in the vector database. Subsequently, the similarity between the node and each neighbor node is calculated by traversing the neighbor nodes of the vector node to be retrieved in the vector database. The similarity calculation can use cosine similarity, Euclidean distance, or other vector space distance metrics. Cosine similarity represents directional similarity by calculating the cosine of the angle between two vectors, while Euclidean distance represents spatial distance by taking the square root of the sum of the squares of the differences in vector coordinates. This traversal process relies on a hierarchical navigable small-world graph (HNSW) index structure. It greedily searches downwards from the high-level entry point, visiting neighboring nodes layer by layer to quickly locate potentially most similar nodes using local shortest paths. Simultaneously, it leverages the interconnections between neighboring nodes to reduce the overhead of global traversal, generating a mapping table containing neighboring nodes and their similarities. This mapping table records the similarity value between each candidate node and the vector node to be retrieved, used for subsequent sorting. After generating the similarity mapping table, the set of neighboring nodes is prioritized to determine the closest first-highest-dimensional vector nodes. This sorting process can be implemented using heap sort or a priority queue. The priority of nodes is determined by their similarity values; nodes with cosine similarity closer to 1 or smaller Euclidean distances have higher priority and are thus visited first in subsequent searches. This sorting not only ensures the accuracy of the retrieval but also controls the computational load by limiting the number of candidates. For example, the top K closest nodes can be selected as the final retrieval targets. In practical applications, if the input query is a product image uploaded by a user, the generated second-highest-dimensional vector will find semantically similar product video data vectors using the above method, thus quickly locating potential matches. Finally, by indexing and reading the video data corresponding to the sorted first high-dimensional vector nodes, specific video segments or complete video information are obtained. Index reading can directly access the storage location using the mapping relationship between vector nodes in the database, thus avoiding a full database scan and improving access efficiency. The read video data maintains the minimum distance to the second high-dimensional vector in the joint embedding space, thus semantically being closest to the user's query. For example, when a user inputs the text "red sneakers," the system generates a second high-dimensional vector through encoding and locates the corresponding node in the HNSW graph. Through neighbor traversal and similarity calculation, it finds the nodes of several video data that are closest to it, and then directly extracts the video data containing "red sneakers" based on the node mapping, making the search results accurately match the user input. The entire process ensures a continuous mapping from the second high-dimensional vector to the actual video data, while utilizing the hierarchical navigation and local neighbor search mechanism of the HNSW index to achieve efficient retrieval in a large-scale vector space.

[0078] Optionally, after the step of generating the vector node to be retrieved based on the position of the second high-dimensional vector in the joint embedding space, the method further includes: determining an adaptive search radius based on the second high-dimensional vector and the distribution density of nodes in the vector database, and generating a dynamic candidate node set based on the search radius; performing multi-path traversal on the dynamic candidate node set, performing parallel similarity calculation on nodes on different paths to generate a multi-path similarity result set, and fusing the result set to obtain a weighted similarity mapping table; performing semantic consistency verification on the candidate nodes based on the weighted similarity mapping table, filtering the node set that satisfies semantic consistency by introducing a semantic constraint function, and generating an optimized target node set.

[0079] In this embodiment, to further enhance the expressive power of the retrieval path during vector retrieval, an adaptive candidate construction mechanism based on vector distribution features can be introduced after the vector nodes to be retrieved are generated. This mechanism determines the search range by analyzing the local density of the second high-dimensional vector in the joint embedding space. Vector distribution density refers to the degree of clustering of vector nodes within a certain region, which can be obtained by statistically counting the number of nodes per unit space or calculating the average neighborhood distance. When the query vector is located in a dense region, the search radius is reduced to avoid introducing too many redundant candidate nodes, while when the query vector is located in a sparse region, the search range is expanded to capture potential similar nodes. Based on this adaptive search radius, a dynamic candidate node set can be generated, so that subsequent retrieval no longer depends on a fixed range, thereby maintaining stable candidate quality under different distribution environments. For example, in scenarios where video semantics are relatively concentrated, such as sports game clips, a large number of similar nodes will cluster near the query vector, and a smaller search radius can cover the main candidates; while in scenarios where semantics are dispersed, such as comprehensive short videos, a larger search range is needed to avoid missing relevant content. After obtaining the dynamic candidate node set, a multi-path traversal mechanism can be further introduced to enhance the ability to explore complex vector space structures. Multi-path traversal refers to simultaneously expanding the search path from multiple starting nodes or different levels. Each path independently performs neighbor expansion and similarity calculation, which are executed concurrently through a parallel computing framework to generate a multi-path similarity result set. Parallel computing refers to executing the similarity calculation task simultaneously on multiple computing units to reduce the overall computation time. The results obtained from each path may differ in local optima, so it is necessary to fuse the multi-path results. The fusion method can be based on a weighted strategy, that is, assigning weights to different paths according to the search depth or node distribution characteristics, thereby generating a weighted similarity mapping table. This mapping table can integrate information from multiple paths, making the ranking results of candidate nodes more comprehensive. After obtaining the weighted similarity mapping table, a semantic consistency verification mechanism can be introduced to further filter candidate nodes. Semantic consistency verification refers to, in addition to vector similarity, using a constructed semantic constraint function to perform a secondary judgment on candidate nodes. This function can be defined based on the directional consistency of embedded vectors or local clustering features, thereby determining whether the candidate node belongs to the same semantic cluster as the query vector in the semantic space. A semantic cluster is a set of vectors that are semantically similar in the embedding space. Vectors within a cluster have high similarity, while they differ significantly from other clusters. By performing semantic consistency filtering on candidate nodes, nodes that are geographically close but semantically misaligned can be eliminated, thus generating an optimized set of target nodes. For example, when inputting the query "basketball game," some video nodes related to "ball sports" but not basketball may be geographically close. Semantic consistency verification can exclude these, retaining only nodes that truly belong to the basketball semantic cluster.Through the aforementioned extension mechanism, the vector retrieval process forms a collaborative optimization structure at the three levels of candidate generation, path search, and result filtering, making the mapping between the second high-dimensional vector and the target video data more refined, while maintaining compatibility with the original vector database structure, thereby constructing a retrieval process with higher expressive power.

[0080] Step 150: Based on the temporal sequence correlation between the second high-dimensional vector and the first high-dimensional vector, perform temporal sequence analysis on the candidate video data to generate a set of temporal sequence related nodes;

[0081] Step 160: By reconstructing the path of the set of temporally associated nodes, a temporal connection relationship between video data is constructed, and a continuous semantic path is generated;

[0082] Step 170: Analyze the context consistency of the candidate video data based on the continuous semantic path, and reorder the candidate video data based on the scoring results to generate the final video data set.

[0083] Specifically, it should be noted that the candidate video data is a collection of video data corresponding to several first high-dimensional vectors selected based on similarity ranking after vector retrieval in the vector database using the second high-dimensional vector. After the vector retrieval results are generated, a semantic enhancement mechanism based on the time dimension can be further introduced to supplement the modeling of the correlation between candidate video data. Since video data itself has continuous temporal attributes, a single vector node can only represent the semantic information of a frame or local segment. Therefore, after obtaining the preliminary matching results, time series correlation analysis can be performed on the corresponding video data based on the matching relationship between the second and first high-dimensional vectors. The time series correlation refers to the sequential order and semantic continuity of multiple video data on the timeline, which can be described by recording the timestamp information of each segment in the video and establishing adjacency relationships. When multiple candidate nodes come from the same video and are adjacent in time, they can be combined into a set of time-series correlated nodes, thereby avoiding isolated segments as the final result and improving the completeness of semantic expression. For example, in an instructional video, the explanation of a certain knowledge point often spans multiple consecutive segments. Through time correlation, the scattered segments can be integrated into a continuous semantic unit. After obtaining the set of temporally related nodes, the relationships between nodes can be reconstructed to form continuous semantic paths. Path reconstruction refers to constructing an ordered path in the time series based on the sequential connections between nodes. This path not only reflects temporal continuity but also embodies a semantic evolution process. Graph traversal algorithms, such as depth-first search or breadth-first search, can be used during path construction. Starting from a highly similar node, the path gradually expands to adjacent time segment nodes, maintaining semantic consistency throughout the expansion process. Continuous semantic paths can represent a complete event or behavioral process. For example, in sports videos, multiple segments from "start" to "sprint" can be connected into a complete path, providing more coherent search results. After forming continuous semantic paths, a contextual consistency scoring mechanism can be further introduced to optimize the ranking of candidate video data. Contextual consistency refers to the degree of similarity between nodes in the path in the semantic space and the overall matching degree with the query vector. The score can be calculated by weighted summation of the similarity between all nodes in the path and the second highest-dimensional vector, while adjusting for path length or temporal continuity. This scoring mechanism prioritizes paths that are both highly relevant to the query semantics and have good temporal continuity, thereby generating the final video dataset. For example, when inputting "scoring moment in a basketball game," not only can a single scoring frame be retrieved, but the entire video process, from dribbling and shooting to scoring, can also be returned through path reconstruction, making the results more consistent with actual semantic requirements.By introducing time-series correlation, path reconstruction, and contextual consistency scoring mechanisms, vector retrieval results are expanded from single-point matching to sequence-level semantic expression, enabling video data to form a unified description in both time and semantic dimensions, thereby constructing a more structured retrieval result generation method.

[0084] According to the video content vectorization retrieval method provided in this application, the keyframes and text content of the input video are obtained through preprocessing; the keyframes and text content are input into a pre-trained multimodal embedding model to obtain the corresponding first high-dimensional vector; the user-input retrieval content is input into the multimodal embedding model to obtain the corresponding second high-dimensional vector; vector retrieval is performed on the first high-dimensional vector based on the second high-dimensional vector to output candidate video data corresponding to the retrieval content. Through the implementation of this application, multimodal joint embedding of video keyframes and text content maps the visual and textual information of the video to a unified high-dimensional vector space, realizing direct vector retrieval between user-input text or images and video content, effectively improving the accuracy of retrieval results.

[0085] Figure 2 This application provides a video content vectorization retrieval device, which can be used to implement the video content vectorization retrieval method described in the foregoing embodiments. For example... Figure 2 As shown, the video content vectorization retrieval device mainly includes:

[0086] Processing module 10 is used to obtain keyframes and text content of the input video by preprocessing the input video;

[0087] The first input module 20 is used to input keyframes and text content into a pre-trained multimodal embedding model to obtain the corresponding first high-dimensional vector;

[0088] The second input module 30 is used to input the search content input by the user into the multimodal embedding model to obtain the corresponding second high-dimensional vector;

[0089] The retrieval module 40 is used to perform vector retrieval on the first high-dimensional vector based on the second high-dimensional vector, and output candidate video data corresponding to the retrieval content;

[0090] Analysis module 50 is used to perform time series analysis on candidate video data based on the time series correlation between the second high-dimensional vector and the first high-dimensional vector, and generate a set of time series related nodes; wherein, the candidate video data is a set of video data corresponding to several first high-dimensional vector nodes selected based on similarity ranking after vector retrieval in the vector database based on the second high-dimensional vector.

[0091] The reconstruction module 60 is used to construct the temporal connection relationship between video data and generate continuous semantic paths by reconstructing the path of the set of temporally associated nodes.

[0092] The generation module 70 is used to perform context consistency scoring on candidate video data based on continuous semantic paths, and to reorder the candidate video data based on the scoring results to generate the final video data set.

[0093] In one optional implementation of this embodiment, the processing module is specifically used to: segment the input video into several shot segments based on the continuous frame sequence of the input video; determine candidate frames representing video semantics in each shot segment by calculating the difference between adjacent frames of the shot segments in the HSL color space, and obtain a set of candidate keyframes; perform feature redundancy detection on the set of candidate keyframes and select non-repeating image frames as keyframes; perform semantic parsing on the keyframes to generate corresponding semantic text content, and combine it with the transcription results obtained by speech recognition of the input video audio track to obtain the text content of the input video.

[0094] In an optional implementation of this embodiment, the first input module is further configured to: extract keyframes from each video based on the labeled video data, input the keyframes into an image coding network to obtain corresponding keyframe image feature vectors, perform text encoding on the semantic text content corresponding to the video to obtain text feature vectors corresponding to the keyframe image feature vectors; generate paired samples by pairing the keyframe image feature vectors and text feature vectors; perform vector alignment training on the keyframe image feature vectors and text feature vectors by inputting the paired samples into a joint embedding network to generate an embedding representation in a shared semantic space; and iteratively update the parameters of the joint embedding network based on the embedding representation to obtain a multimodal embedding model.

[0095] In one optional implementation of this embodiment, the first input module is specifically used to: input keyframes into the image encoding sub-network of the multimodal embedding model to obtain the image embedding vector of the keyframes; input text content into the text encoding sub-network of the multimodal embedding model to obtain the text embedding vector of the text content; and generate a first high-dimensional vector representing the semantics of the input video by fusing and mapping the image embedding vector and the text embedding vector in the joint embedding space.

[0096] In an optional implementation of this embodiment, the first input module is further configured to: generate a set of vector nodes based on the first high-dimensional vector; determine the hierarchical structure of each vector node by performing hierarchical processing on the set of vector nodes, and establish a hierarchical linked list of vector nodes according to the hierarchical order; perform neighbor candidate screening on the vector nodes of each layer, calculate the similarity between nodes based on the distance metric, and determine the set of neighbor nodes for each node; and generate a vector database based on the HNSW index by sequentially inserting the vector nodes and their set of neighbor nodes into the hierarchical navigable small world graph structure.

[0097] In one optional implementation of this embodiment, the second input module is specifically used to: identify the search content input by the user and determine the search type of the search content; if the search type is an image, then input the search content into the image encoding sub-network; if the search type is text, then input the search content into the text encoding sub-network; and generate a second high-dimensional vector by standardizing the initial vector output by the corresponding sub-network and mapping it in the joint embedding space in conjunction with the multimodal fusion module.

[0098] In one optional implementation of this embodiment, the retrieval module is specifically used to: generate a vector node to be retrieved based on the position of the second high-dimensional vector in the joint embedding space; calculate the similarity between the vector node to be retrieved and each neighboring node by traversing the neighboring nodes of the vector node to be retrieved in the vector database, and generate a similarity mapping table; prioritize the set of neighboring nodes according to the similarity mapping table, and determine several first high-dimensional vector nodes that are closest to the vector node to be retrieved; and generate candidate video data corresponding to the retrieval content by indexing and reading the video data corresponding to the several first high-dimensional vector nodes.

[0099] According to the video content vectorization retrieval device provided in this application, the keyframes and text content of the input video are obtained by preprocessing the input video; the keyframes and text content are input into a pre-trained multimodal embedding model to obtain the corresponding first high-dimensional vector; the user-input retrieval content is input into the multimodal embedding model to obtain the corresponding second high-dimensional vector; vector retrieval is performed on the first high-dimensional vector based on the second high-dimensional vector to output candidate video data corresponding to the retrieval content. Through the implementation of this application, multimodal joint embedding of video keyframes and text content maps the visual and textual information of the video to a unified high-dimensional vector space, realizing direct vector retrieval between user-input text or images and video content, effectively improving the accuracy of retrieval results.

[0100] According to the scheme provided in this application Figure 3 An electronic device is provided as an embodiment of this application. This electronic device can be used to implement the video content vectorization retrieval method described in the foregoing embodiments, and mainly includes:

[0101] The system includes a memory 301, a processor 302, and a computer program 303 stored on the memory 301 and executable on the processor 302. The memory 301 and the processor 302 are connected via communication. When the processor 302 executes the computer program 303, it implements the video content vectorization retrieval method described in the foregoing embodiments. The number of processors can be one or more.

[0102] The memory 301 can be a high-speed random access memory (RAM) or a non-volatile memory, such as a disk storage device. The memory 301 is used to store executable program code, and the processor 302 is coupled to the memory 301.

[0103] Furthermore, embodiments of this application also provide a computer-readable storage medium, which may be disposed in the electronic device described in the above embodiments, and the computer-readable storage medium may be as described above. Figure 3 The memory in the illustrated embodiment.

[0104] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the video content vectorization retrieval method described in the foregoing embodiments. Furthermore, the computer-readable storage medium can also be a USB flash drive, external hard drive, read-only memory (ROM), RAM, magnetic disk, or optical disk, or any other medium capable of storing program code.

[0105] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0106] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0107] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for vectorized retrieval of video content, characterized in that, include: The keyframes and text content of the input video are obtained by preprocessing the input video. The keyframes and the text content are input into a pre-trained multimodal embedding model to obtain the corresponding first high-dimensional vector; After obtaining the corresponding first high-dimensional vector, the method further includes: generating a set of vector nodes based on the first high-dimensional vector; determining the hierarchical structure of each vector node by performing hierarchical processing on the set of vector nodes, and establishing a hierarchical linked list of vector nodes according to the hierarchical order; performing neighbor candidate screening on the vector nodes of each layer, calculating the similarity between nodes based on distance metric, and determining the set of neighbor nodes for each node; and generating a vector database based on HNSW index by sequentially inserting the vector nodes and their set of neighbor nodes into the hierarchical navigable small world graph structure. The user-input search content is fed into the multimodal embedding model to obtain the corresponding second high-dimensional vector; Based on the second high-dimensional vector, a vector retrieval is performed on the first high-dimensional vector to output candidate video data corresponding to the retrieval content; Based on the temporal sequence correlation between the second high-dimensional vector and the first high-dimensional vector, temporal analysis is performed on the candidate video data to generate a set of temporal correlation nodes; By reconstructing the path of the set of temporally associated nodes, a temporal connection relationship between video data is constructed, and a continuous semantic path is generated; The candidate video data is scored for context consistency based on the continuous semantic path, and the candidate video data is reordered based on the scoring results to generate the final video data set.

2. The video content vectorization retrieval method according to claim 1, characterized in that, The step of obtaining keyframes and text content of the input video through preprocessing includes: The input video is segmented into several shot segments based on the continuous frame sequence of the input video. By calculating the difference between adjacent frames of the shot segment in the HSL color space, candidate frames representing video semantics in each shot segment are determined, and a set of candidate keyframes is obtained. Feature redundancy detection is performed on the candidate keyframe set, and non-repeating image frames are selected as keyframes; The keyframes are semantically parsed to generate corresponding semantic text content, and combined with the transcription results obtained from speech recognition of the input video audio track, the text content of the input video is obtained.

3. The video content vectorization retrieval method according to claim 1, characterized in that, Before the step of inputting the keyframe and the text content into a pre-trained multimodal embedding model to obtain the corresponding first high-dimensional vector and storing it in the vector database, the method further includes: Keyframes are extracted from each video based on the labeled video data, and the keyframes are input into an image coding network to obtain the corresponding keyframe image feature vectors. The semantic text content corresponding to the input video is text-encoded to obtain a text feature vector corresponding to the keyframe image feature vector; Paired samples are generated by pairing the feature vectors of the keyframe images with the feature vectors of the text. By inputting the paired samples into the joint embedding network, the keyframe image feature vector and the text feature vector are trained to be aligned to generate an embedding representation in a shared semantic space. The parameters of the joint embedding network are iteratively updated based on the embedding representation to obtain a multimodal embedding model.

4. The video content vectorization retrieval method according to claim 3, characterized in that, The step of inputting the keyframe and the text content into a pre-trained multimodal embedding model to obtain the corresponding first high-dimensional vector includes: The keyframe is input into the image coding subnetwork of the multimodal embedding model to obtain the image embedding vector of the keyframe; The text content is input into the text encoding subnetwork of the multimodal embedding model to obtain the text embedding vector of the text content; A first high-dimensional vector representing the input video is generated by fusing and mapping the image embedding vector and the text embedding vector in a joint embedding space.

5. The video content vectorization retrieval method according to claim 4, characterized in that, The step of inputting the user-inputted search content into the multimodal embedding model to obtain the corresponding second high-dimensional vector includes: The search content input by the user is identified to determine the search type of the search content; If the search type is an image, then the search content is input into the image encoding subnetwork; If the search type is text, then the search content is input into the text encoding subnetwork; By standardizing the initial vector output by the corresponding sub-network and mapping it in the joint embedding space using the multimodal fusion module, a second high-dimensional vector is generated.

6. The video content vectorization retrieval method according to claim 5, characterized in that, The step of performing vector retrieval on the first high-dimensional vector based on the second high-dimensional vector and outputting candidate video data corresponding to the retrieved content includes: Generate the vector node to be retrieved based on the position of the second high-dimensional vector in the joint embedding space; By traversing the neighbor nodes of the vector node to be retrieved in the vector database, the similarity between the vector node to be retrieved and each neighbor node is calculated, and a similarity mapping table is generated. The neighbor node set is prioritized according to the similarity mapping table to determine the first high-dimensional vector nodes that are closest to the vector node to be retrieved. By indexing and reading the video data corresponding to the plurality of first high-dimensional vector nodes, candidate video data corresponding to the search content is generated.

7. A video content vectorization retrieval device, characterized in that, The video content vectorization retrieval device is used to implement the video content vectorization retrieval method according to claim 1, and the video content vectorization retrieval device includes: The processing module is used to obtain the keyframes and text content of the input video by preprocessing the input video; The first input module is used to input the keyframes and the text content into a pre-trained multimodal embedding model to obtain the corresponding first high-dimensional vector. After obtaining the corresponding first high-dimensional vector, the module further includes: generating a vector node set based on the first high-dimensional vector; determining the hierarchical structure of each vector node by performing hierarchical processing on the vector node set, and establishing a hierarchical linked list of vector nodes according to the hierarchical order; performing neighbor candidate screening between vector nodes in each layer, calculating the similarity between nodes based on distance metric, and determining the neighbor node set of each node; and generating a vector database based on HNSW index by sequentially inserting the vector nodes and their neighbor node sets into a hierarchical navigable small world graph structure. The second input module is used to input the search content input by the user into the multimodal embedding model to obtain the corresponding second high-dimensional vector; The retrieval module is used to perform vector retrieval on the first high-dimensional vector based on the second high-dimensional vector, and output candidate video data corresponding to the retrieval content; The analysis module is used to perform time series analysis on candidate video data based on the time series correlation between the second high-dimensional vector and the first high-dimensional vector, and generate a set of time series related nodes; wherein, the candidate video data is a set of video data corresponding to several first high-dimensional vector nodes selected based on similarity sorting after vector retrieval in the vector database based on the second high-dimensional vector; The reconstruction module is used to construct the temporal connection relationship between video data and generate continuous semantic paths by reconstructing the path of the set of temporally associated nodes. The generation module is used to perform context consistency scoring on candidate video data based on the continuous semantic path, and to reorder the candidate video data based on the scoring results to generate the final video data set.

8. An electronic device, characterized in that, Includes memory and processor, of which: The processor is used to execute computer programs stored in the memory; When the processor executes the computer program, it implements the steps in the video content vectorization retrieval method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps in the video content vectorization retrieval method according to any one of claims 1 to 6.

Images