A method and system for multi-tag extraction and pre-screening of text-to-video retrieval

Abstract

This invention relates to the field of multimodal data retrieval and cross-modal contrastive learning technology, and discloses a method and system for multi-label extraction and pre-screening of text-to-video retrieval. The method includes: extracting labels from text and video data to generate a label set, and screening the label set; extracting features from videos in a video library and input text to generate multi-label probability distributions for videos and text; constructing an inverted index from labels to video IDs, and matching the generated multi-label probability distributions with the labels in the inverted index; and matching relevant videos based on the matching results of the input text label set. Through a multi-label extraction and pre-screening framework, video and text data are converted into discrete semantic labels, and combined with inverted index technology, fast label matching and video pre-screening are achieved, effectively reducing computational complexity.

Description

Technical Field

[0001] This invention relates to the field of multimodal data retrieval and cross-modal comparative learning technology, specifically to a method and system for multi-label extraction and pre-screening of text to video retrieval. Background Technology

[0002] With the rapid growth of multimedia data, especially the explosive growth of video content, how to quickly retrieve relevant videos from massive video libraries based on natural language queries has become an important research direction. In fields such as social media, education, entertainment, and surveillance, the demand for text-to-video retrieval is constantly increasing. Currently, common text-to-video retrieval methods include real-valued methods and hash-based methods. Real-valued methods calculate the similarity between text and video through feature inner products, which can better preserve semantic information, but they have high computational complexity, large storage requirements, and poor scalability. Hash-based methods utilize binary hash codes to reduce storage and computational overhead, but may lose some semantic information, leading to a decrease in retrieval accuracy. This paper addresses these shortcomings. Summary of the Invention

[0003] In view of the above-mentioned problems, the present invention is proposed.

[0004] Therefore, the technical problem solved by this invention is to address the issues of low efficiency and insufficient accuracy of traditional text-to-video retrieval methods in large-scale video databases.

[0005] To address the aforementioned technical problems, this invention provides the following technical solution: a method for retrieving text from videos using multi-tag extraction and pre-screening, comprising:

[0006] Extract tags from text and video data, generate tag sets, and filter the tag sets;

[0007] Feature extraction is performed on videos and input text in the video library to generate multi-label probability distributions for videos and text;

[0008] Construct an inverted index from tags to video IDs, and match the generated multi-tag probability distribution with the tags in the inverted index;

[0009] Matching results based on the input text tag set, and related videos.

[0010] As a preferred embodiment of the multi-tag extraction and pre-screening text-to-video retrieval method described in this invention, the extraction of tags from text and video data includes: identifying nouns in the text as text tags; and extracting video features from the video content through frame extraction and visual converter processing to form video tags.

[0011] The tag set generation process includes generating a collection containing all text and video noun tags, and designing a selective retention mechanism.

[0012] The selective retention mechanism includes filtering tags based on a preset minimum frequency threshold and a limit on the number of high-frequency tags, generating an optimized subset of tags, as expressed by the formula:

[0013]

[0014] The generated tag set is then intersected with the text and video tags, as shown in the formula:

[0015] A'=A∩K'

[0016] Where K' represents the filtered subset of tags, K = {k1, k2, ..., k S} represents the specific elements of the tag set K', S represents the number of tags contained in the filtered tag set K', A represents the tag set of a specific text, and A' represents the refined tag set.

[0017] As a preferred embodiment of the multi-tag extraction and pre-screening text-to-video retrieval method of the present invention, wherein: the feature extraction of videos in the video library and input text includes video feature extraction and text feature extraction;

[0018] The video feature extraction includes using a video encoder to uniformly sample C frames from each video, simplifying the video content into a set of multiple frames, and processing each frame by the video encoder to obtain a global representation vector.

[0019] By averaging the global representations of all frames, the overall representation of the video and its high-level semantic features are calculated, as expressed by the formula:

[0020] V = {F1, F2, ..., F} c}

[0021]

[0022] Where V = {F1, F2, ..., F C} represents the set of frames in video V, C represents the number of frames in the video, and F represents the number of frames in the video. i This represents each frame as i = 1, 2, ..., C; Indicates frame F i The feature set, Indicates frame F i Classification characteristics, Indicates frame F i Other local features in the frame, where P represents the total number of blocks into which the frame is divided, f V : Represents the overall characteristics of video V, Indicates each frame F i Global classification features.

[0023] f V ∈R D f represents the overall features of the video V It belongs to the D-dimensional real space, where D represents the dimension of the eigenvectors;

[0024] The text feature extraction includes using a 9-layer transformer as a text encoder to process the input text and convert it into a corresponding text representation; and summarizing the representations of the entire text sequence to obtain a global representation of the text, expressed by the formula:

[0025]

[0026] Among them, g T Represents the global representation of text T. Represents the classification feature vector of text T; This represents the text representation obtained using classification tags. f represents other feature vectors in text T. i T Let represent a feature vector, where i = 1, 2, ..., S, represents the feature representation of the i-th tag extracted from the text; S represents the number of tags in the text.

[0027] As a preferred embodiment of the multi-label extraction and pre-screening text-to-video retrieval method described in this invention, wherein: the generation of the multi-label probability distribution of video and text includes calculating the multi-label probability, training target, and output probability distribution;

[0028] The calculation of multi-label probabilities includes inputting video representation and text representation into the multi-label header, and obtaining their respective multi-label probabilities, expressed by the formula:

[0029] p V =f V W+b

[0030] p T =f T W+b

[0031] Where T represents the input text; This represents a set of representations of the input text T after transformation, including the global representation of the text, where S represents the token length of the text, and f... T The final representation of the text is defined as the global representation. Right now W∈R D×K This represents the weight matrix for a multi-label header, with D rows and K columns; b∈R KThis represents the bias vector for multiple tag headers, with size K, where K represents the size of the tag set K', p V p represents the multi-label probability distribution of the video. T Represents the multi-label probability distribution of text;

[0032] The training objectives include video multi-label loss, text multi-label loss, and video-text contrast loss;

[0033] The target L is defined as:

[0034]

[0035] Video multi-label loss includes aligning the predicted label probabilities with the ground fact labels and employing asymmetric loss to address highly imbalanced label frequencies, as expressed in the formula:

[0036]

[0037] Where, when the k-th tag of tag list K' is in video tag set A', y k =1, otherwise y k =0, and

[0038]

[0039] in, Let represent the predicted multi-label probability of the video with the k-th label, and It is a transformed multi-label probability;

[0040] Where L represents the total loss function, Video multi-label asymmetric loss, L represents the multi-label asymmetric loss of text. vtc y represents the video-to-text contrast loss. k This represents the true value of the k-th label. This represents the loss for positive samples. Let α represent the loss of negative samples, K represent the focusing parameter, K represent the total number of labels, K′ represent the filtered subset of labels, A′ represent the true label set of the video, and τ represent the temperature parameter.

[0041] when Furthermore, when δ is a predefined scalar between 0 and 1, negative samples are excluded;

[0042] α + and α - The positive and negative focusing parameters are respectively. At high α, the simple positive contribution of p_V^k >> 0.5 to the loss is weakened.

[0043] In high α - Down, The simple negative contribution to the loss is reduced.

[0044] As a preferred embodiment of the multi-label extraction and pre-screening text-to-video retrieval method described in this invention, the video-text contrast loss includes aligning the video and text, enhancing the ability of the video and text encoders to capture cross-modal consistent semantics, expressed by the formula:

[0045]

[0046] Where L t2v and L v2t The text-to-video and video-to-text contrast losses are represented by the following formulas:

[0047]

[0048] Among them, L vtc L represents the video-to-text contrast loss. t2v L represents the contrast loss from text to video. v2t B represents the video-to-text contrast loss, and B represents the batch size. The feature vector representing the text. Let T represent the feature vector of the video, V represent the feature representation of the text, and τ represent the temperature parameter.

[0049] As a preferred embodiment of the multi-tag extraction and pre-screening text-to-video retrieval method described in this invention, the method of matching the generated multi-tag probability distribution with the tags in the inverted index includes, in an offline stage, scanning all videos in the video library, extracting the tags of each video, mapping the tags to the corresponding video IDs, and generating an inverted index.

[0050] When a text query is received, a set of tags is extracted from the query text, and these tags are used to look up the corresponding video ID column in the inverted index;

[0051] The inverted index quickly locates video IDs that overlap with the query text tags by mapping tags to video IDs;

[0052] These video IDs are merged to form a candidate video set.

[0053] As a preferred embodiment of the multi-tag extraction and pre-screening text-to-video retrieval method described in this invention, the matched relevant videos include: in the selected candidate video set, executing a text-to-video retrieval algorithm to ensure that the returned results have high semantic relevance.

[0054] A preferred embodiment of a multi-tag extraction and pre-screening text-to-video retrieval method, wherein:

[0055] The tag extraction module extracts tags from text and video data, generates a tag set, and filters the tags.

[0056] The multimodal multi-label module, comprising a video encoder and a text encoder, is used to extract features from video and text, generating multi-label probability distributions. The video encoder and text encoder extract global features from video frames and text, respectively, and then generate label probability distributions for the video and text using a shared multi-label header.

[0057] The inverted index construction and tag matching module offline builds an inverted index from tags to video IDs and uses this index for fast video matching by tags during the query phase. The inverted index structure associates text tags with video IDs to quickly filter candidate videos.

[0058] The text-to-video retrieval module executes standard text-to-video retrieval algorithms on the candidate video set, further comparing the semantics of the candidate videos with the input text to return the final matching video.

[0059] A computer device includes: a memory and a processor; the memory stores a computer program, characterized in that: when the processor executes the computer program, it implements the steps of the method described in any one of the present invention.

[0060] A computer-readable storage medium having a computer program stored thereon, characterized in that: when the computer program is executed by a processor, it implements the steps of the method described in any one of the present invention.

[0061] The beneficial effects of the present invention are as follows: The multi-tag extraction and pre-screening text-to-video retrieval method provided by the present invention converts video and text data into discrete semantic tags through a multi-tag extraction and pre-screening framework, and combines it with inverted index technology to achieve fast tag matching and video pre-screening, effectively reducing computational complexity. Attached Figure Description

[0062] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0063] Figure 1 The flowchart illustrates the overall process of a multi-tag extraction and pre-screening text-to-video retrieval method provided in the first embodiment of the present invention.

[0064] Figure 2A diagram illustrating the multi-model multi-tag generator for a text-to-video retrieval method for multi-tag extraction and pre-screening provided in the second embodiment of the present invention;

[0065] Figure 3 This diagram illustrates the tag matching process of a multi-tag extraction and pre-screening text-to-video retrieval method provided in the second embodiment of the present invention. Detailed Implementation

[0066] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the protection scope of the present invention.

[0067] Example 1, referring to Figure 1 As an embodiment of the present invention, a method for retrieving text from video using multi-tag extraction and pre-screening is provided, comprising:

[0068] S1: Extract tags from text and video data, generate a tag set, and filter the tag set.

[0069] Extracting tags from text and video data includes identifying nouns in the text as text tags, and extracting video features from video content through frame extraction and visual converter processing to form video tags.

[0070] The tag set generation process includes generating a collection containing all text and video noun tags, and designing a selective retention mechanism.

[0071] The selective retention mechanism includes filtering tags based on a preset minimum frequency threshold and a limit on the number of high-frequency tags, generating an optimized subset of tags, as expressed by the formula:

[0072]

[0073] The generated tag set is then intersected with the text and video tags, as shown in the formula:

[0074] A'=A∩K'

[0075] Where K' represents the filtered subset of tags, K = {k1, k2, ..., k S} represents the specific elements of the tag set K', S represents the number of tags contained in the filtered tag set K', A represents the tag set of a specific text, and A' represents the refined tag set.

[0076] Furthermore, by extracting labels from text and video data, generating a label set, and filtering it, the model ensures a balanced label distribution and reduces interference from irrelevant or excessively frequent labels. Text labels are obtained through noun recognition, while video labels are generated through feature extraction from frame extraction and a visual converter. The generated label set contains noun labels from all text and video data. Through a selective retention mechanism, based on a minimum frequency threshold and a limit on the number of high-frequency labels, the label subset is optimized to ensure semantic representativeness and model generalization. Finally, the optimized label set is intersected with the text and video labels to form a refined label set, providing high-quality label input for subsequent multi-label matching.

[0077] S2: Extract features from videos in the video library and input text to generate multi-label probability distributions for videos and text.

[0078] Feature extraction from videos in the video library and input text includes video feature extraction and text feature extraction;

[0079] The video feature extraction includes using a video encoder to uniformly sample C frames from each video, simplifying the video content into a set of multiple frames, and processing each frame by the video encoder to obtain a global representation vector.

[0080] By averaging the global representations of all frames, the overall representation of the video and its high-level semantic features are calculated, as expressed by the formula:

[0081] V = {F1, F2, ..., F} c}

[0082]

[0083] Where V = {F1, F2, ..., F C} represents the set of frames in video V, C represents the number of frames in the video, and F represents the number of frames in the video. i This represents each frame as i = 1, 2, ..., C; Indicates frame F i The feature set, Indicates frame F i Classification characteristics, Indicates frame F i Other local features in the frame, where P represents the total number of blocks into which the frame is divided, f V : Represents the overall characteristics of video V, Indicates each frame F i Global classification features.

[0084] f V ∈R D f represents the overall features of the video VIt belongs to the D-dimensional real space, where D represents the dimension of the eigenvectors;

[0085] The text feature extraction includes using a 9-layer transformer as a text encoder to process the input text and convert it into a corresponding text representation; and summarizing the representations of the entire text sequence to obtain a global representation of the text, expressed by the formula:

[0086]

[0087] Among them, g T Represents the global representation of text T. Represents the classification feature vector of text T; This represents the text representation obtained using classification tags. Represents other feature vectors in text T. Let represent a feature vector, where i = 1, 2, ..., S, represents the feature representation of the i-th tag extracted from the text; S represents the number of tags in the text.

[0088] The generation of multi-label probability distributions for videos and texts includes calculating multi-label probabilities, training targets, and output probability distributions.

[0089] The calculation of multi-label probabilities includes inputting video representation and text representation into the multi-label header, and obtaining their respective multi-label probabilities, expressed by the formula:

[0090] p V =f V W+b

[0091] p T =f T W+b

[0092] Where T represents the input text; This represents a set of representations of the input text T after transformation, including the global representation of the text, where S represents the token length of the text, and f... T The final representation of the text is defined as the global representation. Right now W∈R D×K This represents the weight matrix for a multi-label header, with D rows and K columns; b∈R K This represents the bias vector for multiple tag headers, with size K, where K represents the size of the tag set K', p V p represents the multi-label probability distribution of the video. T Represents the multi-label probability distribution of text;

[0093] The training objectives include video multi-label loss, text multi-label loss, and video-text contrast loss;

[0094] The target L is defined as:

[0095]

[0096] Video multi-label loss includes aligning predicted label probabilities with ground fact labels and employing asymmetric loss to address highly imbalanced label frequencies.

[0097]

[0098] The formula for text multi-label loss is expressed as:

[0099]

[0100] Where y k =1, if and only if the k-th tag in tag list K' is in video tag set A', otherwise y k =0, and

[0101]

[0102] in, Let represent the predicted multi-label probability of the video with the k-th label, and It is a transformed multi-label probability;

[0103] Where L represents the total loss function, Video multi-label asymmetric loss, L represents the multi-label asymmetric loss of text. vtc y represents the video-to-text contrast loss. k This represents the true value of the k-th label. This represents the loss for positive samples. Let represent the loss of negative samples, α represent the focusing parameter, K represent the total number of labels, K′ represent the filtered subset of labels, A′ represent the true label set of the video, and τ represent the temperature parameter.

[0104] when Furthermore, when δ is a predefined scalar between 0 and 1, negative samples are excluded;

[0105] α + and α - These are the positive and negative focusing parameters, respectively, at high α. + (or α) - )Down, (or The simple positive (or simple negative) contribution of the loss to the loss is reduced, leading to a greater focus on more challenging samples during training.

[0106] Video-text contrastive loss includes aligning video and text, enhancing the ability of video and text encoders to capture cross-modal consistent semantics, expressed as:

[0107]

[0108] Where L t2v and L v2t The text-to-video and video-to-text contrast losses are represented by the following formulas:

[0109]

[0110] Among them, L vtc L represents the video-to-text contrast loss. t2v L represents the contrast loss from text to video. v2t B represents the video-to-text contrast loss, and B represents the batch size. The feature vector representing the text. Let T represent the feature vector of the video, V represent the feature representation of the text, and τ represent the temperature parameter.

[0111] Furthermore, cross-modal matching is achieved through the calculation of multi-label probability distribution and contrastive loss to improve the accuracy and efficiency of text-to-video retrieval. First, feature extraction is performed on video frames and text tags using video encoders and text encoders respectively, generating global representations of video and text. These global representations are input into a multi-label header to generate a multi-label probability distribution for label prediction of video and text. During this process, an asymmetric loss function is used to handle the imbalance in label frequencies, enabling the model to focus more on less common or hard-to-classify labels. In addition, the video-text contrastive loss enhances the model's ability to capture consistent semantics across modalities by aligning video and text features, thus more effectively identifying semantic relevance between text and video. This design ensures the model's performance in multimodal environments, thereby improving the accuracy and robustness of text-to-video retrieval.

[0112] S3: Construct an inverted index from tags to video IDs, and match the generated multi-tag probability distribution with the tags in the inverted index.

[0113] Matching the generated multi-label probability distribution with the labels in the inverted index includes the following steps: in the offline stage, scanning all videos in the video library, extracting the labels for each video, mapping the labels to the corresponding video IDs, and generating an inverted index.

[0114] When a text query is received, a set of tags is extracted from the query text, and these tags are used to look up the corresponding video ID column in the inverted index;

[0115] The inverted index quickly locates video IDs that overlap with the query text tags by mapping tags to video IDs;

[0116] These video IDs are merged to form a candidate video set.

[0117] Furthermore, by constructing an inverted index that maps tags to video IDs, an efficient mapping relationship is established between each video tag in the video library and its corresponding video ID, enabling rapid matching and retrieval. Building the inverted index offline effectively reduces the computational burden during online queries. When a text query is received, by extracting the tags from the query text and searching for relevant video IDs in the inverted index, a set of candidate videos overlapping with the query tags can be quickly filtered out. This design significantly improves query efficiency, allowing the system to obtain videos semantically relevant to the query without traversing the entire video library, thus improving retrieval speed and accuracy. It is highly suitable for the efficient retrieval needs of large-scale video databases.

[0118] S4: Match relevant videos based on the matching results of the input text tag set.

[0119] The relevant videos to be matched include a text-to-video retrieval algorithm that is executed on the selected candidate video set to ensure that the returned results have high semantic relevance.

[0120] Furthermore, by executing a text-to-video retrieval algorithm on the selected candidate video set, the returned results are further ensured to have high semantic relevance. This process, which performs precise comparison of candidate videos after initial screening, effectively improves retrieval accuracy, avoids the possibility of mismatches, and ensures that the returned videos are highly semantically related to the input text. This refined matching mechanism compensates for any semantic biases that may exist during the initial screening process, thereby improving the overall retrieval quality of the system.

[0121] Example 2, an embodiment of the present invention, provides a multi-tag extraction and pre-screening text-to-video retrieval system, comprising:

[0122] The tag extraction module extracts tags from text and video data, generates a tag set, and filters the tags.

[0123] For a text T, we identify nouns as labels in the text and collect them into a label set A = {a1, a2, ...}, where a iLet K represent the i-th noun label. After collecting a label set for each text in the training dataset, we obtain a comprehensive set of noun labels, denoted as K. However, the noun labels in this set often exhibit an imbalanced distribution, with some nouns appearing more frequently than others. To mitigate the potential overfitting caused by this imbalance, we implement a selective retention mechanism for the set K. Specifically, we retain those that appear within a predefined range bounded by L and H, where L is the minimum frequency threshold and H represents the number of high-frequency noun labels to exclude. Following this retention mechanism, we obtain a subset of K, denoted as Ki.

[0124] Where S is the size of the set. The noun labels in K' are used as labels for subsequent label prediction tasks. Similarly, for the label set A extracted from the text, we further refine it by retaining the labels ai that exist in the label set K', generating a refined label set A' = A ∩ K';

[0125] Here, ∩ represents the intersection operation. Finally, each text T and its associated video V in the training dataset are paired with the refined label set A' to obtain pairs {T,A'} and {V,A'}.

[0126] The multimodal multi-label module, comprising a video encoder and a text encoder, is used to extract features from video and text, generating multi-label probability distributions. The video encoder and text encoder extract global features from video frames and text, respectively, and then generate label probability distributions for the video and text using a shared multi-label header.

[0127] Figure 2 The multimodal multi-labeler module is shown. This module contains a video encoder and a text encoder, used to extract video and text features, respectively. These features are then processed by a parameter-shared multi-label head that predicts labels for both modes. Specifically, we use a 12-layer visual transformer ViT-B / 16 as the video encoder. Given a video V, we first uniformly sample C frames from it, simplifying the video content to V = {F1, F2, ..., FC}. Then, each frame Fi is processed by ViT-B / 16 to obtain the representation. in For an input frame F of dimension D i The global representation is given by P, where P is the number of frame patches. Finally, we compute the video representation f by averaging the global representations of all sampled frames. V :

[0128]

[0129] f V ∈R D

[0130] For the text encoder, we use a 9-layer converter that transforms the text input T into a set of representations. in This is the global representation of the text, where S is the text tag length. Note that we are using the global text representation. As text representation f T ,Right now The parameter-sharing multi-label header consists of a weight matrix W∈R D×K and bias vector b∈R K It consists of a single linear layer, where K is the size of the label set K'. Video and text representations are input into the multi-label header, and their respective multi-label probabilities p are obtained. V and p T This process is described as follows:

[0131] p V =f V W+b

[0132] p T =f T W+b

[0133] The training of the multimodal multi-label module consists of three main training objectives: video multi-label loss. Text multi-label loss Loss of comparison with video text The target L is defined as:

[0134]

[0135] All loss terms contribute equally to the total loss of L.

[0136] The following details each loss item.

[0137] Video and text multi-label loss ( and We align the predicted label probabilities with the ground fact labels. Specifically, we employ asymmetric loss (ASL) to address the highly imbalanced label frequencies. For example, in multi-label video, the ASL loss is defined as:

[0138]

[0139] Where y k =1 if and only if the k-th tag in tag list K' is in video tag set A', otherwise y k =0, and

[0140]

[0141] in, It is the predicted multi-label probability of the video for the k-th label, while It is a transformed multi-label probability, when Furthermore, when δ is a predefined scalar between 0 and 1, negative samples are excluded. + and α - These are the positive and negative focusing parameters, respectively. At high α... + (or α) - )Down, >>0.5 (or The simple positive (or simple negative) contribution of the label to the loss is weakened, leading to a greater focus on more challenging samples during training. Similarly, we compute the multi-label loss for text, denoted as...

[0142] Video-text contrast loss (L vtc Aligning video and text representations enhances the ability of video and text encoders to capture consistent semantics across modalities. Furthermore, it helps multi-tag headers generate consistent tags for both video and text by using these aligned representations as input. vtc Represented as:

[0143]

[0144] Among them, L t2v and L v2t The text-to-video and video-to-text contrast losses are represented by the following formulas:

[0145]

[0146]

[0147] Where B is the batch size and τ is a learnable parameter.

[0148] The inverted index construction and tag matching module builds an inverted index offline, mapping tags to video IDs, and uses this index for fast video matching by tags during the query phase. The inverted index structure associates text tags with video IDs to quickly filter candidate videos.

[0149] Technically, we employ a strategy to build an inverted index that links tags to their corresponding video IDs. For a given text query, we directly query the inverted index using the extracted tags to generate a list of video IDs associated with those tags.

[0150] Figure 3The tag matching process is summarized, comprising two steps: inverted index creation and video pre-screening. Inverted Index Creation: An inverted index is built offline, mapping each tag to a corresponding video ID in the library. This process involves scanning all videos in the library, extracting their tags, and creating an index where each tag serves as a key pointing to a list of video IDs, and as a value associated with that tag. Since the number of video tags is discrete and finite, this inverted index can be stored efficiently with minimal space consumption. This structure ensures fast index access, enabling fast video search in response to text queries.

[0151] Video pre-scanning. When performing video pre-screening using a text query, we extract tags from the query and use these tags to look up an inverted index. For each tag in the query, the inverted index returns a list of associated video IDs. We then merge the lists of video IDs for all tags to create a unique set of video IDs that overlap with the query. This pre-screened set of videos forms a candidate library, which is then used as input, along with the query text, for subsequent text-to-video retrieval algorithms.

[0152] The text-to-video retrieval module executes standard text-to-video retrieval algorithms on the candidate video set, further comparing the semantics of the candidate videos with the input text to return the final matching video.

[0153] Example 3, an embodiment of the present invention, differs from the previous two embodiments in that:

[0154] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0155] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0156] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0157] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0158] Example 4: The experiment involved tag extraction, feature generation, inverted index construction, and tag matching on multiple video datasets, ultimately verifying the efficient retrieval performance based on input text in a large-scale video library. The experiment used a simulated video dataset with multiple different text queries as test conditions. The experimental dataset contained 1000 video clips of different types, covering various scenes such as natural landscapes, urban scenes, and human activities, to ensure tag diversity and a reasonable data volume. The experimental text queries included descriptions related to the video content, such as "city night view," "forest walk," and "sunrise at the beach," to test the system's semantic matching capabilities.

[0159] Tag Extraction: Tags were extracted for each video segment in the video library. The extracted tags covered key information about the video content, such as scene type, action description, and object names. After tag generation, a tag set for the video library was formed. To avoid bias caused by imbalanced tag distribution, a frequency filtering mechanism was used to remove tags with excessively high or low frequencies, ensuring model stability. Text queries also underwent tag extraction to form a query tag set for subsequent matching.

[0160] Feature extraction and multi-label probability distribution generation: A multimodal feature extraction module was used to extract features from the frame content of each video segment and the input text query, generating corresponding multi-label probability distributions. Video feature extraction was based on a ViT-B / 16 encoder, which converted video frames into global feature representations, generating label probability distributions containing key semantics. Text features were generated through a converter, encoding the multi-label probabilities of the text to ensure consistency in the semantic space.

[0161] Inverted Index Construction: In the offline phase, an inverted index structure was built, mapping tags to video IDs. Each tag is mapped to a set of associated video IDs, forming an inverted index relationship between tags and video IDs. The construction process of the inverted index significantly improves the processing efficiency of subsequent query phases.

[0162] Tag matching and pre-screening retrieval: When a query is input, the inverted index is used to match relevant video IDs based on the text query tag set. The candidate video set returned by the inverted index is then input into the retrieval module for further text-to-video comparison, thereby returning videos with high semantic relevance to the input text. This matching process further optimizes the accuracy of the retrieval based on the initial screening of relevant videos.

[0163] The experimental results are shown in Table 1.

[0164] Table 1 Experimental Data

[0165]

[0166] The experimental results show that the multi-tag extraction and pre-screened text-to-video retrieval method of this invention exhibits high retrieval accuracy and efficiency in various scenarios. Through comparative analysis of the data in the table, the exact matching rate of this invention is above 90%, and the recall and precision rates are both above 87%, indicating that this invention can quickly and accurately match semantically relevant videos in different text query scenarios. Furthermore, the retrieval time is less than 0.5 seconds, far lower than the multi-second processing time typically required by traditional real-valued methods, demonstrating the significant improvement in computational efficiency of this invention.

[0167] By extracting and filtering multiple tags, this method effectively reduces the impact of uneven tag distribution and improves tag coverage, thus ensuring the integrity of semantic information. Secondly, the construction of the inverted index significantly improves query efficiency, solving the problem of low retrieval efficiency in traditional methods on large-scale datasets. Finally, by performing further text-to-video retrieval on the candidate video set, high-precision semantic matching is achieved, avoiding semantic bias issues and demonstrating that the system maintains high retrieval accuracy while also considering processing speed. Compared with existing technologies, this invention not only has significant advantages in retrieval efficiency but also exhibits significant advantages in accuracy and semantic relevance, verifying the practical value and innovation of this invention in text-to-video retrieval tasks.

[0168] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for multi-tag extraction and pre-screening of text to video retrieval, characterized in that, include: Extract tags from text and video data, generate tag sets, and filter the tag sets; Feature extraction is performed on videos and input text in the video library to generate multi-label probability distributions for videos and text; Construct an inverted index from tags to video IDs, and match the generated multi-tag probability distribution with the tags in the inverted index; Based on the matching results of the input text tag set, relevant videos are obtained; The multi-label probability distribution of the generated video and text includes calculating the multi-label probability, training target, and output probability distribution; The calculation of multi-label probabilities includes inputting video representation and text representation into the multi-label header, and obtaining their respective multi-label probabilities, expressed by the formula: p V =f V W+b p T =f T W+b Among them, f V f represents the overall features of video V. T The final representation of the text is given by T, where T represents the input text; W ∈ R. D×K This represents the weight matrix for a multi-label header, with D rows and |K′| columns; b∈R |K′| This represents the bias vector for multiple tag headers, with size |K′|, where |K′| represents the size of the tag set K', p V p represents the multi-label probability distribution of the video. T Represents the multi-label probability distribution of the text; V={F1,F2,…,F C } represents the set of frames in video V, and C represents the number of frames in the video; The training objectives include video multi-label loss, text multi-label loss, and video and text contrast loss. The target L is defined as: The video multi-label loss aligns the predicted labels with the ground fact labels, expressed by the formula: Where, when the k-th tag of tag list K' is in video tag set A', y k =1, otherwise y k =0, and in, Let represent the predicted multi-label probability of the video with the k-th label, and It is a transformed multi-label probability; L represents the total loss function. Video multi-label asymmetric loss, L represents the multi-label asymmetric loss of text. vtc y represents the video-to-text contrast loss. k This represents the true value of the k-th label. This represents the loss for positive samples. Let K represent the loss for negative samples, α represent the focusing parameter, K represent the complete set of labels, and K′ represent the filtered subset of labels. when Furthermore, when δ is a predefined scalar between 0 and 1, negative samples are excluded; α + and α - These are the positive and negative focusing parameters, respectively, at high α. + Down, The simple positive contribution to the loss has weakened; In high α - Down, The simple negative contribution to the loss is reduced.

2. The method for multi-tag extraction and pre-screening text-to-video retrieval as described in claim 1, characterized in that: Extracting tags from text and video data includes identifying nouns in the text as text tags, and extracting video features from video content through frame extraction and visual converter processing to form video tags. The tag set generation process includes generating a collection containing all text and video noun tags, and designing a selective retention mechanism. The selective retention mechanism includes filtering tags based on a preset minimum frequency threshold and a limit on the number of high-frequency tags, generating an optimized subset of tags, as expressed by the formula: The generated tag set is then intersected with the text and video tags, as shown in the formula: A'=A∩K′ Where K′ represents the filtered subset of tags, S represents the number of tags contained in the filtered tag set K′, A represents the tag set of the text, and A′ represents the refined tag set.

3. The method for multi-tag extraction and pre-screening text-to-video retrieval as described in claim 2, characterized in that: The feature extraction of videos and input text from the video library includes video feature extraction and text feature extraction; The video feature extraction includes using a video encoder to uniformly sample C frames from each video, simplifying the video content into a set of multiple frames, and processing each frame by the video encoder to obtain a global representation vector. The overall representation of the video is calculated by averaging the global representations of all frames, as shown in the formula: V={F1,F2,……,F c } Where V = {F1, F2, ..., F C } represents the set of frames in video V, C represents the number of frames in the video, and F represents the number of frames in the video. i This represents the i-th frame in the video, where i = 1, 2, ..., C; Indicates frame F i The feature set, Indicates frame F i The classification feature, where P represents the total number of blocks into which the frame is divided; The text feature extraction includes using a 9-layer transformer as a text encoder to process the input text and convert it into a corresponding text representation; and summarizing the representations of the entire text sequence to obtain a global representation of the text, expressed by the formula: Among them, g T Represents the global representation of text T. The classification feature vector of text T; This represents a feature vector, which is the feature representation of the token extracted from the text T.

4. The method for multi-tag extraction and pre-screening text-to-video retrieval as described in claim 3, characterized in that: The video-text contrast loss includes aligning the video and text, enhancing the ability of the video and text encoders to capture cross-modal consistent semantics, as expressed by the formula: Among them, L t2v and L v2t The text-to-video and video-to-text contrast losses are represented by the following formulas: Among them, L vtc B represents the video-to-text contrast loss, and B represents the batch size. This represents the global feature representation of the m-th text sample in the batch. This represents the global feature representation of the m-th video sample in the batch. This represents the global feature representation of the nth video sample in the batch. This represents the global feature representation of the nth text sample in the batch, where m represents the main sample index, n represents the comparison sample index, and τ represents the temperature parameter.

5. The method for multi-tag extraction and pre-screening text-to-video retrieval as described in claim 4, characterized in that: Matching the generated multi-label probability distribution with the labels in the inverted index includes the following steps: in the offline stage, scanning all videos in the video library, extracting the labels for each video, mapping the labels to the corresponding video IDs, and generating an inverted index. When a text query is received, a set of tags is extracted from the query text, and these tags are used to look up the corresponding video ID column in the inverted index; The inverted index quickly locates video IDs that overlap with the query text tags by mapping tags to video IDs; These video IDs are merged to form a candidate video set.

6. The method for multi-tag extraction and pre-screening text-to-video retrieval as described in claim 5, characterized in that: The relevant videos to be matched include a text-to-video retrieval algorithm that is executed on the selected candidate video set to ensure that the returned results have high semantic relevance.

7. A multi-tag extraction and pre-screening text-to-video retrieval system, applied to the multi-tag extraction and pre-screening text-to-video retrieval method as described in any one of claims 1 to 6, characterized in that: The tag extraction module extracts tags from text and video data, generates a tag set, and filters the tags. The multimodal multi-label module includes a video encoder and a text encoder, which are used to extract features from video and text and generate multi-label probability distributions. The video encoder and the text encoder extract global features of video frames and text respectively, and then generate the label probability distributions of video and text through a shared multi-label header. The inverted index building and tag matching module builds an inverted index from tags to video IDs offline, and uses the inverted index to quickly match videos by tags during the query phase. By associating text tags with video IDs through the inverted index structure, candidate videos can be quickly filtered. The text-to-video retrieval module executes standard text-to-video retrieval algorithms on the candidate video set, further comparing the semantics of the candidate videos with the input text to return the final matching video.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, it implements the steps of the multi-tag extraction and pre-screening text-to-video retrieval method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the processor, it implements the steps of the multi-tag extraction and pre-screening text-to-video retrieval method according to any one of claims 1 to 6.

Images