Multi-label extraction and pre-screening text-to-video retrieval method and system

Abstract

The invention relates to the technical field of multi-modal data retrieval and cross-modal contrast learning, and discloses a retrieval method and system for multi-label extraction and pre-screening of texts to videos, and the method comprises the steps: extracting labels from text and video data, generating a label set, and screening the label set; performing feature extraction on the video and the input text in the video library to generate multi-label probability distribution of the video and the text; constructing a reverse index from the labels to the video ID, and matching the generated multi-label probability distribution with the labels in the reverse index; and matching related videos according to a matching result of the input text label set. Video and text data are converted into discrete semantic tags through a multi-tag extraction and pre-screening framework, and in combination with an inverted index technology, rapid tag matching and video pre-screening are realized, and the calculation complexity is effectively reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data retrieval and cross-modal contrastive learning, and in particular to a retrieval method and system for multi-label extraction and pre-screening of text to video. Background Art

[0002] With the rapid growth of multimedia data, especially the explosive growth of video content, how to quickly retrieve relevant videos from a huge video library based on natural language queries has become an important research direction. In the fields of social media, education, entertainment, and surveillance, the demand for text-to-video retrieval is increasing. At present, common text-to-video retrieval methods include real-valued methods and hash-based methods. The real-valued method calculates the similarity between text and video through the feature inner product, which can better retain semantic information, but its computational complexity is high, storage requirements are large, and scalability is poor; while the hash-based method uses binary hash codes to reduce storage and computational overhead, but may lose some semantic information, resulting in a decrease in retrieval accuracy. In response to these shortcomings. Summary of the invention

[0003] In view of the above-mentioned problems, the present invention is proposed.

[0004] Therefore, the technical problem solved by the present invention is to solve the problems of low efficiency and insufficient accuracy of traditional text-to-video retrieval methods in large-scale video databases.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions: a retrieval method for multi-label extraction and pre-screening text to video, comprising:

[0006] Extract labels from text and video data, generate label sets, and filter the label sets;

[0007] Extract features from videos and input texts in the video library to generate multi-label probability distributions of videos and texts;

[0008] Build an inverted index from labels to video IDs, and match the generated multi-label probability distribution with the labels in the inverted index;

[0009] Match related videos based on the matching results of the input text label set.

[0010] As a preferred solution of the multi-label extraction and pre-screening text to video retrieval method of the present invention, wherein: extracting labels from text and video data includes identifying nouns in the text as text labels, extracting video features through frame extraction and visual converter processing of video content, and then forming video labels;

[0011] Generating a label set includes generating a set of all text and video noun labels and designing a selective retention mechanism;

[0012] The selective retention mechanism includes screening tags based on a preset minimum frequency threshold and a limit on the number of high-frequency tags to generate an optimized tag subset, which is expressed as:

[0013]

[0014] The generated label set is intersected with the labels of text and video, and the formula is expressed as:

[0015] A'=A∩K'

[0016] Among them, K' represents the label subset after screening, K = {k 1 ,k 2 ,…,k S} represents the specific elements of the tag set K', S represents the number of tags contained in the filtered tag set K', A represents the tag set of a specific text, and A' represents the refined tag set.

[0017] As a preferred solution of the multi-label extraction and pre-screening text to video retrieval method of the present invention, wherein: the feature extraction of the video and input text in the video library includes video feature extraction and text feature extraction;

[0018] The video feature extraction includes using a video encoder to uniformly sample C frames from each video, simplifying the content of the video into a set of multiple frames, and obtaining a global representation vector after each frame is processed by the video encoder;

[0019] By averaging the global representations of all frames, the overall representation of the video is calculated to obtain the high-level semantic features of the video. The formula is:

[0020] V={F 1 ,F 2 ,……,F c}

[0021]

[0022] Where V = {F 1 ,F 2 ,…,F C} represents the frame set in the video V, C represents the number of frames in the video, F i Indicates each frame i=1,2,…,C; Indicates frame F i The feature set of Indicates frame F i The classification characteristics of Indicates frame F i Other local features in, P represents the total number of blocks the frame is divided into, f V : represents the overall characteristics of the video V, Indicates that each frame F i The global classification features.

[0023] f V ∈R D Represents the overall feature of the video f V Belongs to D-dimensional real space, where D represents the dimension of the eigenvector;

[0024] The text feature extraction includes using a 9-layer converter as a text encoder to process the input text and convert it into a corresponding text representation; by summarizing the representation of the entire text sequence, a global representation of the text is obtained, which is expressed as follows:

[0025]

[0026] Among them, g T represents the global representation of the text T, Represents the classification feature vector of text T; represents the text representation obtained using classification tags, Represents other feature vectors in the text T, f i T Represents a feature vector, where i = 1, 2, …, S, represents the feature representation of the i-th token extracted from the text; S represents the number of tokens in the text.

[0027] As a preferred solution of the multi-label extraction and pre-screening text to video retrieval method of the present invention, wherein: the generating multi-label probability distribution of video and text includes calculating multi-label probability, training target and output probability distribution;

[0028] The calculation of the multi-label probability includes inputting the video representation and the text representation into the multi-label head to obtain their respective multi-label probabilities, which are expressed as follows:

[0029] p V =f V W+b

[0030] p T =f T W+b

[0031] Where T represents the input text; represents a set of representations after the input text T is converted, which includes the global representation of the text, S represents the token length of the text, and f T The final representation of the text is defined as the global representation Right now W∈R D×K represents the weight matrix of the multi-label head, with D rows and K columns; b∈R K represents the bias vector of the multi-label head, with a size of K, where K represents the size of the label set K', and p V Represents the multi-label probability distribution of the video, p T Represents the multi-label probability distribution of text;

[0032] The training objectives include video multi-label loss, text multi-label loss and video-text contrast loss;

[0033] The target L is defined as:

[0034]

[0035] The video multi-label loss consists of aligning the predicted label probabilities with the ground truth labels and adopting an asymmetric loss to the highly unbalanced label frequencies, which is formulated as:

[0036]

[0037] Among them, when the kth tag in the tag list K' is in the video tag set A', y k =1, otherwise y k = 0, and

[0038]

[0039] in, represents the predicted video multi-label probability of the kth label, and is a transformed multi-label probability;

[0040] Among them, L represents the total loss function, Video multi-label asymmetric loss, represents the text multi-label asymmetric loss, L vtc represents the video-text contrast loss, y k represents the true value of the kth label, represents the loss of positive samples, represents the loss of negative samples, α represents the focus parameter, K represents the total number of labels, K′ represents the filtered sub-label set, A′ represents the true label set of the video, and τ represents the temperature parameter;

[0041] when When δ is a predefined scalar between 0 and 1, negative samples are excluded;

[0042] α + and α -are the positive and negative focusing parameters, respectively. At high α_+, p_V^k>>0.5 weakens the simple positive contribution to the loss;

[0043] At high α - Down, The simple negative contribution to the loss weakens.

[0044] As a preferred solution of the multi-label extraction and pre-screening text to video retrieval method of the present invention, the video-text contrast loss includes aligning the video and text, enhancing the ability of the video and text encoders to capture cross-modal consistent semantics, and the formula is expressed as:

[0045]

[0046] Where L t2v and L v2t They represent the text-to-video and video-to-text contrast losses respectively, and the formula is:

[0047]

[0048] Among them, L vtc represents the video-text contrast loss, L t2v represents the contrastive loss from text to video, L v2t represents the contrast loss from video to text, B represents the batch size, The feature vector representing the text, represents the feature vector of the video, T represents the feature representation of the text, V represents the feature representation of the video, and τ represents the temperature parameter.

[0049] As a preferred solution of the multi-label extraction and pre-screening text to video retrieval method of the present invention, wherein: matching the generated multi-label probability distribution with the labels in the inverted index includes, in an offline stage, scanning all videos in the video library, extracting the label of each video, and mapping the label to the corresponding video ID to generate an inverted index;

[0050] When a text query is received, a set of tags is extracted from the query text, and these tags are used to find the corresponding video ID column in the inverted index;

[0051] The inverted index uses the mapping relationship between tags and video IDs to quickly locate video IDs that overlap with the query text tags;

[0052] These video IDs are merged to form a candidate video set.

[0053] As a preferred solution of the multi-label extraction and pre-screening text-to-video retrieval method described in the present invention, the matching related videos include executing a text-to-video retrieval algorithm in the screened candidate video set to ensure that the returned results have high semantic relevance.

[0054] A preferred solution of a multi-label extraction and pre-screening text to video retrieval method, wherein:

[0055] The label extraction module extracts labels from text and video data, generates label sets, and filters labels;

[0056] The multimodal multi-labeler module includes a video encoder and a text encoder, which are used to extract features from videos and texts and generate multi-label probability distributions. The video encoder and the text encoder extract global features of video frames and texts respectively, and then generate label probability distributions of videos and texts through a shared multi-label head;

[0057] The inverted index construction and label matching module builds an inverted index from labels to video IDs offline, and uses the inverted index to quickly match videos by labels during the query phase. Through the inverted index structure, text labels are associated with video IDs to quickly filter candidate videos;

[0058] The text-to-video retrieval module executes the standard text-to-video retrieval algorithm on the candidate video set, and further compares the semantics of the candidate videos with the input text to return the final matching video.

[0059] A computer device comprises: a memory and a processor; the memory stores a computer program, wherein the processor implements the steps of any one of the methods of the present invention when executing the computer program.

[0060] A computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of any one of the methods of the present invention.

[0061] Beneficial effects of the present invention: The multi-label extraction and pre-screening text-to-video retrieval method provided by the present invention converts video and text data into discrete semantic labels through a multi-label extraction and pre-screening framework, and combines the inverted index technology to achieve fast label matching and video pre-screening, effectively reducing the computational complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0063] Figure 1 An overall flow chart of a multi-label extraction and pre-screening text-to-video retrieval method provided by the first embodiment of the present invention;

[0064] Figure 2 A module diagram illustrating a multi-model multi-labeler for a multi-label extraction and pre-screening text-to-video retrieval method provided by a second embodiment of the present invention;

[0065] Figure 3 A label matching process diagram of a multi-label extraction and pre-screening text-to-video retrieval method provided in a second embodiment of the present invention. DETAILED DESCRIPTION

[0066] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0067] Example 1, reference Figure 1 , as an embodiment of the present invention, provides a multi-label extraction and pre-screening text to video retrieval method, comprising:

[0068] S1: Extract labels from text and video data, generate label sets, and filter the label sets.

[0069] Extracting labels from text and video data includes identifying nouns in the text as text labels, extracting video features through frame extraction and visual converter processing of video content, and then forming video labels;

[0070] Generating a label set includes generating a set of all text and video noun labels and designing a selective retention mechanism;

[0071] The selective retention mechanism includes screening tags based on a preset minimum frequency threshold and a limit on the number of high-frequency tags to generate an optimized tag subset, which is expressed as:

[0072]

[0073] The generated label set is intersected with the labels of text and video, and the formula is expressed as:

[0074] A'=A∩K'

[0075] Among them, K' represents the label subset after screening, K = {k 1 ,k 2 ,…,k S} represents the specific elements of the tag set K', S represents the number of tags contained in the filtered tag set K', A represents the tag set of a specific text, and A' represents the refined tag set.

[0076] Furthermore, by extracting labels from text and video data, generating label sets and screening them, we ensure the balance of label distribution and reduce the interference of irrelevant or over-frequency labels on the model. Text labels are obtained through noun recognition, and video labels are generated through frame extraction and features extracted by visual converters. The generated label set contains noun labels for all texts and videos. Through the selective retention mechanism, the label subset is optimized based on the minimum frequency threshold and the number of high-frequency labels to ensure the semantic representativeness of the labels and the generalization of the model. Finally, the optimized label set is intersected with the labels of the text and video to form a refined label set, providing high-quality label input for subsequent multi-label matching.

[0077] S2: Extract features from videos and input texts in the video library to generate multi-label probability distributions of videos and texts.

[0078] Feature extraction of videos and input texts in the video library includes video feature extraction and text feature extraction;

[0079] The video feature extraction includes using a video encoder to uniformly sample C frames from each video, simplifying the content of the video into a set of multiple frames, and obtaining a global representation vector after each frame is processed by the video encoder;

[0080] By averaging the global representations of all frames, the overall representation of the video is calculated to obtain the high-level semantic features of the video. The formula is:

[0081] V={F 1 ,F 2 ,……,F c}

[0082]

[0083] Where V = {F 1 ,F 2 ,…,F C} represents the frame set in the video V, C represents the number of frames in the video, Fi Indicates each frame i=1,2,…,C; Indicates frame F i The feature set of Indicates frame F i The classification characteristics of Indicates frame F i Other local features in, P represents the total number of blocks the frame is divided into, f V : represents the overall characteristics of the video V, Indicates that each frame F i The global classification features.

[0084] f V ∈R D Represents the overall feature of the video f V Belongs to D-dimensional real space, where D represents the dimension of the eigenvector;

[0085] The text feature extraction includes using a 9-layer converter as a text encoder to process the input text and convert it into a corresponding text representation; by summarizing the representation of the entire text sequence, a global representation of the text is obtained, which is expressed as follows:

[0086]

[0087] Among them, g T represents the global representation of the text T, Represents the classification feature vector of text T; represents the text representation obtained using classification tags, represents other feature vectors in the text T, Represents a feature vector, where i = 1, 2, …, S, represents the feature representation of the i-th token extracted from the text; S represents the number of tokens in the text.

[0088] Generating multi-label probability distributions for videos and texts includes calculating multi-label probabilities, training targets, and outputting probability distributions;

[0089] The calculation of the multi-label probability includes inputting the video representation and the text representation into the multi-label head to obtain their respective multi-label probabilities, which are expressed as follows:

[0090] p V =f V W+b

[0091] p T =f T W+b

[0092] Where T represents the input text; represents a set of representations after the input text T is converted, which includes the global representation of the text, S represents the token length of the text, and f T The final representation of the text is defined as the global representation Right now W∈R D×K represents the weight matrix of the multi-label head, with D rows and K columns; b∈R K represents the bias vector of the multi-label head, with a size of K, where K represents the size of the label set K', and p V Represents the multi-label probability distribution of the video, p T Represents the multi-label probability distribution of text;

[0093] The training objectives include video multi-label loss, text multi-label loss and video-text contrast loss;

[0094] The target L is defined as:

[0095]

[0096] The video multi-label loss includes,aligning the predicted label probabilities with the ground truth labels,using an asymmetric loss to deal with highly imbalanced label frequencies;

[0097]

[0098] The formula for text multi-label loss is expressed as:

[0099]

[0100] where y k =1, if and the kth label of the label list K' is in the video label set A', otherwise y k = 0, and

[0101]

[0102] in, represents the predicted video multi-label probability of the kth label, and is a transformed multi-label probability;

[0103] Among them, L represents the total loss function, Video multi-label asymmetric loss, represents the text multi-label asymmetric loss, L vtc represents the video-text contrast loss, y k represents the true value of the kth label, represents the loss of positive samples, represents the loss of negative samples, α represents the focus parameter, K represents the total number of labels, K′ represents the filtered sub-label set, A′ represents the true label set of the video, and τ represents the temperature parameter.

[0104] when When δ is a predefined scalar between 0 and 1, negative samples are excluded;

[0105] α + and α - are positive and negative focusing parameters respectively. + (or α - )Down, (or )’s contribution to the loss is weakened, resulting in more attention being paid to more challenging samples during training.

[0106] The video-text contrast loss involves aligning video and text, which enhances the ability of video and text encoders to capture cross-modal consistent semantics. The formula is expressed as:

[0107]

[0108] Where L t2v and L v2t They represent the text-to-video and video-to-text contrast losses respectively, and the formula is:

[0109]

[0110] Among them, L vtc represents the video-text contrast loss, L t2v represents the contrastive loss from text to video, L v2t represents the contrast loss from video to text, B represents the batch size, The feature vector representing the text, represents the feature vector of the video, T represents the feature representation of the text, V represents the feature representation of the video, and τ represents the temperature parameter.

[0111] Furthermore, cross-modal matching is achieved through the calculation of multi-label probability distribution and contrastive loss to improve the accuracy and efficiency of text-to-video retrieval. First, the video encoder and text encoder are used to extract features from video frames and text tags respectively to generate global representations of video and text. These global representations are input into the multi-label head to generate multi-label probability distributions for label prediction of video and text. In this process, an asymmetric loss function is used to deal with the imbalance of label frequency, allowing the model to pay more attention to rare or difficult-to-classify labels. In addition, the video-text contrastive loss enhances the model's ability to capture cross-modal consistent semantics by aligning video and text features, thereby more effectively identifying the semantic relevance between text and video. This design ensures the performance of the model in a multimodal environment, thereby improving the accuracy and robustness of text-to-video retrieval.

[0112] S3: Build an inverted index from labels to video IDs, and match the generated multi-label probability distribution with the labels in the inverted index.

[0113] Matching the generated multi-label probability distribution with the labels in the inverted index includes, in the offline stage, scanning all videos in the video library, extracting the label of each video, and mapping the label to the corresponding video ID to generate the inverted index;

[0114] When a text query is received, a set of tags is extracted from the query text, and these tags are used to find the corresponding video ID column in the inverted index;

[0115] The inverted index uses the mapping relationship between tags and video IDs to quickly locate video IDs that overlap with the query text tags;

[0116] These video IDs are merged to form a candidate video set.

[0117] Furthermore, by constructing an inverted index from tags to video IDs, an efficient mapping relationship is established between each video tag in the video library and its corresponding video ID, thereby achieving fast matching and retrieval. Constructing an inverted index in the offline stage effectively reduces the computational burden of online queries; and when receiving a text query, by extracting the tags of the query text and searching for related video IDs in the inverted index, the candidate video set that overlaps with the query tag can be quickly screened out. This design greatly improves query efficiency, allowing the system to obtain videos related to the query semantics without traversing the entire video library, improving retrieval speed and accuracy, and is very suitable for efficient retrieval needs in large-scale video databases.

[0118] S4: Matched related videos based on the matching results of the input text label set.

[0119] Matching related videos includes executing text-to-video retrieval algorithms on the screened candidate video set to ensure that the returned results have high semantic relevance.

[0120] Furthermore, by executing the text-to-video retrieval algorithm on the selected candidate video set, we can further ensure that the returned results have high semantic relevance. This process can effectively improve the accuracy of retrieval by accurately comparing the candidate videos after preliminary screening, avoid the possibility of mismatching, and ensure that the returned videos are highly semantically relevant to the input text. This sophisticated matching mechanism compensates for the semantic deviation that may exist in the preliminary screening process, thereby improving the overall retrieval quality of the system.

[0121] Embodiment 2 is an embodiment of the present invention, which provides a retrieval system for multi-label extraction and pre-screening text to video, including:

[0122] The label extraction module extracts labels from text and video data, generates label sets, and filters labels.

[0123] For a text T, we identify nouns as tags in the text and collect them into a tag set A = {a 1 ,a 2 ,····}, where a i Denotes the i-th noun label. After collecting the label set for each text in the training dataset, we obtain a comprehensive set of noun labels, denoted as K. However, the noun labels in this set usually exhibit an unbalanced distribution, with some nouns appearing more frequently than others. To mitigate the potential overfitting caused by this imbalance, we implement a selective retention mechanism for the set K. Specifically, we retain those that appear within a predefined range bounded by L and H, where L is the minimum frequency threshold and H represents the number of high-frequency noun labels to be excluded. Following this retention mechanism, we obtain a subset of K, denoted as

[0124] Where S is the size of the set. The noun tags in K' are used as the tags for the subsequent tag prediction task. Similarly, for the tag set A extracted from the text, we also refine it by retaining the tags ai present in the tag set K', generating a refined tag set A' = A ∩ K';

[0125] Among them, ∩ represents the intersection operation. Finally, each text T and its related video V in the training dataset are paired with the refined label set A' to obtain pairs {T, A'} and {V, A'}.

[0126] The multimodal multi-labeler module includes a video encoder and a text encoder, which are used to extract features from videos and texts and generate multi-label probability distributions. The video encoder and the text encoder extract global features of video frames and texts respectively, and then generate label probability distributions of videos and texts through a shared multi-label head.

[0127] Figure 2 The multimodal multi-labeler module is shown. The module contains a video encoder and a text encoder, which are used to extract video and text features, respectively. These features are then processed by a parameter-shared multi-label head that predicts labels for both modalities. Specifically, we use a 12-layer visual transformer ViT-B / 16 as the video encoder. When given a video V, we first uniformly sample C frames from it and simplify the video content to V = {F1, F2, ···, FC}. Then, each frame Fi is processed by ViT-B / 16 to obtain the representation in For the input frame F with dimension D i The global representation of , P is the number of frame patches. Finally, we calculate the video representation f by averaging the global representations of all sampled frames V :

[0128]

[0129] f V ∈R D

[0130] For the text encoder, we use a 9-layer transformer that transforms the text input T into a set of representations in is the global representation of the text, and S is the token length of the text. Note that we use the global text representation As text representation f T ,Right now The parameter-sharing multi-label head consists of a weight matrix W∈R D×K and the bias vector b∈R K The video and text representations are input into the multi-label head to obtain their respective multi-label probabilities p V and p T . This process is expressed as:

[0131] p V =f V W+b

[0132] p T =f T W+b

[0133] The training of the multimodal multi-labeler module consists of three main training objectives: video multi-label loss Text multi-label loss Comparative loss with video text The target L is defined as:

[0134]

[0135] where all loss terms contribute equally to the total loss of L.

[0136] The following is a detailed description of each loss item.

[0137] Video and text multi-label loss ( and ) aligns the predicted label probabilities with the ground truth labels. Specifically, we adopt an asymmetric loss (ASL) to highly imbalanced label frequencies. Taking video multi-label as an example, its ASL loss is defined as:

[0138]

[0139] where y k = 1 when the kth tag in the tag list K' is in the video tag set A', otherwise y k = 0, and

[0140]

[0141] in, is the predicted video multi-label probability of the kth label, and is a transformed multi-label probability, when When α is a predefined scalar between 0 and 1, negative samples are excluded. + and α - are positive focusing and negative focusing parameters respectively. + (or α - )Down, >>0.5(or )’s contribution to the loss is weakened, resulting in more attention to more challenging samples during training. Similarly, we calculate the multi-label loss for text, denoted as

[0142] Video-text contrast loss (L vtc ) aligns video and text representations, enhancing the ability of video and text encoders to capture consistent semantics across modalities. In addition, it helps the multi-label head generate consistent labels for video and text by using these aligned representations as input. vtc It is expressed as:

[0143]

[0144] Among them, L t2v and L v2t They represent the text-to-video and video-to-text contrast losses respectively, and the formula is:

[0145]

[0146]

[0147] Where B is the batch size and τ is a learnable parameter.

[0148] The inverted index construction and label matching module builds an inverted index from labels to video IDs offline, and uses the inverted index to quickly match videos by labels during the query phase. Through the inverted index structure, text labels are associated with video IDs to quickly filter candidate videos.

[0149] Technically, we apply a strategy to build an inverted index that links tags to the corresponding video IDs they are assigned to. For a given text query, we directly query the inverted index with the extracted tags to generate a list of video IDs associated with these tags.

[0150] Figure 3 The tag matching process is summarized and consists of two steps: inverted list creation and video pre-screening. Inverted index creation. An inverted index is built offline to map each tag to the corresponding video ID in the gallery. This process involves scanning all videos in the gallery, extracting their tags, and creating an index with each tag as a key pointing to a list of video IDs and as the value associated with that tag. Since the number of video tags is discrete and finite, this inverted index can be stored efficiently, taking up minimal space. This structure ensures that the index can be accessed quickly, thereby enabling fast video search in response to text queries.

[0151] Video Pre-scanning. When performing video pre-screening using a text query, we extract tags from the query and use these tags to look up the inverted index. For each tag in the query, the inverted index returns a list of associated video IDs. We then merge the lists of video IDs for all tags to create a set of unique video IDs that have overlapping tags with the query. This set of pre-screened videos forms a candidate gallery, which is then used as input, along with the query text, for the subsequent text-to-video retrieval algorithm.

[0152] The text-to-video retrieval module executes the standard text-to-video retrieval algorithm on the candidate video set, and further compares the semantics of the candidate videos with the input text to return the final matching video.

[0153] Embodiment 3, an embodiment of the present invention, is different from the first two embodiments in that:

[0154] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0155] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0156] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0157] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0158] Example 4, the experiment finally verified the efficient retrieval effect based on input text in a large-scale video library by performing label extraction, feature generation, inverted index construction and label matching on multiple video data sets. The experiment used a simulated video data set and used multiple different text queries as test conditions. The experimental data set contained 1,000 different types of video clips, covering a variety of scenes such as natural landscapes, urban scenes, and human activities to ensure the rationality of label diversity and data volume. The experimental text queries included descriptions related to the video content, such as "city night view", "forest walk", "sunrise at the seaside", etc., to test the semantic matching ability of the system.

[0159] Label extraction: Labels are extracted for each video clip in the video library. The extracted labels cover key information of the video content, such as scene type, action description, and object name. After the labels are generated, a label set of the video library is formed. In order to avoid deviations caused by unbalanced label distribution, a frequency screening mechanism is adopted to remove labels with too high or too low frequency to ensure the stability of the model. Labels are also extracted for text queries to form a query label set for subsequent matching processes.

[0160] Feature extraction and generation of multi-label probability distribution: Through the multimodal feature extraction module, the frame content of each video clip and the input text query are feature extracted to generate the corresponding multi-label probability distribution. The video feature extraction is based on the visual converter ViT-B / 16 encoder, which converts the video frame into a global feature representation and generates a label probability distribution containing key semantics; the text feature is generated through the converter, and the multi-label probability of the text is encoded to ensure consistency in the semantic space.

[0161] Inverted index construction: In the offline phase, an inverted index structure from tags to video IDs is constructed. Each tag is mapped to the set of video IDs related to it, forming an inverted index relationship from tags to video IDs. The inverted index construction process greatly improves the processing efficiency of the subsequent query phase.

[0162] Tag matching and pre-screening retrieval: When a query is entered, the relevant video IDs in the inverted index are matched according to the text query tag set. The candidate video set returned by the inverted index is input into the retrieval module for further text-to-video comparison, thereby returning videos with high semantic relevance to the input text. This matching process further optimizes the accuracy of retrieval based on the screening of preliminary relevant videos.

[0163] The experimental results are shown in Table 1.

[0164] Table 1 Experimental data table

[0165]

[0166] It can be seen from the experimental results that the multi-label extraction and pre-screening text to video retrieval method of the present invention shows high retrieval accuracy and efficiency in a variety of scenarios. Through the comparative analysis of the data in the table, the exact match rate of the method of the present invention is above 90%, and the recall rate and precision rate are both maintained above 87%, indicating that in different text query scenarios, the present invention can quickly and accurately match semantically related videos. In addition, the retrieval time is less than 0.5 seconds, which is much lower than the multi-second processing time usually required by traditional real-valued methods, reflecting the significant improvement in computing efficiency of the present invention.

[0167] Through multi-label extraction and screening, this method effectively reduces the impact of uneven label distribution and improves label coverage, thereby ensuring the integrity of semantic information; secondly, the construction of inverted index significantly improves query efficiency and solves the problem of low retrieval efficiency of traditional methods on large-scale data sets; finally, by performing further text-to-video retrieval on the candidate video set, high-precision semantic matching is achieved, avoiding the problem of semantic deviation, reflecting that the system takes into account processing speed while maintaining high retrieval accuracy. Compared with the prior art, the present invention not only has obvious advantages in retrieval efficiency, but also shows significant advantages in accuracy and semantic relevance, verifying the practical value and innovation of the present invention in text-to-video retrieval tasks.

[0168] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multi-label extraction and pre-screening text to video retrieval method, characterized in that: include: Extract labels from text and video data, generate label sets, and filter the label sets; Extract features from videos and input texts in the video library to generate multi-label probability distributions of videos and texts; Build an inverted index from labels to video IDs, and match the generated multi-label probability distribution with the labels in the inverted index; Match related videos based on the matching results of the input text label set.

2. The multi-label extraction and pre-screening text to video retrieval method as claimed in claim 1, characterized in that: Extracting labels from text and video data includes identifying nouns in the text as text labels, extracting video features through frame extraction and visual converter processing of video content, and then forming video labels; Generating a label set includes generating a set of all text and video noun labels and designing a selective retention mechanism; The selective retention mechanism includes screening tags based on a preset minimum frequency threshold and a limit on the number of high-frequency tags to generate an optimized tag subset, which is expressed as: The generated label set is intersected with the labels of text and video, and the formula is expressed as: A'=A∩K' Among them, K' represents the filtered label subset, K = {k1, k2, ..., k S } represents the specific elements of the tag set K', S represents the number of tags contained in the filtered tag set K', A represents the tag set of a specific text, and A' represents the refined tag set.

3. The multi-label extraction and pre-screening text to video retrieval method as claimed in claim 2, characterized in that: The feature extraction of the video in the video library and the input text includes video feature extraction and text feature extraction; The video feature extraction includes using a video encoder to uniformly sample C frames from each video, simplifying the content of the video into a set of multiple frames, and obtaining a global representation vector after each frame is processed by the video encoder; By averaging the global representations of all frames, the overall representation of the video is calculated to obtain the high-level semantic features of the video. The formula is: V={F1,F2,……,F c } Where V = {F1, F2, …, F C } represents the frame set in the video V, C represents the number of frames in the video, and F i Indicates each frame i=1,2,…,C; Indicates frame F i The feature set of Indicates frame F i The classification characteristics of Indicates frame F i Other local features in, P represents the total number of blocks the frame is divided into, f V : represents the overall characteristics of video V, Indicates that each frame F i The global classification features. f V ∈R D Represents the overall feature of the video f V Belongs to D-dimensional real space, where D represents the dimension of the eigenvector; The text feature extraction includes using a 9-layer converter as a text encoder to process the input text and convert it into a corresponding text representation; by summarizing the representation of the entire text sequence, a global representation of the text is obtained, which is expressed as follows: Among them, g T represents the global representation of the text T, Represents the classification feature vector of text T; represents the text representation obtained using classification tags, Represents other feature vectors in the text T, f i T Represents a feature vector, where i = 1, 2, …, S, represents the feature representation of the i-th token extracted from the text; S represents the number of tokens in the text.

4. The multi-label extraction and pre-screening text to video retrieval method as claimed in claim 3, characterized in that: Generating multi-label probability distribution of video and text includes calculating multi-label probability, training target and outputting probability distribution; The calculation of the multi-label probability includes inputting the video representation and the text representation into the multi-label head to obtain their respective multi-label probabilities, which are expressed as follows: p V =f V W+b p T =f T W+b Where T represents the input text; represents a set of representations after the input text T is converted, which includes the global representation of the text, S represents the token length of the text, and f T The final representation of the text is defined as the global representation Right now W∈R D×K represents the weight matrix of the multi-label head, with D rows and K columns; b∈R K represents the bias vector of the multi-label head, with a size of K, where K represents the size of the label set K', and p V Represents the multi-label probability distribution of the video, p T Represents the multi-label probability distribution of text; The training objectives include video multi-label loss, text multi-label loss and video-text contrast loss; The target L is defined as: The video multi-label loss consists of aligning the predicted label probabilities with the ground truth labels and using an asymmetric loss to deal with highly unbalanced label frequencies. The formula is: Among them, when the kth tag in the tag list K' is in the video tag set A', y k =1, otherwise y k = 0, and in, represents the predicted video multi-label probability of the kth label, and is a transformed multi-label probability; Among them, L represents the total loss function, Video multi-label asymmetric loss, represents the text multi-label asymmetric loss, L vtc represents the video-text contrast loss, y k represents the true value of the kth label, represents the loss of positive samples, represents the loss of negative samples, α represents the focus parameter, K represents the total number of labels, K′ represents the filtered sub-label set, A′ represents the true label set of the video, and τ represents the temperature parameter; when When δ is a predefined scalar between 0 and 1, negative samples are excluded; α + and α - are the positive and negative focusing parameters, respectively. At high α_+, p_V^k>>0.5 weakens the simple positive contribution to the loss; At high α - Down, The simple negative contribution to the loss weakens.

5. The multi-label extraction and pre-screening text to video retrieval method as claimed in claim 4, characterized in that: The video-text contrast loss includes aligning the video and text, enhancing the ability of the video and text encoders to capture cross-modal consistent semantics, and is expressed as: Where L t2v and L v2t They represent the text-to-video and video-to-text contrast losses respectively, and the formula is: Among them, L vtc represents the video-text contrast loss, L t2v represents the contrastive loss from text to video, L v2t represents the contrast loss from video to text, B represents the batch size, The feature vector representing the text, represents the feature vector of the video, T represents the feature representation of the text, V represents the feature representation of the video, and τ represents the temperature parameter.

6. The multi-label extraction and pre-screening text to video retrieval method as claimed in claim 5, characterized in that: Matching the generated multi-label probability distribution with the labels in the inverted index includes, in the offline stage, scanning all videos in the video library, extracting the label of each video, and mapping the label to the corresponding video ID to generate the inverted index; When a text query is received, a set of tags is extracted from the query text, and these tags are used to find the corresponding video ID column in the inverted index; The inverted index uses the mapping relationship between tags and video IDs to quickly locate video IDs that overlap with the query text tags; These video IDs are merged to form a candidate video set.

7. The multi-label extraction and pre-screening text to video retrieval method as claimed in claim 6, characterized in that: Matching related videos includes executing text-to-video retrieval algorithms on the screened candidate video set to ensure that the returned results have high semantic relevance.

8. A multi-label extraction and pre-screening text-to-video retrieval system, characterized by: The label extraction module extracts labels from text and video data, generates label sets, and filters labels; The multimodal multi-labeler module includes a video encoder and a text encoder, which are used to extract features from videos and texts and generate multi-label probability distributions. The video encoder and the text encoder extract global features of video frames and texts respectively, and then generate label probability distributions of videos and texts through a shared multi-label head; The inverted index construction and label matching module builds an inverted index from labels to video IDs offline, and uses the inverted index to quickly match videos by labels during the query phase. Through the inverted index structure, text labels are associated with video IDs to quickly filter candidate videos; The text-to-video retrieval module executes the standard text-to-video retrieval algorithm on the candidate video set, and further compares the semantics of the candidate videos with the input text to return the final matching video.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the multi-label extraction and pre-screening text to video retrieval method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-label extraction and pre-screening text to video retrieval method described in any one of claims 1 to 7 are implemented.

Images