Label sequence generation method based on sequence prompt and retrieval enhancement generation

Abstract

The invention provides a tag sequence generation method based on sequential prompt and retrieval enhancement generation, and belongs to the technical field of data processing, the method comprises the following steps: inputting a target video into a tag sequence generation model, and after the tag sequence generation model obtains frame features and text features of the target video, generating a tag sequence; carrying out feature fusion on the frame features and the text features to obtain multi-modal mixed features; on the basis of the multi-modal mixed features, generating a sequence prompt of the target video, the sequence prompt being used for prompting key attention features of the target video in sequence; after related videos are retrieved from the multiple retrieval systems, prompt statements are generated based on tags of the related videos, and the related videos are videos related to the target video; and generating and outputting a tag sequence of the target video based on the multi-modal mixed features, the sequential prompt and the prompt statement. The method does not depend on a label set, the problem of label long tails can be effectively relieved, and label diversity and correlation are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a tag sequence generation method based on sequential prompting and retrieval enhancement generation. Background Art

[0002] With the rapid development of social media and content sharing platforms, images and short videos have become the main forms of user interaction and online content creation. These platforms face the huge challenge of how to effectively represent and manage massive multimodal content.

[0003] Existing multimodal content representation methods mainly include two categories: one is vectorized representation and the other is labeled representation. Vectorized representation encodes images or videos through multi-dimensional vectors, which is suitable for efficient tasks such as recognition and clustering, but usually lacks interpretability and cross-platform compatibility. Labeled representation uses keywords or phrases provided by users or platforms to intuitively summarize the topic and key features of the content, and has good interpretability. Therefore, multi-label classification methods are widely used in content annotation of images and videos, especially in tasks oriented to predefined categories. These methods usually rely on backbone networks such as CNN (Convolutional Neural Network), GCN (Graph Convolutional Network) and Transformer, and infer multiple labels simultaneously through multiple classification heads or multiple binary classification heads. For example, for the multi-label classification task of images, researchers have proposed many solutions based on deep neural networks. However, most of these methods rely on predefined fixed sets of labels, and it is difficult to effectively handle large-scale long-tail distributed open labels. In addition, with the rise of short video platforms, the number of labels often far exceeds the predefined categories, which makes traditional multi-label classification methods face huge challenges in dealing with content diversity and label generalization capabilities.

[0004] Therefore, how to alleviate the long-tail distribution in large-scale video sets and improve the generalization of label generation has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present invention provides a tag sequence generation method based on sequential prompting and retrieval enhanced generation, so as to solve the defect that it is difficult to effectively process large-scale long-tail distributed open tags in the prior art.

[0006] The present invention provides a tag sequence generation method based on sequential prompting and retrieval enhancement generation, comprising the following steps: The target video is input into the label sequence generation model. After acquiring the frame features and text features of the target video, the label sequence generation model performs feature fusion on the frame features and the text features to obtain multimodal mixed features, wherein the frame features are obtained by encoding the sampled frames in the target video, and the text features are obtained by encoding the title of the target video; Based on the multimodal mixed features, a sequential prompt of the target video is generated, wherein the sequential prompt is used to sequentially prompt key focus features of the target video; After relevant videos are retrieved from multiple retrieval systems, a prompt sentence is generated based on tags of the relevant videos, wherein the relevant videos are videos related to the target video; Based on the multimodal mixed features, the sequential prompts and the prompt sentences, a label sequence of the target video is generated and output.

[0007] According to a label sequence generation method based on sequential prompting and retrieval enhancement generation provided by the present invention, after acquiring the frame features and text features of the target video, the frame features and the text features are subjected to feature fusion to obtain multimodal mixed features, including: Encoding the sampled frames in the target video using a frozen visual encoder to obtain frame features output by the visual encoder; Encoding the title of the target video to obtain text features output by the visual encoder; A multimodal hybrid encoder is used to perform feature fusion on the frame features and the text features to obtain multimodal hybrid features. The multimodal hybrid encoder includes multiple layers of transformer encoders, and each layer of the transformer encoder includes a multi-head self-attention module, a cross attention head and a feedforward neural network.

[0008] According to a label sequence generation method based on sequential prompts and retrieval enhancement generation provided by the present invention, the sequential prompts of the target video are generated based on the multimodal mixed features, including: Inputting a set of randomly initialized learnable queries and the multimodal mixed features into a prompt encoder to obtain an intermediate vector output by the prompt encoder, wherein the prompt encoder comprises a multi-layer transformer encoder, and the prompt encoder uses the learnable query as a query and the multimodal mixed features as a key and a value; The intermediate vectors are mapped using a linear projection function to obtain sequential prompts of the target video.

[0009] According to a label sequence generation method based on sequential prompting and retrieval enhancement generation provided by the present invention, after retrieving relevant videos from multiple retrieval systems, generating prompt sentences based on the labels of the relevant videos includes: For any retrieval system, k relevant videos are retrieved from the retrieval system, and a label and a relevance score of each of the k relevant videos are obtained, where k is a positive integer; After k relevant videos are retrieved from multiple retrieval systems respectively, for each tag, a cumulative relevance score of the tag is calculated, where the cumulative relevance score is the sum of the relevance scores of the relevant videos having the tag; The p tags with the highest cumulative correlation scores are concatenated to obtain a prompt statement, where p is a positive integer.

[0010] According to a label sequence generation method based on sequential prompting and retrieval enhancement generation provided by the present invention, the label sequence generation model is trained based on the following steps: The sample video is input into the label sequence generation model to be trained. After the label sequence generation model to be trained generates the multimodal mixed features and sequential prompts of the sample video, all the real labels and all the sequential prompts of the sample video are binary matched to calculate the first loss value; Based on the multimodal mixed features, the sequential prompts and the generated prompt sentences, a label sequence of the target video is generated, and a second loss value is calculated; determining a total loss value based on the first loss value and the second loss value; Based on the total loss value, adjusting the model parameters of the label sequence generation model to be trained until the total loss value is less than a preset threshold; The model parameters whose total loss value is less than the preset threshold are used as model parameters of the label sequence generation model to be trained to obtain the label sequence generation model.

[0011] According to a tag sequence generation method based on sequential prompting and retrieval enhancement generation provided by the present invention, the first loss value is determined based on the following steps: Taking the true label as an anchor point and the sequential prompt as a positive and negative sample, for any true label, determining a true label loss value based on a ratio of a first similarity to a second similarity, wherein the first similarity is the similarity between the true label and the positive sample, and the second similarity is the sum of the similarities between the true label and all candidate samples; Taking the sequential prompt as an anchor point and the true label as a positive and negative sample, for any sequential prompt, determining a sequential prompt loss value based on a ratio of a third similarity to a fourth similarity, wherein the third similarity is the similarity between the sequential prompt and the positive sample, and the fourth similarity is the sum of the similarities between the sequential prompt and all candidate samples; Based on the true label loss value and the sequential prompt loss value, a first loss value is determined.

[0012] The present invention also provides a label sequence generation device based on sequential prompting and retrieval enhancement generation, comprising the following modules: A multimodal hybrid coding module is used to: input the target video into a label sequence generation model, and after acquiring the frame features and text features of the target video, the label sequence generation model performs feature fusion on the frame features and the text features to obtain multimodal hybrid features, wherein the frame features are obtained by encoding the sampled frames in the target video, and the text features are obtained by encoding the title of the target video; A sequential prompt encoding module, used to: generate sequential prompts of the target video based on the multimodal mixed features, wherein the sequential prompts are used to sequentially prompt key features of the target video; A prompt statement generating module is used to: after retrieving relevant videos from multiple retrieval systems, generate prompt statements based on tags of the relevant videos, wherein the relevant videos are videos related to the target video; The label sequence generation module is used to generate and output the label sequence of the target video based on the multimodal mixed features, the sequential prompts and the prompt statements.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for generating a label sequence based on sequential prompts and retrieval enhancement generation as described in any one of the above-mentioned methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for generating a label sequence based on sequential prompts and retrieval enhancement generation.

[0015] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described methods for generating a label sequence based on sequential prompts and retrieval enhancement.

[0016] The label sequence generation method based on sequential prompts and retrieval enhancement generation provided by the present invention inputs a target video into a label sequence generation model. After acquiring frame features and text features of the target video, the label sequence generation model performs feature fusion on the frame features and the text features to obtain multimodal mixed features, wherein the frame features are obtained by encoding sampled frames in the target video, and the text features are obtained by encoding the title of the target video; based on the multimodal mixed features, sequential prompts of the target video are generated, and the sequential prompts are used to sequentially prompt key features of the target video; after relevant videos are retrieved from multiple retrieval systems, prompt sentences are generated based on the labels of the relevant videos, and the relevant videos are videos related to the target video; based on the multimodal mixed features, the sequential prompts and the prompt sentences, a label sequence of the target video is generated and output. The present invention generates sequential prompts based on multimodal mixed features, and then generates a label sequence based on the sequential prompts. Without relying on a label set, the dependency between labels is flexibly modeled in a generative manner, thereby improving the accuracy of label generation, especially for rare labels, effectively alleviating the long-tail problem of labels and improving label diversity and relevance. In addition, new labels can be inferred and generated based on prompt statements, thereby improving the generalization of label generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of a tag sequence generation method based on sequential prompting and retrieval enhancement generation provided by the present invention; Figure 2 It is a structural schematic diagram of the label sequence generation model provided by the present invention; Figure 3 It is a schematic diagram of the process of generating prompt statements provided by the present invention; Figure 4 It is a schematic diagram of the process of binary matching of sequential titles and real labels provided by the present invention; Figure 5 This is one of the model simulation experiment performance comparison diagrams provided by the present invention; Figure 6 This is the second comparison chart of the model simulation experiment performance provided by the present invention; Figure 7 It is a schematic diagram of label gains at the scene level and sample level of different models provided by the present invention; Figure 8 It is a structural schematic diagram of a label sequence generation device based on sequential prompting and retrieval enhancement generation provided by the present invention; Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] It should be noted that in the description of the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or position relationship indicated by the terms "upper", "lower" and the like is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be a connection between two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0021] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" means at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0022] Combine the following Figure 1-Figure 9 The tag sequence generation method based on sequential prompts and retrieval enhancement generation provided by an embodiment of the present invention is described.

[0023] Figure 1 is a flow chart of a tag sequence generation method based on sequential prompting and retrieval enhancement generation provided by the present invention, such as Figure 1 As shown, the method includes the following: S110, inputting the target video into a label sequence generation model, and after acquiring the frame features and text features of the target video, the label sequence generation model performs feature fusion on the frame features and the text features to obtain a multimodal hybrid feature, wherein the frame features are obtained by encoding a sampled frame in the target video, and the text features are obtained by encoding a title of the target video; S120, generating a sequential prompt of the target video based on the multimodal mixed features, wherein the sequential prompt is used to sequentially prompt key focus features of the target video; S130, after relevant videos are retrieved from multiple retrieval systems, a prompt sentence is generated based on tags of the relevant videos, wherein the relevant videos are videos related to the target video; S140, generating and outputting a label sequence of the target video based on the multimodal mixed features, the sequential prompts and the prompt sentences.

[0024] It should be noted that the execution subject of the tag sequence generation method based on sequential prompts and retrieval enhancement generation provided in the embodiment of the present invention can be a server, a computer device, such as a mobile phone, a tablet computer, a laptop computer, a PDA, an ultra-mobile personal computer (UMPC), a netbook or a personal digital assistant (PDA), etc.

[0025] In S110, the target video is a video for which a label sequence generation is required. Sampling is performed at a preset frequency from the target video to obtain a plurality of sample frames, and the sample frames are encoded to obtain frame features. Since video labels often contain specific entity concepts (such as specific celebrities or places), it is difficult to identify these concepts solely relying on visual information, and the title or description of the video usually contains these entity words. Therefore, the title of the target video is encoded and the text features of each word are extracted. The frame features and the text features are fused to obtain multimodal hybrid features that fully capture the multimodal information in the video.

[0026] In S120, the sequential hint is additional information added to guide the label sequence generation model to focus on specific features or image regions. Different from traditional visual hints (shared between samples), in the embodiment of the present invention, the sequential hint contains specific content of the target video, that is, the sequential hint is designed to contain specific content of each video, thereby making it a sample-specific hint.

[0027] In S130, in combination with an external database, video-to-video, title-to-title, and cross-modal video-to-title retrieval are used to search for relevant videos from multiple angles and generate prompt sentences, thereby broadening the scope of tag recommendations.

[0028] Understandably, videos containing similar objects will often share the same labels. For example, if the model has never encountered the dog breed "Springer Spaniel" during training, it will not be able to generate the label "Springer Spaniel". However, by finding similar videos through visual similarity retrieval, labels can be extracted from the retrieved videos as hints, thereby increasing the chances of generating previously unseen labels.

[0029] The label sequence generation method based on sequential prompts and retrieval enhancement generation provided by an embodiment of the present invention inputs a target video into a label sequence generation model. After acquiring the frame features and text features of the target video, the label sequence generation model performs feature fusion on the frame features and the text features to obtain multimodal mixed features, wherein the frame features are obtained by encoding the sampled frames in the target video, and the text features are obtained by encoding the title of the target video; based on the multimodal mixed features, sequential prompts of the target video are generated, and the sequential prompts are used to sequentially prompt the key features of the target video; after retrieving relevant videos from multiple retrieval systems, prompt sentences are generated based on the labels of the relevant videos, and the relevant videos are videos related to the target video; based on the multimodal mixed features, the sequential prompts and the prompt sentences, a label sequence of the target video is generated and output. The present invention generates sequential prompts based on multimodal mixed features, and then generates a label sequence based on the sequential prompts. Without relying on a label set, the dependency between labels is flexibly modeled in a generative manner, thereby improving the accuracy of label generation, especially for rare labels, effectively alleviating the long-tail problem of labels and improving label diversity and relevance. In addition, new labels can be inferred and generated based on prompt statements, thereby improving the generalization of label generation.

[0030] In an optional embodiment, after acquiring the frame features and text features of the target video, feature fusion is performed on the frame features and the text features to obtain multimodal mixed features, including: Encoding the sampled frames in the target video using a frozen visual encoder to obtain frame features output by the visual encoder; Encoding the title of the target video to obtain text features output by the visual encoder; A multimodal hybrid encoder is used to perform feature fusion on the frame features and the text features to obtain multimodal hybrid features. The multimodal hybrid encoder includes multiple layers of transformer encoders, and each layer of the transformer encoder includes a multi-head self-attention module, a cross attention head and a feedforward neural network.

[0031] Figure 2 is a schematic diagram of the structure of the label sequence generation model provided by the present invention, such as Figure 2 As shown in (a) Multimodal Hybrid Encoding, from image / video Sampling is performed in the image and frame features are obtained through the frozen visual encoder (Visual Encoder) Here, the visual encoder is pre-trained, and the visual encoder is frozen to ensure that the weights of the trained visual encoder are not updated. Through the textual encoder, the title / description (fight and run, even pandas have learned the rules of human survival) is encoded to extract the textual features of each word (Textual Features) .

[0032] In the embodiment of the present invention, a multimodal hybrid encoder is used. Perform feature fusion and multi-modal hybrid encoder It is a multi-layer transformer encoder, each layer includes a multi-head self-attention module, a cross attention head and a feed-forward network. Frame features and text features are obtained by Fusion is performed, where text features are used as queries and frame features are used as keys and values.

[0033] The label sequence generation method based on sequential prompts and retrieval enhancement generation provided in an embodiment of the present invention obtains frame features through a frozen visual encoder, takes text features as queries and frame features as keys and values, and uses a multi-layer transformer encoder to generate multimodal mixed features, which provides a basis for subsequent sequential prompt encoding and label sequence decoding.

[0034] In an optional embodiment, the generating of the sequential prompts of the target video based on the multimodal mixed features includes: Inputting a set of randomly initialized learnable queries and the multimodal mixed features into a prompt encoder to obtain an intermediate vector output by the prompt encoder, wherein the prompt encoder comprises a multi-layer transformer encoder, and the prompt encoder uses the learnable query as a query and the multimodal mixed features as a key and a value; The intermediate vectors are mapped using a linear projection function to obtain sequential prompts of the target video.

[0035] like Figure 2 As shown in (b) Order Prompt Encoding, the initialized learnable queries interact with multimodal hybrid features to generate learnable, sample-specific order prompts. Specifically: Define a set of learnable queries ， These queries are shared across all videos and are randomly initialized. Order Tips By combining these learnable queries With multimodal mixed features Interaction results in: ; Among them, the prompt encoder is a criss-cross attention module consisting of multiple layers of transformers that uses a learnable query As a query, use multimodal mixed features As keys and values. Function Represents a linear projection.

[0036] It should be noted that, unlike traditional visual cues, the sequential cues in the embodiment of the present invention are designed to include specific content of each video, thereby making them sample-specific cues.

[0037] Here, learnable queries allow the model to dynamically generate query vectors that can be used in the attention mechanism to adapt to the needs of specific tasks. Compared to fixed or predefined queries, learnable queries can be automatically adjusted and optimized through the training process to better capture the complex patterns and relationships in the input data. Specifically, a randomly initialized vector is assigned to each query, and the query is regarded as an embedding of a feature, mapped to the appropriate dimension through a linear transformation. During the attention calculation process, the query vector interacts with the key and value to produce a new feature representation. As the model training progresses, the query vector is continuously adjusted to minimize the loss function and improve the model performance.

[0038] The label sequence generation method based on sequential prompts and retrieval enhanced generation provided by the embodiment of the present invention interacts the initialized learnable query with the multimodal hybrid features to generate learnable, sample-specific sequential prompts, providing implicit guidance for label sequence generation.

[0039] In an optional embodiment, after the relevant videos are retrieved from the multiple retrieval systems, a prompt statement is generated based on the tags of the relevant videos, including: For any retrieval system, k relevant videos are retrieved from the retrieval system, and a label and a relevance score of each of the k relevant videos are obtained, where k is a positive integer; After k relevant videos are retrieved from multiple retrieval systems respectively, for each tag, a cumulative relevance score of the tag is calculated, where the cumulative relevance score is the sum of the relevance scores of the relevant videos having the tag; The p tags with the highest cumulative correlation scores are concatenated to obtain a prompt statement, where p is a positive integer.

[0040] like Figure 2 As shown in (c) Multi-source Tag Recommendation, after retrieving multiple tags from N retrieval systems, the tag cumulative relevance score (Tag Score Accumulation) is calculated, and the first k tags are sorted and selected to obtain the prompt sentence Prompt.

[0041] Figure 3 It is a schematic diagram of the flow chart of generating the prompt statement provided by the present invention, such as Figure 3 As shown in Figure 2, we assume that the database is an open world dataset containing videos and their corresponding labels. A retrieval system is used to search for videos similar to the query video. A retrieval system that returns the most relevant top videos, their labels and relevance scores. The formal expression is as follows: ; in, Indicates Videos, Indicates the corresponding label, Indicates that from Since each retrieval system has its own scoring criteria, the relevance score Normalization is performed within each retrieval system. By combining multiple retrieval systems, the accuracy and diversity of the retrieval results are enhanced.

[0042] Furthermore, the final output of each retrieval system is aggregated to form a comprehensive collection of similar videos: ; If a video is recommended by multiple retrieval systems, it indicates that it has significant similarities with the query video in multiple aspects, thereby increasing its weight in the final output.

[0043] To construct consensus tags and their cumulative relevance scores, aggregate the relevance scores of the same tags in all retrieval systems. Specifically: For each unique tag Initialize the cumulative score: ; Furthermore, the relevance score of each tag is accumulated: ; in, Representation list The tags, is an indicator function, when 1 when it is, otherwise 0; Then a label set and its cumulative score are formed: ; The top scorer Tags: ; in Yes First in descending order A collection of tags; In order to give Target video generates prompt sentence , put the front Tags in Splice together: .

[0044] The label sequence generation method based on sequential prompts and retrieval enhanced generation provided in the embodiment of the present invention dynamically integrates external knowledge bases through the retrieval enhanced generation method, compensates for label generation errors caused by knowledge loss during the generation process, and improves the diversity and coverage of label generation.

[0045] Based on any of the above embodiments, the label sequence generation model is trained based on the following steps: The sample video is input into the label sequence generation model to be trained. After the label sequence generation model to be trained generates the multimodal mixed features and sequential prompts of the sample video, all the real labels and all the sequential prompts of the sample video are binary matched to calculate the first loss value; Based on the multimodal mixed features, the sequential prompts and the generated prompt sentences, a label sequence of the target video is generated, and a second loss value is calculated; determining a total loss value based on the first loss value and the second loss value; Based on the total loss value, adjusting the model parameters of the label sequence generation model to be trained until the total loss value is less than a preset threshold; The model parameters whose total loss value is less than the preset threshold are used as model parameters of the label sequence generation model to be trained to obtain the label sequence generation model.

[0046] In the embodiment of the present invention, in order to ensure that the sequential hints cover diverse semantic information, bipartite matching is used to uniquely align each sequential hint with the unordered video label.

[0047] Optionally, a total loss value is determined based on the sum of the first loss value and the second loss value: ; in, represents the first loss value, Represents the second loss value.

[0048] Figure 4 is a flow chart of binary matching of sequential titles and real labels provided by the present invention, such as Figure 4 As shown, Represents the true label of Embeddings, Video of Using a frozen pre-trained text representation model To extract the embedding of the label, . In order to make the number of sequential prompts and label embeddings match ( ), through the predefined meaningless label [PAD] Fill in. Function Represents a collection of sequential prompts To the tag embedding collection The index mapping of .

[0049] To find a bipartite match between these two sets, we search for an order using the following cost function , which is minimized: ; in, is the pairwise matching cost, calculated as the sequential cue embedding Embedded with true labels The cosine similarity between .

[0050] Optionally, the Hungarian algorithm is used to find the optimal match.

[0051] In the embodiment of the present invention, the final alignment is expressed as , the corresponding label is defined as: .

[0052] In the embodiment of the present invention, since there is no explicit alignment annotation between the label embedding and the sequential prompt, a self-supervised learning method is used to allow the model to automatically learn alignment from unlabeled data. The initial prompt and label alignment is obtained by bipartite matching, and the aligned pairs are assumed as positive samples and the unaligned pairs as negative samples. The contrastive learning method is used, and the InfoNCE loss is applied to bring the positive sample pairs closer and push the negative sample pairs further away.

[0053] Furthermore, once the alignment between the sequential cues and the true labels is established, the labels are rearranged according to the sequential cues to generate a comma-delimited sequence of target labels: .

[0054] It should be noted that the matching of sequential prompts and labels is only used during the model training process to semantically align prompts and labels. This process is discarded when the model is specifically applied.

[0055] The label sequence generation method based on sequential prompts and retrieval enhancement generation provided by the embodiment of the present invention optimizes the sequential prompts and label comparison loss. and label sequence generation loss Training the entire model end-to-end helps inject semantic information into sequential cues and enables the model to generate ordered sequences of labels based on these cues.

[0056] In an optional embodiment, the first loss value is determined based on the following steps: Taking the true label as an anchor point and the sequential prompt as a positive and negative sample, for any true label, determining a true label loss value based on a ratio of a first similarity to a second similarity, wherein the first similarity is the similarity between the true label and the positive sample, and the second similarity is the sum of the similarities between the true label and all candidate samples; Taking the sequential prompt as an anchor point and the true label as a positive and negative sample, for any sequential prompt, determining a sequential prompt loss value based on a ratio of a third similarity to a fourth similarity, wherein the third similarity is the similarity between the sequential prompt and the positive sample, and the fourth similarity is the sum of the similarities between the sequential prompt and all candidate samples; Based on the true label loss value and the sequential prompt loss value, a first loss value is determined.

[0057] In the embodiment of the present invention, the loss function formula is as follows: ; ; ; in, Using labels as anchors, sequential cues are considered as positive and negative samples, while Perform the opposite operation, is the first loss value; function Represents a learnable temperature parameter The index similarity of represents the batch size. Only hints associated with valid labels are considered valid positive pairs, where .

[0058] In an optional embodiment, the order-aware tag decoder, , constructed through a multi-layer Transformer decoder, is used to sequentially generate a sequence of target labels, one label at a time. This generation process uses multimodal mixed features , Sequence prompt and prompt statements is the condition, guiding the model to generate a label sequence based on the prompt. Predict the next label The probability is expressed as: .

[0059] Optionally, to train the autoregressive generative model, the second loss value is calculated using a cross entropy loss function.

[0060] It should be noted that the frequent appearance of [PAD] tags in the sequence may cause the model to take shortcuts and over-focus on these tags to converge quickly. To solve this problem, a lower weight is given to the [PAD] tag and a higher weight is given to meaningful tags. The weights are defined as follows: ; in, ; Then the weighted loss function for label sequence generation is: .

[0061] In addition, it should be noted that the model will generate a label sequence that may contain the [PAD] label. The generated sequence is post-processed by splitting it at the comma and removing all [PAD] labels. The remaining labels are the final output label sequence.

[0062] The effect of the tag sequence generation method based on sequential prompts and retrieval enhancement generation provided by the present invention is described below in conjunction with simulation experiments.

[0063] Experiment 1: The tag sequence generation model based on multimodal retrieval enhancement (Open-Tag) is trained on the CREATE-Tag dataset to evaluate its performance in the video tag generation task.

[0064] Here, CREATE-Tag is a multimodal video tagging dataset derived from the CREATE dataset, which is a large-scale Chinese short video retrieval and caption generation dataset. CREATE-Tag contains 162K training samples and 5K test samples, with 17,573 unique tags, of which 2,795 tags are shared between the training set and the test set. These tags are generated through the visual information of the video, the title information, and the retrieval-enhanced prompts.

[0065] In order to evaluate the generalization ability of the model, the label set of the CREATE-Tag dataset is divided into 705 common labels and 2,090 rare labels, and a closed set evaluation method is adopted. For each video, the model generates a label sequence based on the video content, title, and retrieved similar video labels, and the output is a comma-separated label sequence.

[0066] Here, the Open-Tag model uses the CLIP-B / 32 image encoder as the frozen image encoder and a 6-layer Transformer as the caption encoder. The multimodal hybrid encoder and the prompt encoder also consist of 6-layer Transformers, while the sequence-aware label decoder is a 12-layer autoregressive Transformer. The label generation process is conditioned on the multimodal hybrid features, sequential prompts, and retrieval-enhanced prompt sentences, and the target label sequence is gradually generated through the decoder.

[0067] On the CREATE-Tag dataset, the model was first pre-trained for 20 rounds with a batch size of 1024 using 8 NVIDIA V100 GPUs. After pre-training, the model was fine-tuned for 30 rounds with a batch size of 512 and the hardware configuration remained unchanged. During the training process, the AdamW optimizer was used, the maximum learning rate was set to 2e^(-4), the weight decay was 0.002, the loss weight adjustment parameter λ was set to 0.3, and the number of sequential cues was set to 30. After training, the model was evaluated on the test set to calculate the accuracy and diversity of label generation. By comparing the label distribution of the training set and the test set, some rare labels in the training set appeared more frequently in the test set, indicating that the model has good generalization ability.

[0068] Experiment 2: The Open-Tag model is applied to the PEXEL-Tag dataset for image tag generation tasks.

[0069] Here, PEXEL-Tag is a new multimodal image tag dataset from the Pexels platform, containing 162K training samples and 5K test samples. Each image is accompanied by a title and related tags, with an average of 5.52 tags per image. The dataset contains a total of 28,094 unique tags, of which 27,752 tags appear in the training set and 6,003 tags appear in the test set.

[0070] In order to evaluate the generalization ability of the model, the label set of the PEXEL-Tag dataset is divided into 1627 common labels and 4042 rare labels for evaluation. Unlike the CREATE-Tag dataset, the label distribution of the PEXEL-Tag dataset in the training set and the test set is relatively consistent, and the label distribution of the test set is similar to that of the training set.

[0071] Here, for the PEXEL-Tag dataset, the structure of the Open-Tag model is the same as the implementation for the CREATE-Tag dataset above. The only difference is that the image patch features are used as input in the image labeling task instead of the video frame features. The model is trained on the PEXEL-Tag dataset for 20 epochs with a batch size of 512 using 8 NVIDIA V100 GPUs. After the model is trained, its performance is evaluated on the PEXEL-Tag dataset, with a special focus on the performance of the model on common and rare tags. The performance of the model in the image labeling task is verified by calculating the accuracy and diversity of the generated tags.

[0072] Figure 5 This is one of the model simulation experiment performance comparison diagrams provided by the present invention; Figure 6This is the second comparison chart of the model simulation experiment performance provided by the present invention; Figure 7 It is a schematic diagram of label gains at the scene level and sample level of different models provided by the present invention; like Figure 5-Figure 6 As shown, Figure 5 This is a performance comparison chart on the CREATE-Tag dataset. Figure 6 This is a performance comparison chart on the PEXEL-Tag dataset. By introducing order prompts, the Open-Tag model can better capture the dependencies between tags, thereby improving the accuracy of tag generation, especially when dealing with common and rare tags. Experimental results show that the F1 score of Open-Tag on the CREATE-Tag dataset is 17.5% and 15.5% higher than that of the most advanced methods, effectively solving the long-tail problem of tags and improving tag diversity and relevance. On the PEXEL-Tag dataset, Open-Tag achieved the highest F1 score in all settings, especially in the recognition of rare tags, which was improved by 29.9% and 14.7% respectively compared with existing methods. This result highlights that Open-Tag has good generalization ability in different fields, can recognize more unseen tags, and still performs well in scenarios where tags are scarce; In the open set evaluation, traditional classification methods (such as Bin and ASL) cannot infer new labels because their classification heads are limited to a fixed set of labels, while generation-based methods (such as Order-Free and Open-Book) use word-based generation strategies to generate new labels. Compared with other generation models, Open-Tag performs best in the open set evaluation. By combining sequential prompts and retrieval-augmented generation technology (RAG), Open-Tag improves sample-level label gain and scene-level label gain by 64.2% and 14.2%, respectively.

[0073] In addition, experimental results show that Open-Tag can be effectively extended to large-scale pre-training tasks, especially when using CREATE-3M for pre-training, the model can still maintain good performance and stability. This shows that the present invention has good scalability and adaptability, can handle data sets of different sizes, and adapt to more complex label generation requirements.

[0074] In summary, the label sequence generation method based on sequential prompts and retrieval enhanced generation provided by the present invention improves the accuracy, diversity and generalization ability in label generation tasks through sequential prompts and retrieval enhanced generation technology, especially has significant advantages in the generation of rare labels and open set labels, and can provide efficient solutions in multiple visual and language fields.

[0075] The following is a description of a label sequence generation device based on sequential prompts and retrieval enhancement generation provided by an embodiment of the present invention. The label sequence generation device based on sequential prompts and retrieval enhancement generation described below and the label sequence generation method based on sequential prompts and retrieval enhancement generation described above can be referenced to each other.

[0076] Figure 8 is a schematic diagram of the structure of a label sequence generation device based on sequential prompting and retrieval enhancement generation provided by the present invention, such as Figure 8 As shown, the label sequence generation device based on sequential prompts and retrieval enhancement generation may include but is not limited to; The multimodal hybrid coding module 810 is used to: input the target video into the label sequence generation model, and after acquiring the frame features and text features of the target video, the label sequence generation model performs feature fusion on the frame features and the text features to obtain multimodal hybrid features, wherein the frame features are obtained by encoding the sampled frames in the target video, and the text features are obtained by encoding the title of the target video; A sequential prompt encoding module 820 is used to: generate a sequential prompt of the target video based on the multimodal mixed feature, wherein the sequential prompt is used to sequentially prompt the key focus features of the target video; The prompt statement generating module 830 is used to: after retrieving relevant videos from multiple retrieval systems, generate prompt statements based on tags of the relevant videos, wherein the relevant videos are videos related to the target video; The label sequence generation module 840 is used to generate and output the label sequence of the target video based on the multimodal mixed features, the sequential prompts and the prompt sentences.

[0077] It should be noted that the label sequence generation device based on sequential prompts and retrieval enhancement generation provided in an embodiment of the present invention can execute the label sequence generation method based on sequential prompts and retrieval enhancement generation described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0078] Fig. 9 An example of a physical structure diagram of an electronic device is shown in FIG. Fig. 9As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute a label sequence generation method based on sequential prompts and retrieval enhancement generation, the method comprising: inputting a target video into a label sequence generation model, the label sequence generation model after acquiring the frame features and text features of the target video, performs feature fusion on the frame features and the text features to obtain a multimodal hybrid feature, the frame features are obtained by encoding a sample frame in the target video, and the text features are obtained by encoding a title of the target video; Based on the multimodal mixed features, a sequential prompt of the target video is generated, wherein the sequential prompt is used to sequentially prompt key focus features of the target video; After relevant videos are retrieved from multiple retrieval systems, a prompt sentence is generated based on tags of the relevant videos, wherein the relevant videos are videos related to the target video; Based on the multimodal mixed features, the sequential prompts and the prompt sentences, a label sequence of the target video is generated and output.

[0079] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0080] On the other hand, the present invention also provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the label sequence generation method based on sequential prompts and retrieval enhancement generation provided by the above methods, the method comprising: inputting a target video into a label sequence generation model, the label sequence generation model after acquiring the frame features and text features of the target video, feature fusion of the frame features and the text features to obtain a multimodal hybrid feature, the frame features are obtained by encoding a sample frame in the target video, the text features are obtained by encoding a title of the target video; Based on the multimodal mixed features, a sequential prompt of the target video is generated, wherein the sequential prompt is used to sequentially prompt key focus features of the target video; After relevant videos are retrieved from multiple retrieval systems, a prompt sentence is generated based on tags of the relevant videos, wherein the relevant videos are videos related to the target video; Based on the multimodal mixed features, the sequential prompts and the prompt sentences, a label sequence of the target video is generated and output.

[0081] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to execute the label sequence generation method based on sequential prompts and retrieval enhancement generation provided by the above methods, the method comprising: inputting a target video into a label sequence generation model, the label sequence generation model, after acquiring the frame features and text features of the target video, performs feature fusion on the frame features and the text features to obtain a multimodal hybrid feature, the frame features are obtained by encoding a sampled frame in the target video, and the text features are obtained by encoding a title of the target video; Based on the multimodal mixed features, a sequential prompt of the target video is generated, wherein the sequential prompt is used to sequentially prompt key focus features of the target video; After relevant videos are retrieved from multiple retrieval systems, a prompt sentence is generated based on tags of the relevant videos, wherein the relevant videos are videos related to the target video; Based on the multimodal mixed features, the sequential prompts and the prompt sentences, a label sequence of the target video is generated and output.

[0082] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0083] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A tag sequence generation method based on sequential prompting and retrieval enhancement generation, characterized in that: include: The target video is input into the label sequence generation model. After acquiring the frame features and text features of the target video, the label sequence generation model performs feature fusion on the frame features and the text features to obtain multimodal mixed features, wherein the frame features are obtained by encoding the sampled frames in the target video, and the text features are obtained by encoding the title of the target video; Based on the multimodal mixed features, a sequential prompt of the target video is generated, wherein the sequential prompt is used to sequentially prompt key focus features of the target video; After relevant videos are retrieved from multiple retrieval systems, a prompt sentence is generated based on tags of the relevant videos, wherein the relevant videos are videos related to the target video; Based on the multimodal mixed features, the sequential prompts and the prompt sentences, a label sequence of the target video is generated and output.

2. The tag sequence generation method based on sequential prompting and retrieval enhancement generation according to claim 1, characterized in that: After acquiring the frame features and text features of the target video, the frame features and the text features are subjected to feature fusion to obtain multimodal mixed features, including: Encoding the sampled frames in the target video using a frozen visual encoder to obtain frame features output by the visual encoder; Encoding the title of the target video to obtain text features output by the visual encoder; A multimodal hybrid encoder is used to perform feature fusion on the frame features and the text features to obtain multimodal hybrid features. The multimodal hybrid encoder includes multiple layers of transformer encoders, and each layer of the transformer encoder includes a multi-head self-attention module, a cross attention head and a feedforward neural network.

3. The tag sequence generation method based on sequential prompting and retrieval enhancement generation according to claim 1 is characterized in that: The step of generating sequential prompts of the target video based on the multimodal mixed features includes: Inputting a set of randomly initialized learnable queries and the multimodal mixed features into a prompt encoder to obtain an intermediate vector output by the prompt encoder, wherein the prompt encoder comprises a multi-layer transformer encoder, and the prompt encoder uses the learnable query as a query and the multimodal mixed features as a key and a value; The intermediate vectors are mapped using a linear projection function to obtain sequential prompts of the target video.

4. The tag sequence generation method based on sequential prompting and retrieval enhancement generation according to claim 1, characterized in that: After the relevant videos are retrieved from the multiple retrieval systems, a prompt statement is generated based on the tags of the relevant videos, including: For any retrieval system, k relevant videos are retrieved from the retrieval system, and a label and a relevance score of each of the k relevant videos are obtained, where k is a positive integer; After k relevant videos are retrieved from multiple retrieval systems respectively, for each tag, a cumulative relevance score of the tag is calculated, where the cumulative relevance score is the sum of the relevance scores of the relevant videos having the tag; The p tags with the highest cumulative correlation scores are concatenated to obtain a prompt statement, where p is a positive integer.

5. The tag sequence generation method based on sequential prompting and retrieval enhancement generation according to any one of claims 1 to 4, characterized in that: The label sequence generation model is trained based on the following steps: The sample video is input into the label sequence generation model to be trained. After the label sequence generation model to be trained generates the multimodal mixed features and sequential prompts of the sample video, all the real labels and all the sequential prompts of the sample video are binary matched to calculate the first loss value; Based on the multimodal mixed features, the sequential prompts and the generated prompt sentences, a label sequence of the target video is generated, and a second loss value is calculated; determining a total loss value based on the first loss value and the second loss value; Based on the total loss value, adjusting the model parameters of the label sequence generation model to be trained until the total loss value is less than a preset threshold; The model parameters whose total loss value is less than the preset threshold are used as model parameters of the label sequence generation model to be trained to obtain the label sequence generation model.

6. The tag sequence generation method based on sequential prompting and retrieval enhancement generation according to claim 5, characterized in that: The first loss value is determined based on the following steps: Taking the true label as an anchor point and the sequential prompt as a positive and negative sample, for any true label, determining a true label loss value based on a ratio of a first similarity to a second similarity, wherein the first similarity is the similarity between the true label and the positive sample, and the second similarity is the sum of the similarities between the true label and all candidate samples; Taking the sequential prompt as an anchor point and the true label as a positive and negative sample, for any sequential prompt, determining a sequential prompt loss value based on a ratio of a third similarity to a fourth similarity, wherein the third similarity is the similarity between the sequential prompt and the positive sample, and the fourth similarity is the sum of the similarities between the sequential prompt and all candidate samples; Based on the true label loss value and the sequential prompt loss value, a first loss value is determined.

7. A label sequence generation device based on sequential prompting and retrieval enhancement generation, characterized in that: include: A multimodal hybrid coding module is used to: input the target video into a label sequence generation model, and after acquiring the frame features and text features of the target video, the label sequence generation model performs feature fusion on the frame features and the text features to obtain multimodal hybrid features, wherein the frame features are obtained by encoding the sampled frames in the target video, and the text features are obtained by encoding the title of the target video; A sequential prompt encoding module, used to: generate sequential prompts of the target video based on the multimodal mixed features, wherein the sequential prompts are used to sequentially prompt key features of the target video; A prompt statement generating module is used to: after retrieving relevant videos from multiple retrieval systems, generate prompt statements based on tags of the relevant videos, wherein the relevant videos are videos related to the target video; The label sequence generation module is used to generate and output the label sequence of the target video based on the multimodal mixed features, the sequential prompts and the prompt statements.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the label sequence generation method based on sequential prompts and retrieval enhancement generation as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the label sequence generation method based on sequential prompts and retrieval enhancement generation is implemented as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the label sequence generation method based on sequential prompts and retrieval enhancement generation is implemented as described in any one of claims 1 to 6.

Images