A video tag recognition and model training method, device, equipment and medium

Abstract

The present application relates to the field of video processing technology, and in particular to a video tag recognition and model training method, device, equipment and medium for efficiently and accurately identifying the tags of medium and long videos. The method includes: encoding each video frame separately to obtain corresponding global features and local features, and based on feature similarity, compressing the obtained global feature set and local feature set respectively to obtain a global feature sequence and a local feature sequence of a preset storage length; after splicing the global query features and local query features obtained by pre-training, the corresponding self-attention features are extracted using a self-attention mechanism; using a cross-attention mechanism, the first cross-attention feature between the self-attention feature and each global feature in the global feature sequence, and the second cross-attention feature between each local feature in the local feature sequence are extracted; based on each first cross-attention feature and each second cross-attention feature, the video tag is identified.

Description

Background Art

[0002] Video tag recognition is an important part of video content features. By automatically generating tags for massive videos, machines can provide video content features of different granularities for the downstream content distribution link, improve the efficiency of content distribution, and significantly reduce the cost of manual content review.

[0003] Taking short videos as an example, the length of a video is generally no more than 30 seconds, and the number of video frames is relatively small. Therefore, after sparse frame extraction, the visual tokens of each image can be extracted, and then spliced ​​together and input into the Large Language Model (LLM) to identify the label of the short video.

[0004] However, for medium videos, long videos or video streams (such as live broadcast scenarios), the video length is much longer than 30 seconds and can reach tens of minutes or even hours. The total number of video frames is much higher than that of short videos. If sparse frame extraction (such as extracting N frames at equal intervals) is still used, the extracted visual tokens cannot cover most of the video content, so it is difficult to ensure the recognition accuracy and recall rate of video tags.

[0005] In summary, how to efficiently and accurately identify video tags for medium and long videos is an urgent problem to be solved. Summary of the invention

[0006] The embodiments of the present application provide a video tag recognition and model training method, apparatus, device and medium for efficiently and accurately identifying video tags of medium videos and long videos.

[0007] A video tag recognition method provided by an embodiment of the present application includes:

[0008] Based on the video tag recognition model, each video frame in the target video is encoded respectively to obtain the corresponding global features and local features, and based on the feature similarity, the obtained global feature set and local feature set are compressed respectively to obtain the global feature sequence and local feature sequence of the preset storage length;

[0009] After concatenating the global query features and local query features obtained through pre-training, the corresponding self-attention features are extracted using a self-attention mechanism; the global query features and the local query features are obtained by training the corresponding learnable query features based on sample videos; the self-attention features integrate the key information in the global query features and the local query features;

[0010] Adopting a cross attention mechanism, respectively extracting a first cross attention feature between the self-attention feature and each global feature in the global feature sequence, and respectively extracting a second cross attention feature between the self-attention feature and each local feature in the local feature sequence;

[0011] Based on the obtained first cross-attention features and second cross-attention features, a video tag of the target video is identified.

[0012] The present invention provides a method for training a video tag recognition model, including:

[0013] Based on the video tag recognition model to be trained, each video frame in the sample video is encoded respectively to obtain corresponding global features and local features, and based on feature similarity, the obtained global feature set and local feature set are compressed respectively to obtain global feature sequences and local feature sequences of preset storage lengths;

[0014] After concatenating the first learnable query feature to be trained and the second learnable query feature, a corresponding self-attention feature is extracted using a self-attention mechanism; the self-attention feature integrates key information in the first learnable query feature and the second learnable query feature;

[0015] Adopting a cross attention mechanism, respectively extracting a first cross attention feature between the self-attention feature and each global feature in the global feature sequence, and a second cross attention feature between the self-attention feature and each local feature in the local feature sequence;

[0016] According to the difference between the video label and video text description information of the sample video predicted based on the obtained first cross-attention features and the second cross-attention features and the corresponding sample labels, the parameters of the video label recognition model are adjusted, and the trained video label recognition model is used to extract the video label of the target video.

[0017] An embodiment of the present application provides a video tag recognition device, comprising:

[0018] The coding and compression module is used to encode each video frame in the target video based on the video tag recognition model to obtain corresponding global features and local features, and compress the obtained global feature set and local feature set based on feature similarity to obtain global feature sequences and local feature sequences of preset storage lengths;

[0019] A fusion module is used to extract corresponding self-attention features by using a self-attention mechanism after splicing the global query features and local query features obtained by pre-training; the global query features and the local query features are obtained by training the corresponding learnable query features based on sample videos; the self-attention features fuse the key information in the global query features and the local query features;

[0020] A cross compression module, used for adopting a cross attention mechanism to respectively extract a first cross attention feature between the self-attention feature and each global feature in the global feature sequence, and respectively extract a second cross attention feature between the self-attention feature and each local feature in the local feature sequence;

[0021] The tag identification module is used to identify the video tag of the target video based on the obtained first cross-attention features and second cross-attention features.

[0022] Optionally, the video tag recognition model includes a query converter; and the cross compression module is specifically used for:

[0023] Splitting the self-attention feature into a first self-attention feature corresponding to the global feature and a second self-attention feature corresponding to the local feature;

[0024] Extracting the first self-attention feature and the first cross-attention feature between each global feature in the global feature sequence respectively based on the cross-attention mechanism through the first cross-attention network in the query converter;

[0025] Through the second cross-attention network in the query converter, based on the cross-attention mechanism, the second self-attention feature and the second cross-attention feature between each local feature in the local feature sequence are extracted respectively.

[0026] Optionally, the query converter includes multiple layers of stacked decoders; for a first decoder, the input features include the global feature sequence and the local feature sequence; for other decoders, the input features include each first cross-attention feature and each second cross-attention feature output by a previous decoder;

[0027] The tag identification module is specifically used for:

[0028] Based on the first cross-attention features and the second cross-attention features output by the last decoder, a video tag of the target video is identified.

[0029] Optionally, the video tag recognition model includes a large language model; and the tag recognition module is specifically used for:

[0030] After concatenating the first cross-attention features and the second cross-attention features, the concatenation is converted into a visual feature aligned with the text feature dimension of the target video;

[0031] The text features and text instructions of the target video are used as question-answer prompt information, combined with the visual features, and input into the large language model to generate a video tag for the target video.

[0032] Optionally, each local feature corresponds to an image block in a video frame; the encoding and compression module is specifically used for:

[0033] Each time a video frame is encoded, the global features of the video frame are stored in a global memory unit, and the local features of the video frame are stored in a local memory unit;

[0034] When the number of global features stored in the global memory unit reaches the preset storage length, based on feature similarity, feature fusion is performed on the global features of at least two video frames in the global memory unit, and the global features to be stored are stored in the global memory unit;

[0035] And, for each image block, when the number of corresponding local features stored in the local memory unit reaches the preset storage length, based on feature similarity, feature fusion is performed on the corresponding local features of at least two video frames in the local memory unit, and the local features to be stored are stored in the local memory unit.

[0036] Optionally, the device further comprises:

[0037] A reading module is used to determine that a specified recognition time has been reached before the cross-compression module adopts a cross-attention mechanism to respectively extract the first cross-attention feature between the self-attention feature and each global feature in the global feature sequence, and respectively extract the second cross-attention feature between the self-attention feature and each local feature in the local feature sequence, then read the global feature sequence stored in the global memory unit, and read the local feature sequence stored in the local memory unit.

[0038] Optionally, the encoding and compression module is further used to perform feature fusion on target features of at least two video frames in any of the following ways:

[0039] Based on feature similarity, the target features of the most similar adjacent video frames in the memory unit are averaged and pooled;

[0040] Based on feature similarity, the target features to be stored are clustered with the target features stored in the memory unit, and the clustered target features of each video frame are average pooled;

[0041] The target feature is a global feature or a local feature, and correspondingly, the memory unit is a global memory unit or a local memory unit.

[0042] Optionally, if there are multiple groups of most similar adjacent video frames, the encoding and compression module is specifically used to:

[0043] Average pooling of target features of adjacent video frames that have the longest storage time and are most similar in the memory unit; or

[0044] The target features of each group of adjacent video frames that are most similar in the memory unit are average-pooled respectively.

[0045] The present application provides a training device for a video tag recognition model, including:

[0046] The first feature extraction module is used to encode each video frame in the sample video based on the video tag recognition model to be trained, obtain corresponding global features and local features, and compress the obtained global feature set and local feature set based on feature similarity to obtain global feature sequences and local feature sequences of preset storage lengths;

[0047] A second feature extraction module is used to extract a corresponding self-attention feature by using a self-attention mechanism after concatenating the first learnable query feature to be trained with the second learnable query feature; the self-attention feature integrates key information in the first learnable query feature and the second learnable query feature;

[0048] A third feature extraction module is used to adopt a cross attention mechanism to respectively extract a first cross attention feature between the self-attention feature and each global feature in the global feature sequence, and a second cross attention feature between the self-attention feature and each local feature in the local feature sequence;

[0049] The parameter adjustment module is used to adjust the parameters of the video label recognition model according to the difference between the video label and video text description information of the sample video predicted based on the obtained first cross-attention features and the second cross-attention features and the corresponding sample labels. The trained video label recognition model is used to extract the video label of the target video.

[0050] Optionally, the video tag recognition model includes a large language model, and a query converter implemented based on a self-attention mechanism and a cross-attention mechanism, wherein the query converter is used to extract the first cross-attention feature and the second cross-attention feature; the device also includes:

[0051] A prediction module is used to, in a first training phase, concatenate the first cross-attention features and the second cross-attention features to convert them into visual features aligned with the text feature dimension of the sample video; input the first text instruction as question-answer prompt information in combination with the visual feature into the large language model to generate video text description information of the sample video;

[0052] In the second training stage, the first cross-attention feature and the second cross-attention feature are concatenated and converted into visual features aligned with the text feature dimension of the sample video; the text features and the second text instructions are used as question-answer prompt information and combined with the visual features to input into the large language model to generate a video label for the sample video.

[0053] Optionally, the parameter adjustment module is specifically used for:

[0054] In the first training stage, updating parameters of the query converter according to the difference between the predicted video text description information and the video text description information in the sample label;

[0055] In the second training stage, parameters of the query converter and the large language model are updated according to the difference between the predicted video label and the video label in the sample label.

[0056] Optionally, the parameter adjustment module is further used to update the parameters of the large language model in the following manner:

[0057] Fixing original parameters of the large language model and adding a low-rank adaptive matrix to the large language model;

[0058] The parameters of the large language model are updated by updating the parameters of the low-rank adaptive matrix.

[0059] An electronic device provided by an embodiment of the present application includes a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the above-mentioned video tag recognition methods.

[0060] An embodiment of the present application provides a computer-readable storage medium, which includes a computer program. When the computer program is executed on an electronic device, the computer program is used to enable the electronic device to perform any step of the above-mentioned video tag identification method.

[0061] An embodiment of the present application provides a computer program product, which includes a computer program, and the computer program is stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device performs the steps of any one of the above-mentioned video tag recognition methods.

[0062] The beneficial effects of this application are as follows:

[0063] The embodiment of the present application provides a method, device, equipment and medium for video tag recognition and model training. Due to the present application, when extracting the visual features of the target video based on the video tag recognition model, it is not necessary to extract frames, but to encode each video frame in the target video. After encoding, a global feature set composed of the global features of each video frame and a local feature set composed of the local features of each video frame can be obtained. On this basis, based on feature similarity, the obtained global feature set and local feature set are compressed respectively, specifically referring to feature fusion of the global features of some video frames, and feature fusion of the local features of some video frames, to obtain a global feature sequence and a local feature sequence of a preset storage length. Through the above operation, it can be ensured that the global feature sequence and the local feature sequence at this time can cover most of the content of the target video. On this basis, the key information in the global query feature and the local query feature is fused through the self-attention mechanism, and then further compressed through the cross-attention mechanism to obtain a more compact visual feature sequence.

[0064] The above method can effectively compress the visual features of medium and long videos and reduce information loss. At the same time, it can improve the efficiency of label recognition by compressing the features, thereby realizing efficient and accurate recognition of video labels of medium and long videos.

[0065] Other features and advantages of the present application will be described in the following description, and partly become apparent from the description, or be understood by practicing the present application. The purpose and other advantages of the present application can be realized and obtained by the structures specifically pointed out in the written description, claims, and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0067] Figure 1 An optional schematic diagram of an application scenario in an embodiment of the present application;

[0068] Figure 2A schematic diagram of a process flow of a content distribution product side application in an embodiment of the present application;

[0069] Figure 3 A flowchart of a video tag recognition method provided in an embodiment of the present application;

[0070] Figure 4 A schematic diagram of the structure of a video tag recognition model in an embodiment of the present application;

[0071] Figure 5 A schematic diagram of the structure of a query converter in an embodiment of the present application;

[0072] Figure 6 A schematic diagram of a phased training process of a video tag recognition model in an embodiment of the present application;

[0073] Figure 7 A schematic diagram of updating a model parameter in an embodiment of the present application;

[0074] Figure 8 A schematic diagram of another model parameter update in an embodiment of the present application;

[0075] Fig. 9 A flowchart of a video tag recognition method provided in an embodiment of the present application;

[0076] Fig.10 A schematic diagram of image segmentation provided in an embodiment of the present application;

[0077] Fig.11 A schematic diagram of a global feature compression process in an embodiment of the present application;

[0078] Fig.12 A schematic diagram of a calculation process of an attention mechanism in an embodiment of the present application;

[0079] Fig.13 A schematic diagram of a video tag extraction process in an embodiment of the present application;

[0080] Fig.14 A schematic diagram of the interaction logic between a terminal device and a server in an embodiment of the present application;

[0081] Fig.15 A schematic diagram of the structure of a video tag recognition device in an embodiment of the present application;

[0082] Fig.16 A schematic diagram of the structure of a training device for a video tag recognition model in an embodiment of the present application;

[0083] Fig.17A schematic diagram of a hardware structure of an electronic device to which an embodiment of the present application is applied;

[0084] Fig.18 The present invention is a schematic diagram of the hardware structure of another electronic device to which the embodiments of the present application are applied. DETAILED DESCRIPTION

[0085] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the technical solution of the present application, rather than all of the embodiments. Based on the embodiments recorded in the application documents, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the technical solution of the present application.

[0086] The following is an introduction to some concepts involved in the embodiments of the present application.

[0087] LLM (Large Language Model): learns the statistical laws and semantic information of language by training a large amount of text data, so as to predict the next word or sentence. LLM has a wide range of applications in the field of Natural Language Processing (NLP), such as machine translation, speech recognition, text generation, etc. LLM can adopt different training strategies and model structures, such as pre-training and fine-tuning. Pre-training refers to large-scale training of the model under unsupervised or self-supervised conditions, so that it has a general understanding and expression ability of language data. Fine-tuning is to optimize the model for specific application scenarios or tasks through supervised learning based on the pre-trained model. LLM has achieved remarkable results in tasks such as language understanding, generation and translation, and its performance and application scope are also expanding with the continuous expansion of model scale and the increase of training data.

[0088] MLLM (multimodal large language model): Based on LLM, media data of other modalities (such as images, videos, audio, etc.) are integrated, so that the model can process information of different modalities at the same time, better understand and express semantics, thereby improving the effect and accuracy of the application. The visual tag recognition model in the embodiment of the present application belongs to a kind of MLLM, which integrates media data of video modality based on LLM.

[0089] Global query features and local query features: are obtained by training the corresponding learnable query features (Learnable query) based on sample videos, and can be in the form of vectors. Specifically, the global query feature is trained based on the first learnable query feature, corresponding to the global feature, and the local query feature is trained based on the second learnable query feature, corresponding to the layout feature. The first learnable query feature and the second learnable query feature in this article refer to query vectors that can be learned and adjusted through the training process.

[0090] Transformer: A network structure model suitable for processing sequence data. It uses a full attention structure instead of the traditional recurrent neural network (RNN) or convolutional neural network (CNN). Its core idea is to model the relationship between each element in the input sequence and other elements as an attention weight matrix, and calculate the representation of each element through the self-attention mechanism. This structure makes the Transformer very capable in capturing long-distance dependencies in the sequence, thus achieving good performance in NLP tasks. The Transformer model is widely used in the field of NLP, including tasks such as machine translation, text summarization, and question-answering systems. It introduced the self-attention mechanism for the first time and surpassed the traditional sequence-to-sequence (Seq2Seq) model in many aspects, such as the Long Short-Term Memory (LSTM) and the Gated Recurrent Unit (GRU). The success of Transformer has also spawned many variant models based on self-attention mechanism, such as Bidirectional Encoder Representations from Transformers (BERT), Generative Pre-trained Transformer (GPT), etc. These models have made significant progress in NLP tasks.

[0091] Query transformer: used for feature fusion and compression based on self-attention mechanism and cross-attention mechanism. The query transformer is a multi-layer decoder structure that uses two learnable query features to further compress and fuse global features and local features to obtain a shorter visual token sequence. Specifically, a self-attention network is used in each decoder so that the global feature query sequence and the local feature query sequence can see each other, so that the two features can interact with each other, and then two parallel cross-attention networks are used to process global features and local features respectively; in the multi-layer stacked decoder structure, between each layer, global features and local features are fused through self-attention, and further compressed through cross attention to obtain a more compact visual token sequence.

[0092] A text token in Transformer refers to a word or subword in a text, and is the basic unit used by the model when processing text. Correspondingly, a visual token is the basic unit or the smallest processable unit in a video.

[0093] Memory Bank (MB): A module used to store and compress video features. In this article, it is divided into global memory bank and local memory bank. The global memory bank is used to store and compress the global features of the video frame sequence, and the local memory bank is used to store and compress the local features of the video frame sequence. The compression method is not specifically limited in this article.

[0094] The following is a brief introduction to the design concept of the embodiment of the present application:

[0095] With the rapid advancement of information technology, especially the booming development of artificial intelligence technology, the demand for video data analysis is growing. Video tag recognition is an important part of video content features. By automatically generating tags for massive videos, machines can provide video content features of different granularities for downstream content distribution links (such as recommendation systems, content operations, etc.), improve the efficiency of content distribution, and significantly reduce the cost of manual content review.

[0096] In related technologies, MLLM is often used to solve the task of understanding text + image / video content, in which visual features are a key modal feature. For images, it is only necessary to convert the image patch features of the image into tokens for processing (the number of tokens is controllable); short videos are generally no longer than 30 seconds, and this framework can basically be used. A common method is to extract visual tokens from each image after sparse frame extraction, and then splice them together and input them into LLM.

[0097] However, for medium videos, long videos, or video streams (such as live broadcast scenarios), the video length is much longer than 30 seconds and can reach tens of minutes or even hours. Sparse frame extraction is still used, so that the visual token cannot cover most of the video content. The model cannot see the complete video content, making it difficult to guarantee the accuracy and recall rate of content labels.

[0098] In addition, if the above framework is used, in order to allow the model to see sufficiently long video content, the value of N has to be set very large, so that the number of visual tokens reaches thousands or even more, which is not feasible for offline model training and online deployment.

[0099] In view of this, the embodiment of the present application proposes a method, device, equipment and medium for video tag recognition and model training. Due to the present application, when extracting the visual features of the target video based on the video tag recognition model, it is not necessary to extract frames, but each video frame in the target video is encoded. After encoding, a global feature set composed of the global features of each video frame and a local feature set composed of the local features of each video frame can be obtained. On this basis, based on feature similarity, the obtained global feature set and local feature set are compressed respectively, specifically referring to feature fusion of the global features of some video frames, and feature fusion of the local features of some video frames, to obtain a global feature sequence and a local feature sequence of a preset storage length. Through the above operation, it can be ensured that the global feature sequence and the local feature sequence at this time can cover most of the content of the target video. On this basis, the key information in the global query feature and the local query feature is fused through the self-attention mechanism, and then further compressed through the cross-attention mechanism to obtain a more compact visual feature sequence.

[0100] The above method can effectively compress the visual features of medium and long videos and reduce information loss. At the same time, it can improve the efficiency of label recognition by compressing the features, thereby realizing efficient and accurate recognition of video labels of medium and long videos.

[0101] The preferred embodiments of the present application are described below in conjunction with the drawings in the specification. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and are not used to limit the present application. In addition, the embodiments and features in the embodiments of the present application may be combined with each other if there is no conflict.

[0102] like Figure 1 As shown, it is a schematic diagram of an application scenario of an embodiment of the present application. The application scenario diagram includes two terminal devices 110 and a server 120.

[0103] In the embodiment of the present application, the terminal device 110 includes but is not limited to mobile phones, tablet computers, laptop computers, desktop computers, e-book readers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals and other devices; a client related to video tag recognition can be installed on the terminal device, which can be software (such as a browser, content distribution software, video software, etc.), or a web page, a small program, etc. The server 120 is a background server corresponding to the software or web page, small program, etc., or a server specifically used for video tag recognition, which is not specifically limited in the present application. The server 120 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0104] It should be noted that the video tag identification method in each embodiment of the present application can be executed by an electronic device, which may be a terminal device 110 or a server 120, that is, the method can be executed by the terminal device 110 or the server 120 alone, or can be executed jointly by the terminal device 110 and the server 120.

[0105] For example, when the terminal device 110 and the server 120 jointly execute, a client related to video tag recognition can be installed on the terminal device 110, and the object can determine the target video based on the client, and send the target video to the server 120 through the terminal device 110. The server 120 has pre-deployed a trained video tag recognition model. Furthermore, the server 120 uses the video tag recognition method in the embodiment of the present application to identify the video tag of the target video based on the video tag recognition model, and finally returns the identified video tag to the terminal device 110, which is presented to the object by the terminal device 110 through the client. Among them, the specific recognition process on the server 120 side is as follows:

[0106] Based on the video label recognition model, each video frame in the target video is encoded respectively to obtain the corresponding global features and local features, and based on the feature similarity, the obtained global feature set and local feature set are compressed respectively to obtain the global feature sequence and local feature sequence of preset storage length; after splicing the global query features and local query features obtained by pre-training, the self-attention mechanism is used to extract the corresponding self-attention features; the global query features and local query features are obtained by training the corresponding learnable query features based on sample videos; the self-attention features integrate the key information in the global query features and the local query features; the cross-attention mechanism is used to extract the first cross-attention features between the self-attention features and each global feature in the global feature sequence, and the second cross-attention features between the self-attention features and each local feature in the local feature sequence; based on the obtained first cross-attention features and second cross-attention features, the video label of the target video is identified.

[0107] The above implementations are just simple examples, and other implementations are also applicable to the embodiments of the present application, which will not be described in detail here.

[0108] In an optional implementation, the terminal device 110 and the server 120 may communicate with each other via a communication network.

[0109] In an optional implementation, the communication network is a wired network or a wireless network.

[0110] It should be noted that Figure 1 The figure is only an example. In fact, the number of terminal devices and servers is not limited and is not specifically limited in the embodiments of the present application.

[0111] In an embodiment of the present application, when there are multiple servers, the multiple servers can form a blockchain, and the servers are nodes on the blockchain; as in the video tag recognition method disclosed in the embodiment of the present application, the video-related data involved can be stored on the blockchain, for example, the video features of the target video, the relevant features extracted from the target video, the video tags, the model parameters of the video tag recognition model, etc.

[0112] In addition, the embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving and other scenarios.

[0113] Specifically, video tag recognition technology uses deep learning and computer vision algorithms to analyze video content, extract key features and annotate them, which allows unstructured video data to be structured and applied to multiple fields. For example, video tag recognition can be used in scenarios such as content classification, intelligent recommendation, video search and summary generation to improve the organization and utilization efficiency of video data. The following are some specific application scenario examples:

[0114] (I) Content classification: Video platforms can use the video tag recognition method in this application to automatically add metadata tags to uploaded videos to help users quickly find content they are interested in. For example, a video containing a football match may be labeled "football", "sports event", "goal", etc., so that the video can be classified into the corresponding category.

[0115] (II) Intelligent recommendation system: Streaming service providers can use the video tag recognition method in this application to understand video content and recommend similar or related videos to subjects based on their viewing history and preferences. For example, if a subject often watches science fiction movies, the system will recognize this tendency and recommend more movies with the "science fiction" tag to enhance the viewing experience, promote content consumption, enhance subject stickiness, and achieve personalized content distribution.

[0116] See also Figure 2 As shown, it is a flow chart of a content distribution product side application in an embodiment of the present application. The video enters the content processing link from the content production link, obtains the corresponding content features through human-machine collaboration, and enters the downstream content distribution link. Specifically, in the content processing process for the video, the video can be labeled based on the collaborative method of machine labeling and manual labeling, that is, the corresponding video tag is added to the video. The video tag recognition method in this application belongs to the machine labeling link.

[0117] (III) Video search and summary generation: In large-scale video databases, objects can find video clips containing specific elements through keyword search. In addition, tag recognition can also be used to automatically generate video summaries to highlight key scenes or events in the video, such as news highlights and movie highlights.

[0118] In addition to the above scenarios, video tag recognition can also be applied to:

[0119] (IV) Advertisement targeting: Media companies can use video tags to accurately place advertisements, ensure that the advertisements are relevant to the video content, and improve the advertising effect. For example, sports brand advertisements can be played in sports event videos.

[0120] (V) Copyright protection and content review: The video tag recognition method in this application can help content platforms detect and prevent copyright infringement, such as identifying unauthorized use of copyrighted music or images. At the same time, it is also used to review video content and automatically filter inappropriate or illegal content. These applications not only improve the management and distribution efficiency of video data, but also enhance the object experience and platform security.

[0121] It should be noted that the application scenarios listed above are just simple examples. In addition, other application scenarios are also applicable to the embodiments of the present application and will not be described one by one here.

[0122] In addition, it should be emphasized that in the specific implementation of this application, the video-related data involved, such as the target video that needs to be labeled, the viewing history and preferences of the subject listed above, etc. When the above embodiments of this application are applied to specific products or technologies, the subject's permission or consent is required, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0123] The following describes the video tag recognition method provided by the exemplary embodiment of the present application in combination with the application scenarios described above and with reference to the accompanying drawings. It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the implementation of the present application is not limited in this respect.

[0124] The solution provided in the embodiment of the present application mainly involves the application of artificial intelligence in video tag recognition. When identifying video tags, it is necessary to adopt an artificial neural network model based on deep learning, that is, the embodiment of the present application combines machine learning and video detection architecture-related methods to obtain a video tag recognition model. The query converter, LLM and other machine learning-based methods in the video tag recognition model are obtained through domain adaptive training. Then, in actual scenarios, the video tag recognition model can be used to achieve the purpose of tagging the target video.

[0125] Specifically, the video label recognition process in the embodiment of the present application can be divided into two parts, including a training part and an application part. Among them, the training part involves the technical field of machine learning. In the training part, the embodiment of the present application adopts a phased training method. First, in the first training stage, the video text description information of the sample video is used as a sample label to train the query converter in the video label recognition model. In the second training stage, the video label of the sample video is used as a sample label to train the query converter and LLM in the video label recognition model together; the application part is used to use the trained video label recognition model to identify the label of the target video and output the identified label list (including one or more video labels). In addition, it should be noted that the artificial neural network model in the embodiment of the present application can be either online training or offline training, which is not specifically limited here. In this article, offline training is used as an example for illustration.

[0126] Below, the training process of the video tag recognition model of the embodiment of the present application is first described:

[0127] Before training the video tag recognition model, it is necessary to first construct a video-related training sample set. The training sample set in the embodiment of the present application includes multiple training samples, each of which includes: a sample video and its corresponding sample label.

[0128] It should be noted that the sample video in the model training phase and the target video in the model application phase in the embodiment of the present application can be any video, specifically various types of videos on the video platform, including live videos and non-live videos. According to the length of time, the non-live video can be any short video, medium video and long video of fixed length, or any video stream of non-fixed length; according to the type, the non-live video can be film and television video, information video, conference video, teaching video and life sharing video, etc.

[0129] Optionally, the sample tag in the embodiment of the present application includes two parts: a video tag of the sample video and video text description information.

[0130] Among them, video tags are keywords or phrases used to describe the content, features or context of a video, which can help the subject quickly understand the theme and type of the video. A video may have one or more tags. Specifically, if a video has multiple tags, these multiple tags can cover different aspects of it.

[0131] For example, a tutorial video about baking cakes may have tags including "baking", "cake", "recipe", "teaching", and "food". These tags not only describe the specific content of the video (baking cakes), but also indicate its function (teaching) and category (food); for another example, a documentary about outdoor adventures may have tags including "outdoor adventures", "natural scenery", "wildlife", "hiking", "photography skills", "survival skills", "travel guide", and "human geography"; and so on.

[0132] The video text description information may refer to any text information related to the video, which may provide additional context or directly describe the media content. In the embodiment of the present application, the text modality of the video text description information includes but is not limited to some or all of the following:

[0133] Video titles, description text, key text extracted by Optical Character Recognition (OCR), subtitles or transcriptions generated by Automatic Speech Recognition (ASR), comments and forum discussions, metadata information.

[0134] The description text refers to the detailed description below the video or on the video page, which usually contains information such as the theme, purpose, key points, characters, and locations of the video. The description text can be provided by the video producer, or it can be compiled from post-editing or audience comments. Metadata information includes creator information, upload date, etc. Although these data do not directly describe the video content, they are crucial for the organization and retrieval of videos.

[0135] In addition, this application uses OCR technology to identify and extract text information from images in the video, such as text on posters, text displayed on the screen, book covers, menus, road signs, etc. These texts can provide additional information about the video scene to help understand the video content. ASR technology can be used to convert the speech in the video into text and generate subtitles or speech transcriptions. These texts include not only the content of the dialogue, but also descriptions of narration, commentary, and background sounds, which provide convenience for the hearing impaired and also help video search engines understand and index video content.

[0136] In an embodiment of the present application, the above-mentioned text modal information can be summarized as [video_txt], which represents the video text description information of a video, which can greatly enrich the understanding and analysis of the video.

[0137] In an optional implementation, the sample labels can be obtained by manual labeling. Considering that manual labeling may be affected by the subjective consciousness of the labeler, in order to reduce the labeling error, the manual labeling results of multiple people can be obtained, and then the final sample labels can be obtained by combining the labeling results of multiple people.

[0138] In addition, it can also be labeled based on the historical annotation information of the sample video (such as historical tags, titles, etc.), or video tags and video text description information detected by other trained models (such as LLM).

[0139] In the embodiment of the present application, after obtaining the training sample set, the video label model to be trained can be trained iteratively based on the training sample set to obtain a trained video label recognition model. The training process is briefly described below:

[0140] See also Figure 3 As shown, it is an implementation flow chart of a video tag identification method provided by an embodiment of the present application. Taking the server as the execution subject as an example, the specific implementation process of the method is as follows S31~S34:

[0141] S31: Based on the video label recognition model to be trained, each video frame in the sample video is encoded respectively to obtain corresponding global features and local features, and based on feature similarity, the obtained global feature set and local feature set are compressed respectively to obtain global feature sequences and local feature sequences of preset storage lengths.

[0142] Among them, a video frame can be an image of a corresponding frame in a video, so the video frame can also be called a video frame image, or image for short. For example, assuming that a video has a frame rate of 30 and a duration of 1 second, the video has a total of 30 frames of images, and the 30 frames of images are all the video frames of the video.

[0143] The video tag recognition model in the embodiment of the present application belongs to a multimodal model. Taking MLLM as an example, MLLM needs a bridge connecting non-text tokens and LLM, which converts inputs from different modalities into tokens that can be understood by LLM. This application implements this function based on the query converter. Optionally, the visual tag recognition model in the embodiment of the present application is a retrieval-enhanced video tag recognition model, including a visual encoder, a query converter, and a large language model. Among them, the visual encoder is used to extract global features and local features of video frame images, and the query converter (i.e., Querying transformer) is used to perform feature fusion and compression based on self-attention mechanism and cross-attention mechanism. In addition, the query converter also includes a fully connected layer to achieve dimensional alignment of visual features and text features. Finally, video tags are generated by asking questions to the large language model.

[0144] In this process, each local feature corresponds to an image block in the video frame; the extraction of global features and local features can be realized based on the visual encoder in the video tag recognition model, such as Figure 4 The visual encoder 401 in Figure 4 This is a structural diagram of a video tag recognition model in an embodiment of the present application, wherein the function of the visual encoder is to convert an input video frame image into a visual feature (column vector). Specifically, a visual converter (Vision Transformer, ViT) can be selected, or a convolutional neural network (Convolutional Neural Network, CNN) can be selected, or other encoding models, such as Residual Networks (Residual Networks, ResNet), Inception Networks (InceptionNetworks), Swin Transformers (Swin Transformers), mobile networks (MobileNet), efficient networks (EfficientNet), etc.

[0145] Specifically, no matter which encoding model is selected, the extracted features can be divided into two categories: global features and local features.

[0146] Among them, global features refer to the overall representation of video frame images, which usually include information such as the image's macroscopic structure, color distribution, and overall style. Global features are used to help the model understand the overall semantics of the image, such as identifying whether an image is a landscape, a person, or abstract art.

[0147] Local features refer to the features of specific areas in the video frame image, such as the shape, texture and position information of objects such as faces, vehicles, and trees. Local features are used to help the model identify and locate specific objects in the image. Specifically, in the embodiment of the present application, the video frame image can be pre-divided into N image blocks (patches), where a patch refers to a smaller, fixed-size area in the image, and local features are extracted at these patch levels. Each patch corresponds to a local feature, reflecting the details of a specific area of ​​the image.

[0148] Specifically, for ViT, the input image is first divided into multiple image patches of fixed size, and each patch is flattened and linearly projected into an embedding vector, namely, patch tokens. In addition, ViT introduces a special classification token (CLS token), which is at the beginning of the sequence and is input into the Transformer encoder together with the patch tokens. The CLS token does not correspond to any specific image patch, but interacts with all patch tokens through the self-attention mechanism, gradually aggregating the information of the entire image, and finally its embedding vector (emb) represents the global features of the image, which can be recorded as f glob , size is 1*d. At the same time, the embedding vector (emb) of each patch token retains the local features of the corresponding image area after passing through the Transformer encoder, which can be recorded as f loc,i , size is N*d, N is the number of patches, i=1,2,…,N.

[0149] For CNN, the average pooling operation is usually located at the end of the network. Its function is to compress each channel of the feature maps generated by the previous convolutional layer into a single value, so that the output vector after the operation condenses the information of the entire input image and is therefore regarded as a global feature. The feature map before average pooling retains the spatial structure of the image, and the value of each point corresponds to the characteristics of the local area of ​​the image, so these feature maps correspond to local features.

[0150] The above is just a brief explanation of the process of extracting global and local features of video frame images using ViT and CNN as examples. For other networks, the extraction methods are related to their network structures, and this article will not go into details one by one.

[0151] In an embodiment of the present application, the global features and local features of the video frame image are extracted simultaneously, so that the model can understand the macro context of the image and capture the detailed information, thereby achieving better performance in subsequent tasks.

[0152] The following uses a trained ViT as a visual feature encoder as an example, and its parameters are no longer updated in the subsequent training process. In addition, it should be noted that the visual encoder in this article is based on ViT as an example, and of course it can be replaced with other network structures, which is not specifically limited in this article.

[0153] In addition, in order to solve the problem in the related art that it is difficult to ensure the recognition accuracy and recall rate of video labels for medium-length videos, long videos or video streams (such as live broadcast scenes) due to sparse frame extraction, this application proposes a video stream label generation method based on a memory-enhanced multimodal large model. By introducing two memory units (i.e., MB modules) to store and compress the global features and local features of the video frame sequence respectively, such as Figure 4 The global memory unit 402 and the local memory unit 403 in FIG. Therefore, an optional implementation is as follows:

[0154] Each time a video frame is encoded, the global features of the video frame are stored in a global memory unit, and the local features of the video frame are stored in a local memory unit.

[0155] When the number of global features stored in the global memory unit reaches a preset storage length, based on feature similarity, feature fusion is performed on the global features of at least two video frames in the global memory unit, and the global features to be stored are stored in the global memory unit; and for each image block, when the number of corresponding local features stored in the local memory unit reaches a preset storage length, based on feature similarity, feature fusion is performed on the corresponding local features of at least two video frames in the local memory unit, and the local features to be stored are stored in the local memory unit.

[0156] In the embodiment of the present application, the effect of video frame sequence compression is achieved by fusing global features or local features. The feature fusion method can be to compress adjacent video frames, cluster multiple video frames, etc. For details, please refer to the relevant description of the following model application stage, such as the following compression method 1, compression method 2, compression method 3, etc., which will not be repeated here.

[0157] Then, we can identify the query transformer in the model based on the visual label, such as Figure 4 The query converter 404 shown executes the following steps S32 and S33:

[0158] S32: After concatenating the first learnable query feature to be trained and the second learnable query feature, a corresponding self-attention feature is extracted using a self-attention mechanism; the self-attention feature integrates key information in the first learnable query feature and the second learnable query feature.

[0159] The first learnable query feature is a learnable global feature query sequence for global features, which can be in the form of a vector and can be recorded as Q1. The second learnable query feature is a learnable local feature query sequence for local features, which can also be in the form of a vector and can be recorded as Q2. The first learnable query feature and the second learnable query feature can be understood as model parameters in the query converter.

[0160] It can be randomly initialized before model training. For example, Q1 and Q2 can be randomly initialized as a Learnablequery, which will be learned and optimized during the model training process. Q1 represents the focus on global features, while Q2 represents the focus on local features. The concatenation of Q1 and Q2 is used as the input of the self-attention mechanism. This process allows the global feature query sequence to capture local details while allowing the local feature query sequence to obtain the global context, and finally generates a fused feature representation that is both meticulous and broad-minded, thereby improving the model's ability to understand complex visual patterns. The model learns how to effectively detect and locate global and local features in images without any prior knowledge, thereby enhancing the model's adaptability and expressiveness.

[0161] S33: Using the cross-attention mechanism, the first cross-attention features between the self-attention feature and each global feature in the global feature sequence, and the second cross-attention features between the self-attention feature and each local feature in the local feature sequence are extracted respectively.

[0162] In the above S32, the first learnable query feature and the second learnable query feature are spliced. After splicing, the key information in the first learnable query feature and the second learnable query feature can be fused through the self-attention mechanism. On this basis, the cross-attention mechanism is adopted to extract the first cross-attention feature between the self-attention feature and each global feature in the global feature sequence, and the second cross-attention feature between the self-attention feature and each local feature in the local feature sequence.

[0163] Optionally, the self-attention feature obtained in the above S32 can be first split into two parts, recorded as the first self-attention feature and the first self-attention feature. The splitting process is equivalent to the inverse process of splicing in S32. For example, in the above S32, the first learnable query feature and the second learnable query feature are spliced ​​on the row vector, then they can be split on the row vector here, and so on.

[0164] On this basis, the cross-attention mechanism can be used to extract the first cross-attention feature between the first self-attention feature and each global feature in the global feature sequence, and to extract the second cross-attention feature between the second self-attention feature and each local feature in the local feature sequence.

[0165] Optionally, the query converter includes a multi-layer stacked decoder Decoder; for the first decoder, the input features include a global feature sequence and a local feature sequence output by a memory unit; for other decoders, the input features include each first cross-attention feature and each second cross-attention feature output by a previous decoder.

[0166] like Figure 5 As shown, it is a structural diagram of a query converter in an embodiment of the present application. Figure 5 The part above the dotted line represents the content structure of a decoder, and the part below the dotted line represents a multi-layer stacked decoder structure.

[0167] Specifically, a Decoder includes a self-attention network, such as Figure 5 The multi-head self-attention 501 in the two parallel cross-attention networks can be recorded as the first cross-attention network and the second cross-attention network, such as Figure 5 The first multi-head cross attention 502 (i.e., the first cross attention network), the second multi-head cross attention 503 (i.e., the second cross attention network), and the feed-forward neural network (Feed-Forward Network, FFN) after each cross attention network, such as Figure 5 The first feedforward neural network 504 and the second feedforward neural network 505. The cross attention network in each decoder is divided into two parallel networks, wherein the first cross attention network processes global features and the second cross attention network processes local features.

[0168] Among them, FFN can perform nonlinear transformation on the features output by the cross-attention network through its internal linear layer and nonlinear activation function, enhance the feature expression ability, and make it more suitable for the processing of subsequent tasks. This transformation helps the model learn more complex feature relationships and patterns.

[0169] It should be noted that the cross-attention feature in the embodiment of the present application may refer to the output of the cross-attention network. On this basis, the output of the cross-attention network may be nonlinearly changed through FFN. For the sake of simple description and ease of understanding, the feature of nonlinearly changing the output of the cross-attention network through FFN may still be referred to as the cross-attention feature.

[0170] It should be emphasized here that each Decoder in the embodiment of the present application corresponds to a set of Q1 and Q2, which serve as the input of the self-attention network in the decoder, and the input of the cross-attention network in the Decoder is determined based on the above method. For example, the input of 502 in Decoder1 is the global features output by the global memory unit, and the input of 503 in Decoder1 is the local features output by the local memory unit; the input of 502 in Decoder2 is the first cross-attention features output by 504 in Decoder1, and the input of 503 in Decoder2 is the second cross-attention features output by 505 in Decoder1; the input of 502 in Decoder3 is the first cross-attention features output by 504 in Decoder2, and the input of 503 in Decoder3 is the second cross-attention features output by 505 in Decoder2; the same applies to subsequent Decoders, and so on.

[0171] S34: Adjust parameters of the video label recognition model according to the differences between the video label and video text description information of the sample video predicted based on the obtained first cross-attention features and the second cross-attention features, and the corresponding sample labels.

[0172] Among them, the trained video label recognition model is used to extract the video label of the target video.

[0173] When the query converter includes multiple decoders, S34 specifically refers to the first cross-attention features and the second cross-attention features output by the last decoder, and then the video label of the sample video and the video text description information can be predicted based on the first cross-attention features and the second cross-attention features output by the next decoder in combination with LLM.

[0174] The following is a detailed description of the prediction process:

[0175] For MLLM, there are two types of inputs received by LLM, including visual tokens and text tokens. Among them, the visual token is obtained based on the visual encoder processing. In order to align the input of LLM, the visual token is followed by a query converter. In addition to fusing and compressing global features and local features, it is also used to map the visual token to the input dimension of LLM and convert it into a token adapted to LLM, that is, the visual feature in this article, denoted as [video_emb]. For text tokens, including video titles, description texts, OCR, ASR, etc., denoted as [video_txt].

[0176] Specifically, each first cross-attention feature and each second cross-attention feature output by the query converter can be used as a visual token (a non-text token). Combined with the text token of the video, the query converter can be asked to the large language model 406 to obtain the corresponding video tag. That is, the query converter in this article, as a bridge connecting non-text tokens and LLM, is used to convert inputs from different modalities into tokens that can be understood by LLM. The compressed and fused visual tokens are spliced ​​together with the text tokens of the text prompt and input into LLM for training.

[0177] In addition, considering that the dimensions of the visual token and text token output by the query converter are not necessarily the same, the dimensions of the features of different modalities can also be aligned through the fully connected layer (FC). Among them, FC can be divided into a part of the query converter, connecting the output of the last Decoder in the query converter; of course, FC can also be regarded as an independent part between the query converter and LLM, such as Figure 4 In either form, FC is used to align the dimension of visual features with the dimension of text features.

[0178] The following example uses the query transformer to include a fully connected layer:

[0179] In an embodiment of the present application, in order to ensure that the visual tokens extracted by the query converter can more accurately represent the characteristics of the video and thus extract more accurate video labels, the training process of the video label recognition model in the present application may include two main training stages, including pre-training for the query converter in the first training stage and instruction fine-tuning training for the query converter and LLM in the second training stage.

[0180] See also Figure 6 As shown, it is a schematic diagram of the phased training process of a video tag recognition model in an embodiment of the present application. In these two training stages, the input of LLM is different, the output is different, and the adjusted parameters are also different. In the first training stage, the input of LLM is the visual features obtained based on the query converter (which can be recorded as [video_emb]), and the first text instruction, and the output is the video text description information, and only the parameters of the query converter are adjusted; in the second training stage, the input of LLM is the visual features, text features (which can be recorded as [video_txt]) and the second text instruction obtained based on the query converter, and the output is the video tag, and the parameters of the query converter and LLM are adjusted. The following is a detailed description of these two training stages:

[0181] (i) First training stage: pre-training of the query converter.

[0182] In this stage, only the Querying transformer is trained, and all the parameters of the visual encoder and LLM are fixed (i.e. frozen), without adding LoRA. The main goal is to align the visual token to the LLM.

[0183] The training data at this stage uses <sample video, video description text information> pairs, such as <short video, video title> pairs in actual business scenarios, to train the model with the goal of generating a brief description of the video content. An optional training method is as follows:

[0184] In the first training stage, each first cross-attention feature and each second cross-attention feature are concatenated and converted into a visual feature aligned with the text feature dimension of the sample video; then, the first text instruction is used as question-answer prompt information and combined with the visual features to input the large language model to generate video text description information of the sample video; then, according to the difference between the predicted video text description information and the video text description information in the sample label, the query converter parameters are updated.

[0185] The text feature of the sample video refers to the video text description information encoded in the sample video. For example, if the video text description information is the video title, the text feature is obtained by encoding the video title. If the video text description information includes the video title, extracted key text, etc., the text feature is obtained by encoding and concatenating these video text description information. Specifically, it can be generated by any text encoder, such as the word embedding (Word2Vec) network, BERT, and RoBERTa (Robustly Optimized BERT Pretraining Approach). Similar to the visual encoder, the network parameters of this part do not need to be updated during the model training process.

[0186] The first text instruction is used to prompt the large language model to output video text description information of the sample video. Therefore, the first text instruction can be one of the following sentences:

[0187] “Briefly describe the following video.”, “Provide a brief description of the given video clip.”, “Concisely explain the provided video clip.”, “Outline the visual content of the following video.”, “Give a brief and clear explanation of the following video clip.”, “Briefly elaborate on the meaning of the provided video.”, “Briefly describe the key features of the clip.”, “Briefly describe the content of the presented video.”, “Provide a clear and concise summary of the following video.”, “Write an informative summary of the following video clip.”, “Present the provided video in a concise narrative style.”, etc.

[0188] It should be noted that the above-listed sentences are just simple examples, and other sentences that can prompt the large language model to output video text description information of the sample video are also applicable to the embodiments of the present application and will not be described one by one here.

[0189] Among them, LLM can choose open source Chinese LLM, such as chinese-llama, BigScience Large Open-science Open-access Multilingual (BLOOM), etc. The input prompt word template is:

[0190] “###Human: [video_emb][Alignment instructions]###Assistant:”

[0191] Note that [video_txt] is not input in the first training stage, and only visual information, i.e., [video_emb], is provided to LLM, combined with the alignment instructions to guide the Querying transformer to better train.

[0192] Among them, "###Human:" and "###Assistant:" are two commonly used templates for training, which are used to make the machine understand that Human is followed by instructions, and seeing Assistant means starting to answer questions. Alignment instructions are guidance or commands given to LLM during the training process, telling LLM what kind of output should be generated when given input. The alignment instructions can be diverse, such as using at least one of the sentences listed in the first text instruction above.

[0193] In the first training phase, when adjusting the model parameters, only the query converter parameters are adjusted as follows:

[0194] According to the difference between the predicted video text description information and the video text description information in the sample label, a loss function is constructed, and the parameters of the query converter are updated according to the loss function.

[0195] Among them, the loss function can be cross-entropy loss (Cross-Entropy Loss), cosine similarity loss (Cosine Similarity Loss), BLEU Score (Bilingual Evaluation Understudy), etc., which are not specifically limited in this article. Combined with the output rules of the large language model, the design and optimization process of the loss function will consider the probability distribution of the model-generated sequence (i.e., the predicted video text description information) and the difference between the generated sequence and the target sequence (i.e., the video text description information in the sample label). By adjusting the model parameters, the consistency between the model-generated text description and the real label description can be gradually improved, thereby improving the accuracy of video content understanding and the quality of natural language description.

[0196] After constructing the loss function, the gradient can be calculated through the back-propagation algorithm to adjust the parameters of the query transformer to minimize the loss.

[0197] (ii) Second training phase: instruction fine-tuning training for the query converter and LLM.

[0198] In this stage, only the parameters of the visual encoder are fixed, and the Querying transformer and LLM are trained.

[0199] There are two ways to train LLM. One is to adjust the parameters of LLM directly, and the other is to fix the LLM parameters, add a low-rank adaptive matrix (LoRA), and train the LoRA parameters.

[0200] When LoRA is added, the parameters of the large language model can be updated in the following way:

[0201] The original parameters of the large language model are fixed, and a low-rank adaptive matrix is ​​added to the large language model; the parameters of the large language model are updated by updating the parameters of the low-rank adaptive matrix.

[0202] Among them, LoRA is a technology for fine-tuning on a pre-trained model. Its main goal is to reduce the resources and time required for fine-tuning by introducing additional low-rank matrices to adapt to specific tasks without modifying the original model parameters.

[0203] Specifically, two additional small matrices can be added to the linear layer of the large language model, one is the left matrix after low-rank decomposition, and the other is the right matrix. The output of the product of these two matrices is added to the output of the original linear layer to form the final output. In this way, LoRA can effectively adjust the behavior of the model to adapt to specific task requirements without changing the architecture of the large language model.

[0204] In the above implementation, LoRA greatly reduces the memory and computing resources required for fine-tuning by adjusting only a small number of newly added parameters instead of all the parameters of the entire large language model, thereby improving the efficiency of model training while maintaining the performance of the original model. Moreover, compared with the fine-tuning method of the complete model, the LoRA model occupies less storage space and saves storage resources.

[0205] The training data at this stage is constructed based on human-reviewed label data of sample videos (such as medium videos, long videos, or video streams) in actual business scenarios, denoted as <sample video, video label>. For example, <video stream> in actual business scenarios i (t) , tag list i (t) >, where i is the index of the i-th sample video in the training sample set, the superscript t represents the time t, and the label list i (t) That is, at time t, the video stream i A list of recognized video tags.

[0206] It should be noted that the same video stream can be reviewed multiple times at different times and obtain different labeling results.

[0207] In this stage, the generated label list text is used as the target for instruction fine-tuning. An optional training method is as follows:

[0208] In the second training stage, the first cross-attention features and the second cross-attention features are concatenated and converted into visual features that are aligned with the text feature dimension of the sample video; the text features and the second text instructions are used as question-answer prompt information and combined with the visual features to input into the large language model to generate video labels for the sample video; then, in the second training stage, the parameters of the query converter and the large language model are updated according to the difference between the predicted video labels and the video labels in the sample labels.

[0209] The text features of the sample video are the same as above and will not be repeated here. The second text instruction is used to prompt the large language model to output the video label of the sample video. For example, the second text instruction can be one of the following statements:

[0210] "What tags can be used to summarize the above video?", "Please output some key content tags based on the content of the above video.", "Based on the above video content, generate main content tags.", etc.

[0211] It should be noted that the above-mentioned statements are just simple examples, and other statements that can prompt the large language model to output video labels of sample videos are also applicable to the embodiments of the present application and will not be described in detail here.

[0212] Among them, LLM can choose open source Chinese LLM, such as Chinese-llama, BLOOM, etc. The input prompt word template is:

[0213] "###Human: [video_emb][video_txt][instruct] ###Assistant:"

[0214] Among them, [video_emb], [video_txt] and [instruct] are the visual token of the video, the text token of the video and the token of the instruction text respectively.

[0215] In the second training stage, [video_txt] and [video_emb] are input to provide visual information and text information to LLM, and the Querying transformer and LLM are trained in combination with the instruct command.

[0216] Among them, "###Human:" and "###Assistant:" are two commonly used templates for training, which are used to make the machine understand that Human is followed by instructions, and seeing Assistant means starting to answer questions. Similar to the alignment instruction in the first training stage, it is also a guide or command given to LLM during the training process, telling LLM what kind of output should be generated when given input. The instruct instruction can also be designed with synonyms to improve the training effect, such as using at least one of the sentences listed in the second text instruction above.

[0217] In the second training phase, when adjusting the model parameters, both the query converter and the LLM need to be adjusted as follows:

[0218] According to the difference between the predicted video label and the video label in the sample label, a loss function is constructed; when LoRA is not added, the query converter and LLM parameters are updated according to the loss function. Alternatively, when LoRA is added, the query converter and LoRA parameters are updated according to the loss function.

[0219] The type and construction method of the loss function are similar to those in the first training stage and will not be repeated here.

[0220] After constructing the loss function, the gradient can be calculated through the back-propagation algorithm to adjust the parameters of the query transformer to minimize the loss.

[0221] It should be noted that the above is an example in which the query converter includes FC. If FC is independent of the query converter, the parameters of FC are also updated in both the first training stage and the second training stage.

[0222] And this article mentions that there are two ways to update LLM parameters: without LoRA and with LoRA. Assuming that FC is independent of the query converter, the following is a brief description of the parameter update logic during model training with the attached figure:

[0223] like Figure 7 As shown, it is a schematic diagram of a model parameter update in an embodiment of the present application. Figure 7 A more specific network structure of a video tag recognition model is listed, in which a visual encoder is used to encode each video frame in the input video stream in sequence to generate global features and local features of each video frame. The global feature MB is the global memory unit in this article, which is used to store and compress global features, and the local feature MB is the local memory unit in this article, which is used to store and compress local features.

[0224] When it is necessary to identify the video tag, the global feature sequence in the global feature MB is input into the multi-head cross attention 1 in the query converter on the right, and the local feature sequence in the local feature MB is input into the multi-head cross attention 2 in the query converter on the right. The query converter contains N stacked decoders. After being processed by the self-attention mechanism and the cross-attention mechanism, the output is then dimensionally aligned by FC, and then combined with the prompt input LLM for prediction. Among them, the prompt consists of video description text information and text instructions (such as the first text instruction and the second text instruction).

[0225] exist Figure 7 In the , different patterns are used to indicate whether different modules in the model need to update parameters. Among them, the snowflake pattern indicates that the parameters are frozen and not updated, and the flame pattern indicates that the parameters are updated. Therefore, Figure 7 That is to say, during the model training process, the visual encoder is not updated, but the converter, FC and LLM are queried for parameter update, corresponding to the method without adding LoRA.

[0226] like Figure 8 As shown, it is a schematic diagram of another model parameter update in an embodiment of the present application. Figure 7 The difference is that Figure 8 The original parameters of LLM are frozen, the visual encoder and LLM are not updated, and the converter, FC and LoRA are queried to update the parameters, which corresponds to the method of adding LoRA.

[0227] Of course, the FC layer can also be replaced by a linear projection module or a multi-layer perceptron (MLP), etc., which is not specifically limited in this article.

[0228] In the above implementation, the query converter is trained in the first training phase, and the query converter parameters are adjusted based on the video text description information, which can improve the accuracy of the query converter in extracting visual features and ensure that the extracted visual features can more accurately describe the main content of the video. On this basis, the query converter and LLM are adjusted synchronously in the second training phase, and the query converter and LLM parameters are adjusted based on the video tags to ensure that the LLM can generate more accurate video tags based on the visual features extracted by the query converter, so as to improve the accuracy of the video tags extracted by the model.

[0229] In addition, it should be noted that when training the video tag recognition model, the first training stage and the second training stage can be performed alternately, that is, the parameters of the query converter are updated once, and then the parameters of the query converter and the LLM are updated once at the same time, and then the parameters of the query converter are updated once again, and then the parameters of the query converter and the LLM are updated once at the same time...

[0230] In addition, after the first training phase is completed, the second training phase may be performed, that is, the parameters of the query converter may be updated multiple times, and then the parameters of the query converter and the LLM may be updated multiple times simultaneously.

[0231] Regardless of the above training method or training stage, the iteration stop condition includes at least one of the following conditions:

[0232] (1) The number of iterations reaches the set threshold.

[0233] (2) The model loss value is less than the set loss threshold.

[0234] In an embodiment of the present application, when the number of iterations does not exceed the pre-set number threshold, and the model loss value is not less than the set loss threshold, it is considered that the current model does not meet the iteration termination condition, and the model parameters need to be adjusted to continue training. After the parameter adjustment, the next iterative training process is entered. When the number of iterations exceeds the pre-set number threshold, or the model loss value is less than the set loss threshold, it is considered that the current model has met the convergence condition and the model training ends.

[0235] It should be noted that the visual encoder and text encoder in this application can be replaced by networks of other structures; the query transformer is used to align visual features to the input of LLM, and can therefore be understood as a feature alignment module. In addition to being designed as a querying transformer, the feature alignment module can also be designed with other forms of adapter networks, q-former networks, etc., and is not limited to the querying transformer designed in this application; in addition, the memory unit can also be changed to other structures similar to memory banks, and the main process of this application can also replace the LLM framework with other generative models, etc. Simple modifications made to the units in the video tag recognition model listed in this application, or the application of the core process of this application to other scenarios, are all within the scope of protection of this application.

[0236] After the model training is completed, it can be deployed online. Whether it is a short video, a medium video, a long video or a video stream, it is input in the form of a video frame sequence. After compression by MB, an audit can be triggered at any time (that is, triggering the recognition of video tags). The compressed and fused visual tokens are calculated based on the Querying transformer, and then combined with the video's text tokens and text instructions to input LLM to generate predicted label results.

[0237] The above is mainly an introduction to the model training process. The following is an explanation of the model application process:

[0238] See also Fig. 9 As shown, it is an implementation flow chart of a video tag identification method provided by an embodiment of the present application. Taking the server as the execution subject as an example, the specific implementation process of the method is as follows S91~S94:

[0239] S91: Based on the video tag recognition model, encode each video frame in the target video to obtain corresponding global features and local features, and based on feature similarity, compress the obtained global feature set and local feature set to obtain global feature sequences and local feature sequences with preset storage lengths.

[0240] Similar to the sample videos listed above, the target video in the embodiments of the present application may be any video, including but not limited to short videos, medium videos, long videos, and video streams.

[0241] Among them, short videos, medium videos and long videos refer to videos of different lengths. For example, short videos usually refer to video content with a duration of 15 seconds to 10 minutes, medium videos cover the range of 1 minute to 30 minutes, and long videos refer to videos with a duration of more than 30 minutes, such as TV series and movies.

[0242] In this step, based on the visual encoder in the video tag recognition model, each video frame in the target video can be encoded separately to obtain corresponding global features and local features, and each local feature corresponds to an image block in the video frame.

[0243] In one embodiment, before encoding the target video, the server may divide the video frames in the target video into the same number of image blocks; or, divide the video frames in the target video in a pyramid-like manner to obtain corresponding image blocks; wherein the number of image blocks in different video frames is different.

[0244] The number of image blocks in each video frame obtained by the above segmentation method can be the same, or can be increased in geometric proportion. For example, the video frames in the target video are segmented in the same segmentation method, so that the number of image blocks in each video frame is the same. For example, each video frame is segmented in a 3×3 manner. Fig.10 As shown; in addition, the video frames in the video can also be segmented in a pyramid-like manner, such as the first video frame is segmented in a 2×2 manner to obtain 4 image blocks; the second video frame is segmented in a 3×3 manner to obtain 9 image blocks, and so on.

[0245] It should be noted that, when segmenting, each image block may or may not have overlapping content, and this article does not make a specific limitation for comparison.

[0246] In the following, we take the case where all the video frames in the target video are segmented in the same way as an example. Assuming that the number of image blocks in each video frame is the same, which is N, when the video frame is encoded by the visual encoder ViT, the global feature is recorded as f glob , size is 1*d, local features are recorded as f loc,i , size is N*d, N is the number of patches, i=1,2,…,N.

[0247] In the embodiment of the present application, the global feature set includes at least one global feature, specifically including the global features of each of the currently encoded video frames in the target video. As time goes by, more and more video frames are encoded, and the global features included in the global feature set also increase.

[0248] Similarly, the local feature set includes at least one local feature, specifically including the local features of each currently encoded video frame in the target video. If each video frame can be divided into N image blocks, then each video frame corresponds to N local features.

[0249] In the embodiment of the present application, compression can be performed after encoding all video frames in the target video, or compression can be performed while encoding. When encoding and compressing, an optional implementation method is as follows:

[0250] Each time a video frame is encoded, the global features of the video frame are stored in a global memory unit, and the local features of the video frame are stored in a local memory unit.

[0251] Specifically, the two memory units designed in this application store and compress global features and local features respectively. The input video frame sequence can be represented as , T is the total number of frames of the target video, where T can be continuously increased and is not limited to processing video frame sequences of known length. As described above, each video frame After the visual encoder extracts features, global features and local features can be obtained, which can be expressed as , enter the corresponding MB.

[0252] In addition, due to the large number of video tokens, the global memory unit and the local memory unit are both set with their own corresponding preset storage lengths. For example, a maximum length is set in the global memory unit, that is, the preset storage length of the global memory unit is M, which means that the global features of M frames of video can be accommodated at most. Correspondingly, the preset storage length of the local memory unit is , which means that the local features of M frames of video can be accommodated at most, and each frame of video corresponds to N local features.

[0253] Furthermore, when the number of global features stored in the global memory unit reaches a preset storage length, feature fusion is performed on the global features of at least two video frames in the global memory unit based on feature similarity, and the global features to be stored are stored in the global memory unit; and, for each image block, when the number of corresponding local features stored in the local memory unit reaches a preset storage length, feature fusion is performed on the corresponding local features of at least two video frames in the local memory unit based on feature similarity, and the local features to be stored are stored in the local memory unit.

[0254] For example, for a global memory unit, when the amount of data in MB reaches M, if global features of a new video frame need to be added to MB, compression is required so that the data in MB does not exceed M.

[0255] Similarly, for local memory units, when the amount of data in MB reaches If there are local features of a new video frame that need to be added to the MB, a compression is required so that the data in the MB does not exceed .

[0256] It should be noted that when compressing local features, the image block is used as a unit, and the local features of each image block are fused.

[0257] In addition, it should be noted that when calculating the similarity between features, it is usually achieved by calculating the distance between feature vectors. There are many ways to calculate the feature similarity in the embodiments of the present application, including but not limited to any of the following:

[0258] Cosine similarity, Euclidean distance, Manhattan distance, Jaccard similarity, Pearson Correlation Coefficient.

[0259] In the above implementation, the global feature unit can capture the macro information of the video, while the local feature unit focuses on the detail changes. The two are complementary. By introducing two memory units to respectively store and compress the global features and local features of the video frame sequence, the video data can be processed more efficiently, which not only ensures the comprehensiveness of the information but also reduces redundancy, thereby improving storage efficiency and computing performance.

[0260] The following will take cosine similarity as an example to illustrate.

[0261] Optionally, the global memory unit and the local memory unit are collectively referred to as memory units, and the global features and the local features are collectively referred to as target features. In the embodiment of the present application, the target features of at least two video frames can be fused by any of the following methods:

[0262] Compression method 1: Based on feature similarity, the target features of the most similar adjacent video frames in the memory unit are averaged and pooled.

[0263] Specifically, when the storage capacity of a memory unit has reached its upper limit, each time a new target feature, i.e., a target feature to be stored, is to be stored, the target features of the most similar adjacent video frames in the memory unit can be averaged and pooled, and then the target feature to be stored can be stored in the memory unit.

[0264] For example, for the global memory unit: calculate the global feature with the largest cosine similarity between adjacent video frames, compress it into a global feature in an average pooling manner, record it as a token, and it can be expressed as:

[0265] (Formula 1)

[0266] (Formula 2)

[0267] Among them, k represents the previous frame in the most similar adjacent video frames, That is, the global feature of the video frame; k+1 represents the next frame in the most similar adjacent video frames, That is, the global feature of the video frame. After the features of these two frames are fused, the global feature obtained can still be recorded as .

[0268] Specifically, assuming that the global features of frames 1 to 10 are currently stored in the global memory unit, M=10, then after the global features of the 11th video frame are encoded, the data of the global memory unit reaches the upper limit, and it is necessary to analyze and calculate the similarity between each adjacent frame from the 1st to the 10th frame, and then average pool the global features of the most similar adjacent video frames.

[0269] like Fig.11 As shown, it is a schematic diagram of a global feature compression process in an embodiment of the present application. Assuming that from right to left are several frames of global features that are successively stored in the global memory unit, assuming k=5, the global features of the 5th frame and the 6th frame need to be averaged and pooled as the new global features of the 5th frame, which can be recorded as 5', and then the global features of the 11th frame of the video frame can be stored in the global memory unit.

[0270] Furthermore, after encoding the 12th video frame to obtain the global features, the data of the global memory unit has also reached the upper limit, and it is necessary to analyze and calculate the similarity between each two adjacent frames in the 1st frame, the 2nd frame, the 3rd frame, the 4th frame, the compressed 5th frame, the 7th frame, the 8th frame, the 9th frame, the 10th frame and the 11th frame, and then average pool the global features of the most similar adjacent video frames.

[0271] It should be emphasized that, considering that each compression in this application will affect the number of the video frames corresponding to the global features, each adjacent frame here refers to the adjacent positions in the global memory unit, not the adjacent frames in the strict sense. For example, the original 4th frame and the compressed 5th frame are adjacent frames, the compressed 5th frame and the original 7th frame are also adjacent frames, and so on.

[0272] Assume that the global features of the 8th frame and the 9th frame need to be average-pooled as the new global features of the 8th frame, denoted as 8', and then the global features of the 12th frame of the video frame can be stored in the global memory unit.

[0273] Similarly, after encoding the 13th video frame to obtain the global features, the data of the global memory unit has also reached the upper limit, and it is necessary to analyze and calculate the similarity between each adjacent frame in the 1st frame, the 2nd frame, the 3rd frame, the 4th frame, the compressed 5th frame, the compressed 8th frame, the 10th frame, the 11th frame and the 12th frame, and then average pool the global features of the most similar adjacent video frames.

[0274] Assume that the global features of the compressed 5th and 7th frames need to be averaged and pooled as the global features of the second compressed 5th frame, recorded as 5'', and then the global features of the 13th frame of the video frame can be stored in the global memory unit.

[0275] The same principle applies to storing the global features of new video frames later, which will not be repeated here.

[0276] Similarly, for local memory units: a similar method is used to calculate the local feature with the largest cosine similarity between adjacent video frames (specifically, the local feature of the corresponding patch position), and it is compressed into a local feature by average pooling, which is recorded as a token. It should be noted that, unlike global features, a video frame can be divided into N patches, each patch corresponds to a local feature, so when fusing local features of adjacent video frames, it is necessary to pay attention to fusing the corresponding patch positions, which can be specifically expressed as:

[0277] (Formula 3)

[0278] (Formula 4)

[0279] Among them, k represents the previous frame in the most similar adjacent video frames, That is, the local features of the ith patch in the video frame; k+1 represents the next frame in the most similar adjacent video frame. That is, the local features of the ith patch in the video frame. After the features of the ith patch in these two frames are fused, the local features obtained can still be recorded as .

[0280] It should be noted that in the embodiment of the present application, the compression of global features and local features is implemented based on two memory units. Therefore, during each compression, it is not limited to compressing the global features and local features of the same two adjacent video frames at the same time.

[0281] In addition, considering that there may be multiple groups of most similar adjacent video frames at each compression, in this case, when average pooling the target features of the most similar adjacent video frames, an optional implementation is as follows:

[0282] The target features of the adjacent video frames with the longest storage time and the most similarity in the memory unit are averagely pooled; or, the target features of each group of adjacent video frames that are most similar in the memory unit are averagely pooled respectively.

[0283] Among them, the first way means that if there are multiple groups of most similar adjacent video frames, the adjacent video frames with relatively longer storage time are compressed according to the storage time. Of course, all groups can also be compressed.

[0284] For example, the global memory unit currently stores the global features of frames 1 to 10, M=10. After encoding the global features of the 11th video frame, the data of the global memory unit reaches the upper limit. It is necessary to analyze and calculate the similarity between each adjacent frame from the 1st to the 10th frame, and then average pool the global features of the most similar adjacent video frames.

[0285] Assuming that the most similar adjacent video frames calculated are [frame 5, frame 6] and [frame 8, frame 9], among which frame 5 has the longest storage time, the global features of frames 5 and 6 can be averaged and pooled as new global features of frame 5, and then the global features of frame 11 can be stored in the global memory unit.

[0286] Of course, the global features of the 5th and 6th frames can be averaged and pooled as new global features of the 5th frame, and the global features of the 8th and 9th frames can be averaged and pooled as new global features of the 8th frame, and then the global features of the 11th video frame can be stored in the global memory unit.

[0287] In addition, there is still one position left in the global memory unit, so after encoding the 12th video frame to obtain the global feature, the data of the global memory unit has not yet reached the upper limit, so the global feature of the 12th frame can be directly stored in the global memory unit. Furthermore, after encoding the 13th video frame to obtain the global feature, the data of the global memory unit has reached the upper limit, and compression can be performed again.

[0288] In the above implementation, the feature of compressing video frames with longer storage time according to storage time can not only reduce data redundancy and save space occupied by data, but also free up more space for new data to ensure the timeliness of video tag recognition results.

[0289] Compression method 2: Based on feature similarity, the target features to be stored are clustered with the target features stored in the memory unit, and the clustered target features of each video frame are averaged and pooled.

[0290] Among them, the target feature is a global feature or a local feature, and correspondingly, the memory unit is a global memory unit or a local memory unit.

[0291] Specifically, when the storage capacity of the memory unit has reached its upper limit, each time a new target feature, that is, a target feature to be stored, is to be stored, the target feature to be stored can be clustered with the most similar target feature in the memory unit, and the target features of the two clustered video frames can be average pooled to achieve a compression effect.

[0292] Assuming that the global features of frames 1 to 10 are currently stored in the global memory unit, M=10, after encoding the 11th video frame to obtain the global features, the data of the global memory unit reaches the upper limit, and it is necessary to analyze the frames 1 to 10 that are most similar to the 11th frame. For example, if it is the 10th frame, the 10th frame and the 11th frame can be average pooled, and the pooled global features can be stored in the global memory unit.

[0293] Similarly, during clustering, if there are multiple target features in the memory unit that are most similar to the target feature to be stored, the target feature with the longest storage time in the memory unit can be clustered with the target feature to be stored; or, the multiple target features in the memory unit that are most similar to the target feature to be stored and the target feature to be stored can be clustered.

[0294] Assuming that the global memory unit currently stores the compressed global features of the 1st to 11th frames, M=10, then after encoding the 12th video frame to obtain the global features, the data of the global memory unit reaches the upper limit, and it is necessary to analyze the global memory unit that is most similar to the 12th frame, such as the 2nd and 3rd frames. Among them, the 2nd frame has the longest storage time, and the 2nd and 12th frames can be averaged and pooled, and the pooled global features are stored in the global memory unit. In this way, after encoding the 13th video frame to obtain the global features, the data of the global memory unit still reaches the upper limit, and it is necessary to analyze the global memory unit that is most similar to the 13th frame and perform average pooling.

[0295] Of course, the average pooling operation can also be performed on the second, third and twelfth frames, and the pooled global features can be stored in the global memory unit. In this way, after encoding the 13th frame of video to obtain the global features, the data of the global memory unit has not yet reached the upper limit, so the global features of the 13th frame can be directly stored in the global memory unit.

[0296] Of course, when the storage capacity of the memory unit has reached the upper limit, each time a new target feature, i.e., a target feature to be stored, is to be stored, the target feature to be stored can be clustered with several similar target features in the memory unit, and the target features of the clustered video frames can be averaged and pooled to achieve the compression effect. Among them, several similar target features refer to those whose feature similarity with the target feature to be stored is greater than a certain threshold, or the top few in feature similarity ranking, etc., which are not specifically limited in this article.

[0297] Similarly, a similar compression method is used for local memory units. The only difference is that when local features of at least two video frames are fused, it is necessary to pay attention to fusing corresponding patch positions.

[0298] In the above embodiment, the target features are clustered into different groups based on feature similarity. If a new feature is similar enough to a feature in the memory unit, they will be classified into the same cluster, so that similar video frame features can be grouped together to reduce duplicate storage while retaining key information of the video.

[0299] It should be noted that the above-mentioned compression methods are all based on feature similarity. In addition, feature compression can also be performed based on storage time, for example:

[0300] Compression method three: Average pooling of target features of continuous video frames with the longest storage time to reduce inter-frame redundancy.

[0301] Specifically, when the storage capacity of a memory unit has reached its upper limit, each time a new target feature, i.e., a target feature to be stored, is to be stored, the target features of several adjacent video frames with the longest storage time in the memory unit can be average pooled, and then the target feature to be stored can be stored in the memory unit.

[0302] For example, each time a new target feature is to be stored, the target features of the first two video frames stored may be average pooled in chronological order.

[0303] In addition, a sliding window can also be used for feature compression. The minimum size of the sliding window is 2 and the maximum is M. For example, a sliding window of length 2 is selected in this article. Each time a new target feature is to be stored, the target features of the two video frames in the sliding window can be averaged and pooled, and so on.

[0304] It should be noted that the several compression methods listed above are just simple examples. In addition, other compression methods are also applicable to the embodiments of the present application and will not be described in detail here.

[0305] In the above implementation, the target features of two or more video frames are fused by using average pooling, which can avoid storing too many almost identical features, save storage space, and more accurately reflect the common attributes of these video frames. In addition, the average pooling operation can also reduce the impact of noise and improve the stability of the features.

[0306] In the embodiments of the present application, the target video is input in the form of a video frame sequence. After being compressed by the memory unit, an audit can be triggered at any time. The setting of the triggering time is not specifically limited in this article. An optional implementation is as follows:

[0307] Before adopting the cross-attention mechanism to extract the first cross-attention feature between the self-attention feature and each global feature in the global feature sequence, and before extracting the second cross-attention feature between the self-attention feature and each local feature in the local feature sequence, it is also necessary to determine whether the specified recognition time has been reached.

[0308] If it is determined that the specified recognition time has been reached, the global feature sequence stored in the global memory unit is read, and the local feature sequence stored in the local memory unit is read, and then the subsequent process is executed to realize the recognition of the video tag.

[0309] If it is not determined that the specified recognition time has been reached, the feature sequences stored in the global memory unit and the local memory unit will not be read temporarily. Accordingly, the subsequent tag recognition process will not be executed temporarily, and only feature compression based on the memory unit is required.

[0310] The designated recognition time in the embodiment of the present application can be flexibly set according to actual needs, and can also be set according to the video length, type, etc. of the target video. For example, for medium videos and long videos, prediction can be made after the video frame sequence input is completed; for live broadcast scenes, such as video streams, prediction can be made every N minutes on a regular basis, and so on.

[0311] That is to say, the review of the target video can be triggered after the target video input is completed. In this case, all video frames of the target video can be encoded and input into MB, and MB is compressed once or multiple times. Finally, the features of each video frame in the target video are compressed into a feature sequence of fixed length and then output. Correspondingly, the specified recognition time refers to the time when the feature compression is completed.

[0312] In addition, as the target video is continuously input, each time a specified recognition time is reached, a review of the target video is triggered. In this case, the feature sequence output by MB during each review is based on the feature sequence output during the previous review, and is further compressed according to the features of the video frames input during this time period.

[0313] It should be noted that the several audit methods listed above are just simple examples. In addition, other audit methods are also applicable to the embodiments of the present application and will not be described in detail here.

[0314] In the above embodiment, by setting a specified recognition time, the subsequent tag recognition process is triggered when the specified recognition time is reached, which can avoid the model from constantly recognizing video tags and causing unnecessary waste of resources. In addition, the specified recognition time can be flexibly set by the object, making the recognition of video tags more flexible.

[0315] S92: After concatenating the global query features and local query features obtained through pre-training, the self-attention mechanism is used to extract the corresponding self-attention features; the global query features and local query features are obtained by training the corresponding learnable query features based on sample videos; the self-attention features integrate the key information in the global query features and the local query features.

[0316] The global query feature is obtained by training the first learnable query feature, and the local query feature is obtained by training the second learnable query feature. The specific training process is as above and will not be repeated here.

[0317] The same training process, in this step, can be implemented based on the query converter in the video tag recognition model. The query converter includes a self-attention network and two parallel cross-attention networks. The concatenation result of the global query feature and the local query feature is used as the input of the self-attention network in the query converter. This process allows the global feature query sequence to capture local details, while allowing the local feature query sequence to obtain the global context, and finally generates a fusion feature representation that is both meticulous and broad-viewed.

[0318] like Fig.12 As shown, it is a schematic diagram of the calculation process of an attention mechanism in an embodiment of the present application. The attention mechanism allows the model to focus on different parts of the input sequence in order to better capture contextual information. Specifically, Fig.12 The left part of the figure shows three matrix multiplication operations, corresponding to the query, key, and value matrices in the attention mechanism. The result of these matrix multiplications is to generate Q, K, and V matrices, which are all of the same size. These matrix multiplications are obtained by multiplying the input matrix X with the weight matrices Wq, Wk, and Wv. Among them, the three weight matrices are trainable parameters used to extract features.

[0319] The formula on the right shows the calculation process of the attention mechanism. First, the Q and K matrices are dot-producted (inner product), and then normalized by the softmax function to form the attention score matrix.k The size of the key dimension is often used to scale the dot product attention scores to help gradients flow through the network. This normalized attention score matrix represents the similarity between the query vector at each position and the key vector at each position. Next, the attention score matrix is ​​multiplied by the V matrix to obtain the final attention output Z. This output contains information from all positions of the input, but the weights are assigned based on the attention scores, so the positions of the key vectors that are most relevant to the query vector have higher weights. This process can be repeated multiple times to form Multi-Head Attention, where each head generates an attention output, allowing the model to focus on different aspects of the input, which are then merged to enhance the model's expressiveness and generalization capabilities.

[0320] In the self-attention network, the input X is the concatenation result of the global query feature and the local query feature, which can be expressed as X=concat(Q1, Q2), Wq, Wk and Wv are the weight matrices of the self-attention network, and the output Z of the self-attention network is the self-attention feature in the embodiment of the present application.

[0321] S93: Using the cross-attention mechanism, respectively extract the first cross-attention feature between the self-attention feature and each global feature in the global feature sequence, and respectively extract the second cross-attention feature between the self-attention feature and each local feature in the local feature sequence.

[0322] Since the self-attention feature obtained in step S92 is obtained by concatenating the global query feature and the local query feature, in step S93, it can also be implemented based on the query converter in the video tag recognition model. The two cross-attention networks in the query converter can be recorded as: a first cross-attention network (corresponding to the global feature) and a second cross-attention network (corresponding to the local feature); S93 can be implemented according to the flowchart shown below, including the following steps S931~S933 ( Fig. 9 not shown):

[0323] S931: Split the self-attention feature into a first self-attention feature corresponding to the global feature and a second self-attention feature corresponding to the local feature.

[0324] In the same training process, before extracting the cross-attention feature, the self-attention feature obtained in the above S92 is first split into two parts, recorded as the first self-attention feature and the first self-attention feature.

[0325] S932: By querying the first cross-attention network in the converter, based on the cross-attention mechanism, extract the first self-attention feature and the first cross-attention feature between each global feature in the global feature sequence respectively.

[0326] Specifically, in the first cross attention network, there are two inputs, one is the global feature and the other is the first self-attention feature. Fig.12 , then there are two X, among which the X in the first matrix multiplication on the left side can be understood as the first cross-attention feature, generating the Q matrix; the X in the second and third matrix multiplications on the left side can be understood as global features, generating K and V matrices. Based on the first self-attention feature that integrates the key information in the global query feature and the local query feature, try to extract relevant information from the global feature. At this time, the output Z is the first cross-attention feature in the embodiment of the present application.

[0327] S933: By querying the second cross-attention network in the converter, based on the cross-attention mechanism, the second self-attention features and the second cross-attention features between each local feature in the local feature sequence are extracted respectively.

[0328] Similar to step S932, S933 and S932 are implemented through two parallel cross attention networks. The specific execution process is similar. The second cross attention network also has two inputs, one for the local feature and the other for the second self-attention feature. Fig.12 , then there are two X, among which, the X in the second matrix multiplication on the left side can be understood as the second cross-attention feature, generating the Q matrix; the X in the second and third matrix multiplications on the left side can be understood as local features, generating the K and V matrices. Based on the second self-attention feature that integrates the local query feature and the key information in the local query feature, try to extract relevant information from the local feature, and the output Z at this time is the second cross-attention feature in the embodiment of the present application.

[0329] In the above implementation, in the self-attention module, the global feature query sequence and the local feature query sequence can see each other, so that the two features can interact with each other, and obtain a self-attention feature that integrates the key information in the global query feature and the local query feature. On this basis, the self-attention feature is split and input into the corresponding cross attention module, and two parallel cross attention modules are used to process the global features and local features respectively, to ensure that the local features are paid attention to while extracting relevant information from the global features, and the global features are also paid attention to while extracting relevant information from the local features, so as to extract a more complete first cross-attention feature and a second cross-attention feature. In addition, further compression through cross attention can obtain a more compact visual token sequence.

[0330] In the embodiment of the present application, the querying transformer serves as a bridge between non-text tokens and LLM. It can convert inputs from different modalities into tokens that can be understood by LLM. Then, the video tag can be predicted by LLM in combination with this. The specific process is as follows:

[0331] S94: Identify the video tag of the target video based on the obtained first cross-attention features and second cross-attention features.

[0332] The MLLM in the embodiment of the present application uses two modes, namely, visual mode and text mode. Fig.13 As shown, after each video frame in the target video (video stream) in the embodiment of the present application is encoded by the visual encoder, features of two visual modalities can be obtained, including global features and local features. Then, the global features are compressed by the global feature MB, and the local features are compressed by the local feature MB. Then, they are further fused and compressed by the multi-head self-attention and multi-head cross-attention in the Querying transformer to obtain the first cross-attention feature and the second cross-attention feature. After splicing, the FC layer is used for dimension conversion to obtain the final visual token, which is denoted as [video_emb], that is, the visual feature in the present application. For the specific implementation method, please refer to the above embodiment, and the repeated parts will not be repeated; in addition to the title of the video, the features of the text modality can also include descriptive text, key text extracted from OCR / ASR, etc., denoted as [video_txt], that is, the text feature in the present application.

[0333] Use [video_txt] and text instructions as prompts, and combine [video_emb] to input LLM to generate predicted label results.

[0334] It should be noted that Fig.13 The visual label recognition models listed are in Figure 8 It is trained on the basis shown in the figure, and therefore also includes the LoRA module.

[0335] In summary, an optional implementation of S94 is as follows:

[0336] After concatenating the first cross-attention features and the second cross-attention features, they are converted into visual features that are aligned with the text feature dimensions of the target video. Then, the text features and text instructions of the target video are used as question-answer prompt information and combined with the visual features to input into the large language model to generate a video label for the target video.

[0337] Specifically, the conversion of feature dimensions can be achieved through a fully connected layer, which can be located in the query converter as the output of the query converter, or can be used as an independent feature dimension conversion layer, which is not specifically limited in this article. Fig.13 In the query converter, after the FFN outputs the first cross attention and the second cross attention, [video_emb] and [video_txt] are dimensionally aligned after passing through the FC layer.

[0338] The text feature [video_txt] of the target video refers to the encoding of the video text description information of the target video, which includes but is not limited to the video title, description text, key text extracted by OCR, subtitles or transcriptions generated by ASR, comments and forum discussions, metadata information, etc. For details, please refer to the relevant description of the model training part, which will not be repeated here.

[0339] After the visual features and text features are aligned, they can be input into the large language model, where the text features and text instructions are used as the question-answering prompt of the large language model. Since LLM is expected to output video tags, the text instructions here can be one of the following statements:

[0340] "What tags can be used to summarize the above video?", "Please output some key content tags based on the content of the above video.", "Based on the above video content, generate main content tags.", etc. This article does not make specific restrictions.

[0341] In the above implementation, by aligning feature dimensions, the representation spaces of different modalities can be unified so as to effectively integrate and understand cross-modal information and narrow the representation differences between modalities, so that LLM can process diverse data in the same semantic space, promote knowledge transfer, and enhance generalization capabilities, thereby improving the recognition effect of video tags.

[0342] Optionally, the query converter includes multiple layers of stacked decoders; for the first decoder, the input features include a global feature sequence and a local feature sequence; for other decoders, the input features include each first cross-attention feature and each second cross-attention feature output by the previous decoder. The specific implementation method is the same as the training process, and please refer to the relevant description of the above training part, which will not be repeated here; on this basis, an optional implementation method of S94 is: based on each first cross-attention feature and each second cross-attention feature output by the last decoder, identify the video label of the target video.

[0343] Still Figure 5As shown in the figure, the input of 502 in Decoder1 is the global features output by the global memory unit, and the input of 503 in Decoder1 is the local features output by the local memory unit; the input of 502 in Decoder2 is the first cross-attention features output by 504 in Decoder1, and the input of 503 in Decoder2 is the second cross-attention features output by 505 in Decoder1; ...; the input of 502 in Decoder5 is the first cross-attention features output by 504 in Decoder4, and the input of 503 in Decoder5 is the second cross-attention features output by 505 in Decoder4. Finally, the outputs of 504 and 505 in Decoder5 are spliced ​​and dimensionally aligned through FC, and then the corresponding video label can be obtained by combining the prompt input LLM.

[0344] In the multi-layer stacked decoder structure listed above, between each layer, global features and local features are fused through self attention and further compressed through cross attention, so that a more compact visual token sequence can be obtained. Extracting video tags on this basis can further reduce the amount of LLM calculation and improve recognition efficiency.

[0345] See also Fig.14 As shown, it is a schematic diagram of the interaction logic between a terminal device and a server in an embodiment of the present application.

[0346] Among them, a client related to video tag recognition can be installed on the terminal device, and the object can determine the target video based on the client, such as Fig.14 The target video is sent to the server through the terminal device. The server 120 has pre-deployed a trained video tag recognition model. Then, the server uses the video tag recognition method in the embodiment of the present application to recognize the video tag of the target video based on the video tag recognition model. The specific recognition process is shown in the above embodiment, and the repeated parts are not repeated. Finally, the server can return the recognized video tag to the terminal device, which is presented to the object through the client.

[0347] in, Fig.14 This is an example of a long video. In addition, the target video can also be a live video. In this way, video tag recognition can be performed every 5 minutes during the live broadcast, and so on.

[0348] In summary, the implementation method of this application is simple and suitable for mainstream multimodal models. It can effectively compress visual features of various videos and reduce information loss. At the same time, it can save costs and improve machine review efficiency.

[0349] Based on the same inventive concept, the present application also provides a video tag recognition device. Fig.15 As shown, it is a schematic diagram of the structure of a video tag identification device 1500, which may include:

[0350] The coding and compression module 1501 is used to encode each video frame in the target video based on the video tag recognition model to obtain corresponding global features and local features, and compress the obtained global feature set and local feature set based on feature similarity to obtain global feature sequences and local feature sequences of preset storage lengths;

[0351] Fusion module 1502, used to extract corresponding self-attention features by using a self-attention mechanism after splicing the global query features and local query features obtained by pre-training; the global query features and the local query features are obtained by training corresponding learnable query features based on sample videos; the self-attention features fuse key information in the global query features and the local query features;

[0352] A cross compression module 1503 is used to adopt a cross attention mechanism to respectively extract a first cross attention feature between the self-attention feature and each global feature in the global feature sequence, and to respectively extract a second cross attention feature between the self-attention feature and each local feature in the local feature sequence;

[0353] The tag identification module 1504 is used to identify the video tag of the target video based on the obtained first cross-attention features and second cross-attention features.

[0354] Optionally, the video tag recognition model includes a query converter; then the cross compression module 1503 is specifically used for:

[0355] Splitting the self-attention feature into a first self-attention feature corresponding to the global feature and a second self-attention feature corresponding to the local feature;

[0356] Extracting the first self-attention feature and the first cross-attention feature between each global feature in the global feature sequence respectively based on the cross-attention mechanism through the first cross-attention network in the query converter;

[0357] Through the second cross-attention network in the query converter, based on the cross-attention mechanism, the second self-attention feature and the second cross-attention feature between each local feature in the local feature sequence are extracted respectively.

[0358] Optionally, the query converter includes multiple layers of stacked decoders; for a first decoder, the input features include the global feature sequence and the local feature sequence; for other decoders, the input features include each first cross-attention feature and each second cross-attention feature output by a previous decoder;

[0359] The tag identification module 1504 is specifically used for:

[0360] Based on the first cross-attention features and the second cross-attention features output by the last decoder, a video tag of the target video is identified.

[0361] Optionally, the video tag recognition model includes a large language model; the tag recognition module 1504 is specifically used for:

[0362] After concatenating the first cross-attention features and the second cross-attention features, the concatenation is converted into a visual feature aligned with the text feature dimension of the target video;

[0363] The text features and text instructions of the target video are used as question-answer prompt information, combined with the visual features, and input into the large language model to generate a video tag for the target video.

[0364] Optionally, each local feature corresponds to an image block in a video frame; the encoding and compression module 1501 is specifically used for:

[0365] Each time a video frame is encoded, the global features of the video frame are stored in a global memory unit, and the local features of the video frame are stored in a local memory unit;

[0366] When the number of global features stored in the global memory unit reaches the preset storage length, based on feature similarity, feature fusion is performed on the global features of at least two video frames in the global memory unit, and the global features to be stored are stored in the global memory unit;

[0367] And, for each image block, when the number of corresponding local features stored in the local memory unit reaches the preset storage length, based on feature similarity, feature fusion is performed on the corresponding local features of at least two video frames in the local memory unit, and the local features to be stored are stored in the local memory unit.

[0368] Optionally, the device further comprises:

[0369] The reading module 1505 is used to determine that a specified recognition time has been reached before the cross-attention mechanism is adopted in the cross-compression module 1503 to respectively extract the first cross-attention feature between the self-attention feature and each global feature in the global feature sequence, and respectively extract the second cross-attention feature between the self-attention feature and each local feature in the local feature sequence, and then read the global feature sequence stored in the global memory unit and the local feature sequence stored in the local memory unit.

[0370] Optionally, the encoding compression module 1501 is further configured to perform feature fusion on target features of at least two video frames in any of the following ways:

[0371] Based on feature similarity, the target features of the most similar adjacent video frames in the memory unit are averaged and pooled;

[0372] Based on feature similarity, the target features to be stored are clustered with the target features stored in the memory unit, and the clustered target features of each video frame are average pooled;

[0373] The target feature is a global feature or a local feature, and correspondingly, the memory unit is a global memory unit or a local memory unit.

[0374] Optionally, if there are multiple groups of most similar adjacent video frames, the encoding compression module 1501 is specifically used to:

[0375] Average pooling of target features of adjacent video frames that have the longest storage time and are most similar in the memory unit; or

[0376] The target features of each group of adjacent video frames that are most similar in the memory unit are average-pooled respectively.

[0377] Based on the same inventive concept, the present application also provides a training device for a video tag recognition model. Fig.16 As shown, it is a structural schematic diagram of a training device 1600 for a video tag recognition model, which may include:

[0378] The first feature extraction module 1601 is used to encode each video frame in the sample video based on the video tag recognition model to be trained, obtain corresponding global features and local features, and compress the obtained global feature set and local feature set based on feature similarity to obtain a global feature sequence and a local feature sequence of a preset storage length;

[0379] A second feature extraction module 1602 is used to extract a corresponding self-attention feature by using a self-attention mechanism after concatenating the first learnable query feature to be trained with the second learnable query feature; the self-attention feature integrates key information in the first learnable query feature and the second learnable query feature;

[0380] The third feature extraction module 1603 is used to adopt a cross attention mechanism to extract a first cross attention feature between the self-attention feature and each global feature in the global feature sequence, and a second cross attention feature between the self-attention feature and each local feature in the local feature sequence;

[0381] The parameter adjustment module 1604 is used to adjust the parameters of the video label recognition model according to the difference between the video label and video text description information of the sample video predicted based on the obtained first cross-attention features and the second cross-attention features and the corresponding sample labels. The trained video label recognition model is used to extract the video label of the target video.

[0382] Optionally, the video tag recognition model includes a large language model, and a query converter implemented based on a self-attention mechanism and a cross-attention mechanism, wherein the query converter is used to extract the first cross-attention feature and the second cross-attention feature; the device also includes:

[0383] The prediction module 1605 is used to convert the first cross-attention features and the second cross-attention features into visual features aligned with the text feature dimension of the sample video in the first training stage; input the first text instruction as question-answering prompt information in combination with the visual feature into the large language model to generate video text description information of the sample video;

[0384] In the second training stage, the first cross-attention feature and the second cross-attention feature are concatenated and converted into visual features aligned with the text feature dimension of the sample video; the text features and the second text instructions are used as question-answer prompt information and combined with the visual features to input into the large language model to generate a video label for the sample video.

[0385] Optionally, the parameter adjustment module 1604 is specifically used for:

[0386] In the first training stage, updating parameters of the query converter according to the difference between the predicted video text description information and the video text description information in the sample label;

[0387] In the second training stage, parameters of the query converter and the large language model are updated according to the difference between the predicted video label and the video label in the sample label.

[0388] Optionally, the parameter adjustment module 1604 is further configured to update the parameters of the large language model in the following manner:

[0389] Fixing original parameters of the large language model and adding a low-rank adaptive matrix to the large language model;

[0390] The parameters of the large language model are updated by updating the parameters of the low-rank adaptive matrix.

[0391] Due to the present application, when extracting the visual features of the target video based on the video tag recognition model, there is no need to extract frames, but each video frame in the target video is encoded. After encoding, a global feature set composed of the global features of each video frame and a local feature set composed of the local features of each video frame can be obtained. On this basis, based on feature similarity, the obtained global feature set and local feature set are compressed respectively, specifically referring to feature fusion of the global features of some video frames, and feature fusion of the local features of some video frames, to obtain a global feature sequence and a local feature sequence of a preset storage length. Through the above operation, it can be ensured that the global feature sequence and the local feature sequence at this time can cover most of the content of the target video. On this basis, the key information in the global query feature and the local query feature is fused through the self-attention mechanism, and then further compressed through the cross-attention mechanism to obtain a more compact visual feature sequence.

[0392] The above method can effectively compress the visual features of medium and long videos and reduce information loss. At the same time, it can improve the efficiency of label recognition by compressing the features, thereby realizing efficient and accurate recognition of video labels of medium and long videos.

[0393] For the convenience of description, the above parts are divided into modules (or units) according to their functions and described separately. Of course, when implementing this application, the functions of each module (or unit) can be implemented in the same or multiple software or hardware.

[0394] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0395] After introducing the video tag recognition method and device according to an exemplary embodiment of the present application, next, an electronic device according to another exemplary embodiment of the present application is introduced.

[0396] Those skilled in the art will appreciate that various aspects of the present application may be implemented as a system, method or program product. Therefore, various aspects of the present application may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to as "circuit", "module" or "system" herein.

[0397] Based on the same inventive concept as the above method embodiment, an electronic device is also provided in the embodiment of the present application. In one embodiment, the electronic device may be a server, such as Figure 1 In this embodiment, the structure of the electronic device can be as follows: Fig.17 As shown, it includes a memory 1701 , a communication module 1703 and one or more processors 1702 .

[0398] The memory 1701 is used to store computer programs executed by the processor 1702. The memory 1701 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and programs required for running the instant messaging function, etc.; the data storage area may store various instant messaging information and operation instruction sets, etc.

[0399] The memory 1701 may be a volatile memory, such as a random-access memory (RAM); the memory 1701 may also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); or the memory 1701 may be any other medium that can be used to carry or store a desired computer program in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 1701 may be a combination of the above memories.

[0400] The processor 1702 may include one or more central processing units (CPU) or a digital processing unit, etc. The processor 1702 is configured to implement the above-mentioned video tag recognition method when calling the computer program stored in the memory 1701 .

[0401] The communication module 1703 is used to communicate with terminal devices and other servers.

[0402] The specific connection medium between the memory 1701, the communication module 1703 and the processor 1702 is not limited in the embodiment of the present application. Fig.17 In the embodiment, the memory 1701 and the processor 1702 are connected via a bus 1704. The bus 1704 is connected to the processor 1702 via a bus 1704. Fig.17 The connections between the other components are described with bold lines, which are only for illustration and are not intended to be limiting. The bus 1704 can be divided into an address bus, a data bus, a control bus, etc. For ease of description, Fig.17 The diagram shows that only one thick line is used, but this does not mean that there is only one bus or only one type of bus.

[0403] The memory 1701 stores a computer storage medium, and the computer storage medium stores computer executable instructions, and the computer executable instructions are used to implement the video tag recognition method or the video tag recognition model training method of the embodiment of the present application. The processor 1702 is used to execute the above-mentioned video tag recognition method or the video tag recognition model training method, such as Figure 3 or Fig. 9 shown.

[0404] In another embodiment, the electronic device may also be other electronic devices, such as Figure 1 The terminal device 110 shown in FIG. 1 is a terminal device 110 shown in FIG. 1 . In this embodiment, the structure of the electronic device can be as follows: Fig.18 As shown, it includes: a communication component 1810, a memory 1820, a display unit 1830, a camera 1840, a sensor 1850, an audio circuit 1860, a Bluetooth module 1870, a processor 1880 and other components.

[0405] The communication component 1810 is used to communicate with the server. In some embodiments, a wireless fidelity (WiFi) module may be included. The WiFi module belongs to a short-range wireless transmission technology. The electronic device can help the user to send and receive information through the WiFi module.

[0406] The memory 1820 can be used to store software programs and data. The processor 1880 executes various functions and data processing of the terminal device 110 by running the software programs or data stored in the memory 1820. The memory 1820 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. The memory 1820 stores an operating system that enables the terminal device 110 to run. In the present application, the memory 1820 can store an operating system and various application programs, and can also store a computer program for executing the video tag recognition method of the embodiment of the present application.

[0407] The display unit 1830 may also be used to display information input by the user or information provided to the user and a graphical user interface (GUI) of various menus of the terminal device 110. Specifically, the display unit 1830 may include a display screen 1832 disposed on the front of the terminal device 110. The display screen 1832 may be configured in the form of a liquid crystal display, a light emitting diode, etc. The display unit 1830 may be used to display a video interface, etc. in the embodiment of the present application.

[0408] The display unit 1830 can also be used to receive input digital or character information and generate signal input related to user settings and function control of the terminal device 110. Specifically, the display unit 1830 may include a touch screen 1831 arranged on the front of the terminal device 110, which can collect user touch operations on or near it, such as clicking a button, dragging a scroll box, etc.

[0409] The touch screen 1831 can be covered on the display screen 1832, or the touch screen 1831 and the display screen 1832 can be integrated to realize the input and output functions of the terminal device 110. The integrated touch screen can be referred to as a touch display screen. In this application, the display unit 1830 can display the application and the corresponding operation steps.

[0410] The camera 1840 can be used to capture still images, and the user can publish the images taken by the camera 1840 through the application. The camera 1840 can be one or more. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then transmits the electrical signal to the processor 1880 to convert it into a digital image signal.

[0411] The terminal device may further include at least one sensor 1850, such as an acceleration sensor 1851, a distance sensor 1852, a fingerprint sensor 1853, and a temperature sensor 1854. The terminal device may also be configured with other sensors such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, and a motion sensor.

[0412] The audio circuit 1860, the speaker 1861, and the microphone 1862 can provide an audio interface between the user and the terminal device 110. The audio circuit 1860 can transmit the electrical signal converted from the received audio data to the speaker 1861, which is converted into a sound signal for output. The terminal device 110 can also be configured with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 1862 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1860 and converted into audio data, and then the audio data is output to the communication component 1810 to be sent to, for example, another terminal device 110, or the audio data is output to the memory 1820 for further processing.

[0413] The Bluetooth module 1870 is used to exchange information with other Bluetooth devices having Bluetooth modules through the Bluetooth protocol. For example, the terminal device can establish a Bluetooth connection with a wearable electronic device (such as a smart watch) that also has a Bluetooth module through the Bluetooth module 1870 to exchange data.

[0414] The processor 1880 is the control center of the terminal device. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal device and processes data by running or executing software programs stored in the memory 1820 and calling data stored in the memory 1820. In some embodiments, the processor 1880 may include one or more processing units; the processor 1880 may also integrate an application processor and a baseband processor, wherein the application processor mainly processes the operating system, the user interface, and the application program, and the baseband processor mainly processes wireless communication. It is understandable that the above-mentioned baseband processor may not be integrated into the processor 1880. In the present application, the processor 1880 can run the operating system, the application program, the user interface display and the touch response, as well as the video tag recognition method or the training method of the video tag recognition model in the embodiment of the present application. In addition, the processor 1880 is coupled to the display unit 1830.

[0415] In some optional implementations, various aspects of the video tag identification method provided by the present application may also be implemented in the form of a program product, which includes a computer program. When the program product is run on an electronic device, the computer program is used to enable the electronic device to perform the steps of the video tag identification method according to various exemplary implementations of the present application described above in this specification. For example, the electronic device may perform the following steps: Figure 3 or Fig. 9 Follow the steps shown in .

[0416] The program product may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0417] The program product of the embodiment of the present application may adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited thereto, and in this document, a readable storage medium may be any tangible medium containing or storing a program, which can be used by or in combination with a command execution system, apparatus, or device.

[0418] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, wherein a readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A readable signal medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with a command execution system, apparatus, or device.

[0419] The computer program embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0420] The computer program for performing the operation of the present application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer program can be executed entirely on the user electronic device, partially on the user electronic device, as an independent software package, partially on the user electronic device and partially on the remote electronic device, or completely on the remote electronic device or server. In the case of involving a remote electronic device, the remote electronic device can be connected to the user electronic device through any type of network including a local area network (LAN) or a wide area network (WAN), or can be connected to an external electronic device (e.g., using an Internet service provider to connect through the Internet).

[0421] It should be noted that, although several units or subunits of the device are mentioned in the above detailed description, this division is merely exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided into multiple units to be embodied.

[0422] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0423] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain a computer-usable computer program.

[0424] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program commands. These computer program commands can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the commands executed by the processor of the computer or other programmable data processing device generate commands for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0425] These computer program commands may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the commands stored in the computer readable memory produce an article of manufacture comprising a command device, the command device implementing the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0426] These computer program commands can also be loaded onto a computer or other programmable data processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the commands executed on the computer or other programmable device provide the instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0427] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0428] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A video tag recognition method, characterized in that: The method comprises: Based on the video tag recognition model, each video frame in the target video is encoded respectively to obtain the corresponding global features and local features, and based on the feature similarity, the obtained global feature set and local feature set are compressed respectively to obtain the global feature sequence and local feature sequence of the preset storage length; After concatenating the global query features and local query features obtained through pre-training, the corresponding self-attention features are extracted using a self-attention mechanism; the global query features and the local query features are obtained by training the corresponding learnable query features based on sample videos; the self-attention features integrate the key information in the global query features and the local query features; Adopting a cross attention mechanism, respectively extracting a first cross attention feature between the self-attention feature and each global feature in the global feature sequence, and respectively extracting a second cross attention feature between the self-attention feature and each local feature in the local feature sequence; Based on the obtained first cross-attention features and second cross-attention features, a video tag of the target video is identified.

2. The method according to claim 1, characterized in that The video tag recognition model includes a query converter; the cross-attention mechanism is adopted to respectively extract the first cross-attention feature between the self-attention feature and each global feature in the global feature sequence, and respectively extract the second cross-attention feature between the self-attention feature and each local feature in the local feature sequence, including: Splitting the self-attention feature into a first self-attention feature corresponding to the global feature and a second self-attention feature corresponding to the local feature; Extracting, through a first cross-attention network in the query converter, first cross-attention features between the first self-attention feature and each global feature in the global feature sequence based on a cross-attention mechanism; Through the second cross-attention network in the query converter, based on the cross-attention mechanism, the second cross-attention features between the second self-attention feature and each local feature in the local feature sequence are extracted respectively.

3. The method according to claim 2, characterized in that The query converter includes a multi-layer stacked decoder; for the first decoder, the input feature includes the global feature sequence and the local feature sequence; for other decoders, the input feature includes each first cross-attention feature and each second cross-attention feature output by the previous decoder; The step of identifying the video tag of the target video based on the obtained first cross-attention features and the obtained second cross-attention features includes: Based on the first cross-attention features and the second cross-attention features output by the last decoder, a video tag of the target video is identified.

4. The method according to claim 1, characterized in that The video tag recognition model includes a large language model; the step of identifying the video tag of the target video based on the obtained first cross-attention features and the obtained second cross-attention features includes: After concatenating the first cross-attention features and the second cross-attention features, the concatenation is converted into a visual feature aligned with the text feature dimension of the target video; The text features and text instructions of the target video are used as question-answer prompt information, combined with the visual features, and input into the large language model to generate a video tag for the target video.

5. The method according to any one of claims 1 to 4, characterized in that Each local feature corresponds to an image block in a video frame; based on feature similarity, the obtained global feature set and local feature set are respectively compressed to obtain a global feature sequence and a local feature sequence of a preset storage length, including: Each time a video frame is encoded, the global features of the video frame are stored in a global memory unit, and the local features of the video frame are stored in a local memory unit; When the number of global features stored in the global memory unit reaches the preset storage length, based on feature similarity, feature fusion is performed on the global features of at least two video frames in the global memory unit, and the global features to be stored are stored in the global memory unit; And, for each image block, when the number of corresponding local features stored in the local memory unit reaches the preset storage length, based on feature similarity, feature fusion is performed on the corresponding local features of at least two video frames in the local memory unit, and the local features to be stored are stored in the local memory unit.

6. The method according to claim 5, characterized in that Before adopting the cross attention mechanism to extract the first cross attention feature between the self-attention feature and each global feature in the global feature sequence, and extracting the second cross attention feature between the self-attention feature and each local feature in the local feature sequence, the method further includes: When it is determined that the designated recognition time has been reached, the global feature sequence stored in the global memory unit is read, and the local feature sequence stored in the local memory unit is read.

7. The method according to claim 5, characterized in that Feature fusion of target features of at least two video frames is performed in any of the following ways: Based on feature similarity, the target features of the most similar adjacent video frames in the memory unit are averaged and pooled; Based on feature similarity, the target features to be stored are clustered with the target features stored in the memory unit, and the clustered target features of each video frame are average pooled; The target feature is a global feature or a local feature, and correspondingly, the memory unit is a global memory unit or a local memory unit.

8. The method according to claim 7, characterized in that If there are multiple groups of most similar adjacent video frames, then the target features of the most similar adjacent video frames in the memory unit are averagely pooled, including: Average pooling of target features of adjacent video frames that have the longest storage time and are most similar in the memory unit; or The target features of each group of adjacent video frames that are most similar in the memory unit are average-pooled respectively.

9. A method for training a video tag recognition model, characterized in that: The method comprises: Based on the video tag recognition model to be trained, each video frame in the sample video is encoded respectively to obtain corresponding global features and local features, and based on feature similarity, the obtained global feature set and local feature set are compressed respectively to obtain global feature sequences and local feature sequences of preset storage lengths; After concatenating the first learnable query feature to be trained and the second learnable query feature, a corresponding self-attention feature is extracted using a self-attention mechanism; the self-attention feature integrates key information in the first learnable query feature and the second learnable query feature; Adopting a cross attention mechanism, respectively extracting a first cross attention feature between the self-attention feature and each global feature in the global feature sequence, and a second cross attention feature between the self-attention feature and each local feature in the local feature sequence; According to the difference between the video label and video text description information of the sample video predicted based on the obtained first cross-attention features and the second cross-attention features and the corresponding sample labels, the parameters of the video label recognition model are adjusted, and the trained video label recognition model is used to extract the video label of the target video.

10. The method according to claim 9, characterized in that The video tag recognition model includes a large language model and a query converter implemented based on a self-attention mechanism and a cross-attention mechanism, wherein the query converter is used to extract the first cross-attention feature and the second cross-attention feature; based on the obtained first cross-attention features and the obtained second cross-attention features, the video tag and the video text description information of the sample video are predicted, including: In a first training stage, the first cross-attention features and the second cross-attention features are concatenated and converted into visual features aligned with the text feature dimension of the sample video; the first text instruction is used as question-answer prompt information and combined with the visual feature to input into the large language model to generate video text description information of the sample video; In the second training stage, the first cross-attention feature and the second cross-attention feature are concatenated and converted into visual features aligned with the text feature dimension of the sample video; the text features and the second text instructions are used as question-answer prompt information and combined with the visual features to input into the large language model to generate a video label for the sample video.

11. The method according to claim 10, characterized in that The step of adjusting parameters of the video label recognition model according to the difference between the video label and the video text description information of the sample video predicted based on each first cross-attention feature and each second cross-attention feature obtained and the corresponding sample label comprises: In the first training stage, updating parameters of the query converter according to the difference between the predicted video text description information and the video text description information in the sample label; In the second training stage, parameters of the query converter and the large language model are updated according to the difference between the predicted video label and the video label in the sample label.

12. A video tag recognition device, characterized in that: include: The coding and compression module is used to encode each video frame in the target video based on the video tag recognition model to obtain corresponding global features and local features, and compress the obtained global feature set and local feature set based on feature similarity to obtain global feature sequences and local feature sequences of preset storage lengths; A fusion module is used to extract corresponding self-attention features by using a self-attention mechanism after splicing the global query features and local query features obtained by pre-training; the global query features and the local query features are obtained by training the corresponding learnable query features based on sample videos; the self-attention features fuse the key information in the global query features and the local query features; A cross compression module, used for adopting a cross attention mechanism to respectively extract a first cross attention feature between the self-attention feature and each global feature in the global feature sequence, and respectively extract a second cross attention feature between the self-attention feature and each local feature in the local feature sequence; The tag identification module is used to identify the video tag of the target video based on the obtained first cross-attention features and second cross-attention features.

13. A video tag recognition device, characterized in that: include: The first feature extraction module is used to encode each video frame in the sample video based on the video tag recognition model to be trained, obtain corresponding global features and local features, and compress the obtained global feature set and local feature set based on feature similarity to obtain global feature sequences and local feature sequences of preset storage lengths; A second feature extraction module is used to extract a corresponding self-attention feature by using a self-attention mechanism after concatenating the first learnable query feature to be trained with the second learnable query feature; the self-attention feature integrates key information in the first learnable query feature and the second learnable query feature; A third feature extraction module is used to adopt a cross attention mechanism to respectively extract a first cross attention feature between the self-attention feature and each global feature in the global feature sequence, and a second cross attention feature between the self-attention feature and each local feature in the local feature sequence; The parameter adjustment module is used to adjust the parameters of the video label recognition model according to the difference between the video label and video text description information of the sample video predicted based on the obtained first cross-attention features and the second cross-attention features and the corresponding sample labels. The trained video label recognition model is used to extract the video label of the target video.

14. An electronic device, characterized in that: It comprises a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods of claims 1 to 11.

15. A computer-readable storage medium, characterized in that: It includes a computer program, and when the computer program is run on an electronic device, the computer program is used to enable the electronic device to execute the steps of any method described in claims 1 to 11.

16. A computer program product, characterized in that The method comprises a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, so that the electronic device executes the steps of any one of the methods described in claims 1 to 11.

Images