Video tag recognition method and apparatus, and model training method and apparatus, device, and medium

Abstract

A video tag recognition method, executed by an electronic device and comprising: by means of a video tag recognition model, respectively encoding video frames in a video to be recognized, to construct a global feature set comprising global features of the video frames obtained by encoding, and a local feature set comprising local features of the video frames obtained by encoding; on the basis of a feature similarity between the global features in the global feature set, compressing the global feature set to obtain a sequence of global features of a preset storage quantity; and on the basis of a feature similarity between the local features in the local feature set, compressing the local feature set to obtain a sequence of local features of a preset storage quantity (S91); concatenating a global query feature and a local query feature obtained by pre-training, and then using a self-attention mechanism to extract a self-attention feature, wherein the global query feature and the local query feature are obtained by training learnable query features on the basis of a sample video, and the self-attention feature fuses key information in the global query feature and the local query feature (S92); using a cross-attention mechanism to respectively extract a first cross-attention feature between the self-attention feature and each global feature in the global feature sequence, and to respectively extract a second cross-attention feature between the self-attention feature and each local feature in the local feature sequence (S93); and on the basis of the obtained first cross-attention features and second cross-attention features, recognizing a video tag of the video to be recognized (S94).

Description

A video tag identification and model training method, device, equipment and medium

[0001] Related applications

[0002] The present application claims priority to the Chinese patent application No. 202411098346.X, filed on August 12, 2024, entitled "A video tag identification and model training method, device, equipment and medium", the contents of which are hereby incorporated by reference in their entirety. TECHNICAL FIELD

[0003] The present application relates to the technical field of video processing, in particular to a video tag identification and model training method, device, equipment and medium. BACKGROUND

[0004] Video tag identification is an important part of video content features. By automatically generating tags for massive videos through machines, different granularity video content features can be provided for the downstream content distribution link to improve the efficiency of content distribution and significantly reduce the cost of manual content review.

[0005] Taking short videos as an example, the video length is generally not more than 30 seconds, and the video frames are relatively few. Therefore, after sparse frame extraction, the visual tokens of each picture can be extracted and then spliced together to input a large language model (LLM) to identify the tags of the short video through the LLM.

[0006] However, for medium videos, long videos or video streams (such as live streaming scenarios), the video length is much longer than 30 seconds, which can reach dozens of minutes or even hours, and the total number of video frames is also much more than that of short videos. If the sparse frame extraction (such as fixed extraction of N frames at equal intervals) is still used, the extracted visual tokens cannot cover most of the content of the video, so it is difficult to guarantee the accuracy and recall rate of video tag identification.

[0007] In summary, how to efficiently and accurately identify the video tags of medium videos and long videos is an urgent problem to be solved. SUMMARY

[0008] The present application provides a video tag identification and model training method, device, equipment and medium.

[0009] In a first aspect, the present application provides a video tag identification method, which is executed by an electronic device, and the method comprises:

[0010] The video tag recognition model is used to respectively encode each video frame in the identification task video, to form a global feature set including global features of each video frame obtained by encoding, and a local feature set including local features of each video frame obtained by encoding;

[0011] The global feature set is compressed based on feature similarities between global features in the global feature set, to obtain a global feature sequence with a preset storage quantity; and the local feature set is compressed based on feature similarities between local features in the local feature set, to obtain a local feature sequence with a preset storage quantity.

[0012] After the global query feature and the local query feature obtained by pre-training are spliced, a self-attention mechanism is used to extract a self-attention feature; the global query feature and the local query feature are obtained by training a learnable query feature based on a sample video; the self-attention feature fuses key information in the global query feature and the local query feature.

[0013] A cross-attention mechanism is used to respectively extract a first cross-attention feature between the self-attention feature and each global feature in the global feature sequence, and a second cross-attention feature between the self-attention feature and each local feature in the local feature sequence.

[0014] Based on the obtained first cross-attention features and second cross-attention features, a video tag of the identification task video is identified.

[0015] In a second aspect, an embodiment of the present application provides a method for training a video tag recognition model, executed by an electronic device, and the method comprises:

[0016] The video tag recognition model to be trained is used to respectively encode each video frame in a sample video, to form a global feature set including global features of each video frame obtained by encoding, and a local feature set including local features of each video frame obtained by encoding; the global feature set is compressed based on feature similarities between global features in the global feature set, to obtain a global feature sequence with a preset storage quantity; and the local feature set is compressed based on feature similarities between local features in the local feature set, to obtain a local feature sequence with a preset storage quantity.

[0017] After the first learnable query feature and the second learnable query feature to be trained are spliced, a self-attention mechanism is used to extract a self-attention feature; the self-attention feature fuses key information in the first learnable query feature and the second learnable query feature.

[0018] extract a first cross-attention feature between the self-attention feature and each global feature in the global feature sequence and a second cross-attention feature between the self-attention feature and each local feature in the local feature sequence by using a cross-attention mechanism; and

[0019] According to differences between video labels and video text description information of the predicted sample video and corresponding sample labels obtained based on the first cross-attention features and the second cross-attention features, the video label recognition model is adjusted in parameters, and the trained video label recognition model is used to extract video labels of a recognition task video.

[0020] In a third aspect, an embodiment of the present application provides a video label recognition device, comprising:

[0021] An encoding compression module is configured to encode each video frame in a recognition task video by using a video label recognition model, to form a global feature set comprising global features of each encoded video frame and a local feature set comprising local features of each encoded video frame, to compress the global feature set based on feature similarities between global features in the global feature set to obtain a global feature sequence with a preset storage quantity, and to compress the local feature set based on feature similarities between local features in the local feature set to obtain a local feature sequence with a preset storage quantity.

[0022] A fusion module is configured to extract a self-attention feature by using a self-attention mechanism after splicing a global query feature and a local query feature obtained by pre-training, wherein the global query feature and the local query feature are obtained by training a learnable query feature based on a sample video, and the self-attention feature fuses key information in the global query feature and the local query feature.

[0023] An intersection compression module is configured to extract a first cross-attention feature between the self-attention feature and each global feature in the global feature sequence and a second cross-attention feature between the self-attention feature and each local feature in the local feature sequence by using a cross-attention mechanism.

[0024] A label recognition module is configured to recognize video labels of the recognition task video based on the first cross-attention features and the second cross-attention features.

[0025] In a fourth aspect, an embodiment of the present application provides a video label recognition device, comprising:

[0026] The first feature extraction module is configured to encode each video frame in the sample video respectively by using the video label recognition model to be trained, to form a global feature set including global features of each video frame obtained by encoding, and a local feature set including local features of each video frame obtained by encoding; to compress the global feature set based on feature similarities between global features in the global feature set, to obtain a global feature sequence with a preset storage quantity; and to compress the local feature set based on feature similarities between local features in the local feature set, to obtain a local feature sequence with a preset storage quantity.

[0027] The second feature extraction module is configured to extract a self-attention feature by using a self-attention mechanism after splicing the first learnable query feature and the second learnable query feature to be trained; the self-attention feature fuses key information in the first learnable query feature and the second learnable query feature.

[0028] The third feature extraction module is configured to extract, by using a cross-attention mechanism, a first cross-attention feature between the self-attention feature and each global feature in the global feature sequence, and a second cross-attention feature between the self-attention feature and each local feature in the local feature sequence.

[0029] The parameter adjustment module is configured to adjust parameters of the video label recognition model based on differences between video label and video text description information of the sample video predicted based on each first cross-attention feature and each second cross-attention feature, and corresponding sample label, and to train the video label recognition model to extract video labels of a recognition task video.

[0030] In a fifth aspect, an electronic device is provided, which includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, the processor executes steps of any of the above video label recognition methods.

[0031] In a sixth aspect, a computer readable storage medium is provided, which includes a computer program. When the computer program is executed on an electronic device, the computer program is configured to cause the electronic device to execute steps of any of the above video label recognition methods.

[0032] In a seventh aspect, a computer program product is provided, which includes a computer program stored in a computer readable storage medium. When a processor of an electronic device reads the computer program from the computer readable storage medium, the processor executes the computer program, so that the electronic device executes steps of any of the above video label recognition methods.

[0033] Other features and advantages of the present application will be set forth in the following specification, and in part will be apparent from the description, or can be learned by practice of the application. The objects and other advantages of the present application will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from the disclosed drawings without creative labor.

[0035] Fig. 1 is an optional schematic diagram of an application scenario in the embodiments of the present application;

[0036] Fig. 2 is a flow diagram of a content distribution product side application in the embodiments of the present application;

[0037] Fig. 3 is an implementation flow diagram of a video tag identification method provided by the embodiments of the present application;

[0038] Fig. 4 is a structural schematic diagram of a video tag identification model in the embodiments of the present application;

[0039] Fig. 5 is a structural schematic diagram of a query converter in the embodiments of the present application;

[0040] Fig. 6 is a schematic diagram of a stage-by-stage training process of a video tag identification model in the embodiments of the present application;

[0041] Fig. 7 is a schematic diagram of model parameter updating in the embodiments of the present application;

[0042] Fig. 8 is a schematic diagram of another model parameter updating in the embodiments of the present application;

[0043] Fig. 9 is an implementation flow diagram of a video tag identification method provided by the embodiments of the present application;

[0044] Fig. 10 is a schematic diagram of image blocking provided by the embodiments of the present application;

[0045] Fig. 11 is a schematic diagram of a global feature compression process in the embodiments of the present application;

[0046] Fig. 12 is a schematic diagram of a calculation process of an attention mechanism in the embodiments of the present application;

[0047] Fig. 13 is a schematic diagram of a video tag extraction process in the embodiments of the present application;

[0048] FIG. 14 is a schematic diagram of interaction logic between a terminal device and a server in an embodiment of the present application;

[0049] FIG. 15 is a schematic diagram of a component structure of a video tag identification device in an embodiment of the present application;

[0050] FIG. 16 is a schematic diagram of a component structure of a training device of a video tag identification model in an embodiment of the present application;

[0051] FIG. 17 is a schematic diagram of a hardware component structure of an electronic device applying an embodiment of the present application;

[0052] FIG. 18 is a schematic diagram of a hardware component structure of another electronic device applying an embodiment of the present application. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0054] Some concepts involved in the embodiments of the present application will be introduced below.

[0055] LLM (Large Language Model): Through training a large amount of text data, it learns the statistical rules and semantic information of language, so as to predict the next word or sentence. LLM has a wide range of applications in the field of natural language processing (NLP), such as machine translation, speech recognition, text generation, etc. LLM can use different training strategies and model structures, such as pre-training and fine-tuning. Pre-training is to train the model on a large scale in an unsupervised or self-supervised manner, so that it has the ability to understand and express language data universally. Fine-tuning is to optimize the model for specific application scenarios or tasks through supervised learning based on the pre-trained model. LLM has made remarkable achievements in language understanding, generation and translation, and with the continuous expansion of model size and the increase of training data, its performance and application range are also expanding.

[0056] MLLM (multimodal large language model): Based on LLM, it integrates media data of other modalities (such as images, videos, audio, etc.), so that the model can process information of different modalities at the same time, better understand and express semantics, and thus improve the effect and accuracy of applications. The video tag identification model in the embodiments of the present application belongs to a kind of MLLM, which integrates media data of video modality on the basis of LLM.

[0057] Global query feature and local query feature: based on sample video, corresponding learnable query is trained to obtain, which can be in the form of vector. Specifically, the global query feature is trained based on the first learnable query feature, corresponding to the global feature, and the local query feature is trained based on the second learnable query feature, corresponding to the local feature. The first learnable query feature and the second learnable query feature in this paper refer to the query vector that can be learned and adjusted through the training process.

[0058] Transformer: a network structure model suitable for processing sequence data, which adopts a full attention structure instead of traditional recurrent neural network (RNN) or convolutional neural network (CNN). Its core idea is to model the relationship between each element in the input sequence and other elements as an attention weight matrix, and calculate the representation of each element through self-attention mechanism. This structure makes the Transformer have strong ability in capturing long-distance dependencies in sequences, and thus achieves good performance in NLP tasks. The application of Transformer model in NLP field is very extensive, including machine translation, text summarization, question and answer system and other tasks. It first introduces the self-attention mechanism, and surpasses the traditional sequence-to-sequence (Seq2Seq) model in many aspects, such as long short-term memory (LSTM) and gated recurrent unit (GRU) etc. The success of Transformer also gives birth to many variant models based on self-attention mechanism, such as bidirectional encoder representations from transformers (BERT), generative pre-trained transformer (GPT) etc., which have made significant progress in NLP tasks.

[0059] Querying transformer: used for feature fusion and compression based on self-attention mechanism and cross-attention mechanism. The Querying transformer is a multi-layer decoder structure to further compress and fuse global features and local features with two learnable query features to obtain shorter visual token sequences. Specifically, a self-attention network is used in each decoder, so that the global feature query sequence and the local feature query sequence can see each other, so that the two features can interact, and then two parallel cross-attention networks are used to process the global features and the local features, respectively; in the multi-layer stacked decoder structure, between each layer, the global features and the local features are fused by self-attention and further compressed by cross-attention to obtain more compact visual token sequences.

[0060] The visual token refers to the basic unit or the smallest processable unit in the video, and is the basic unit used by the video tag identification model when processing the video, corresponding to the text token in the Transformer. The text token is a word or subword in the text.

[0061] Memory Bank (MB): a module for storing and compressing video features. In this paper, it is divided into global memory unit and local memory unit. The global memory unit is used to store and compress the global features of the video frame sequence, and the local memory unit is used to store and compress the local features of the video frame sequence. The compression method is not specifically limited in this paper.

[0062] The design idea of the embodiments of the present application will be briefly introduced as follows:

[0063] With the rapid progress of information technology, especially the booming development of artificial intelligence technology, the demand for video data analysis is growing. Video tag identification is an important part of video content features. By automatically generating tags for massive videos through machines, different granularity video content features can be provided for the downstream content distribution link (such as recommendation system, content operation, etc.), which can improve the efficiency of content distribution and greatly reduce the cost of manual content review.

[0064] In the related art, the task of text + picture / video content understanding is often solved using MLLM, in which visual features are a key modality feature. For a picture, only the image block features of the picture need to be converted into tokens (the number of tokens is controllable); a short video is generally no more than 30 seconds, and basically this framework can be followed. A commonly used method is to extract visual tokens from each picture after sparse frame extraction, and then splice them together to input into the LLM.

[0065] However, for medium videos, long videos or video streams (such as live scenes), the video length is much longer than 30 seconds, and can reach tens of minutes or even hours. If the sparse frame extraction method is still used, the visual tokens cannot cover most of the content of the video, the model cannot see the complete video content, and the accuracy and recall rate of the content label cannot be guaranteed.

[0066] In addition, if the above framework is followed, in order to make the model see enough video content, the value of N has to be set to be very large, so that the number of visual tokens reaches the order of thousands or even more, which is not feasible for offline model training and online deployment.

[0067] Therefore, the embodiments of the present application propose a video label identification and model training method, device, equipment and medium. According to the present application, when extracting visual features of an identification task video through a video label identification model, frame extraction is not required, but each video frame in the identification task video is encoded. After encoding, a global feature set composed of global features of each video frame and a local feature set composed of local features of each video frame can be obtained. On this basis, based on feature similarity, the obtained global feature set and local feature set are compressed respectively, specifically, the global features of part of the video frames are fused, and the local features of part of the video frames are fused, to obtain a global feature sequence and a local feature sequence with a preset storage quantity. Through the above operation, it can be ensured that the global feature sequence and the local feature sequence at this time can cover most of the content of the identification task video. On this basis, the key information in the global query feature and the local query feature is fused through the self-attention mechanism, and then further compressed through the cross-attention mechanism to obtain a more compact visual feature sequence.

[0068] The above method can effectively compress visual features for medium videos and long videos, reduce information loss, and improve label identification efficiency through compressed features, thereby realizing efficient and accurate identification of video labels of medium videos and long videos.

[0069] The preferred embodiments of the present application are described below in conjunction with the accompanying drawings of the specification. It should be understood that the preferred embodiments described herein are only used to explain and illustrate the present application, and are not used to limit the present application, and the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0070] As shown in FIG. 1, it is an application scenario diagram of the embodiments of the present application. The application scenario diagram includes two terminal devices 110 and one server 120.

[0071] In the embodiments of the present application, the terminal device 110 includes but is not limited to a mobile phone, a tablet computer, a notebook computer, a desktop computer, an electronic book reader, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, etc. A video tag identification related client can be installed on the terminal device, which can be a software (such as a browser, a content distribution software, a video software, etc.), a webpage, an applet, etc. The server 120 is a background server corresponding to the software or the webpage, the applet, etc., or a server specially used for video tag identification, which is not limited in the present application. The server 120 can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and basic cloud computing services such as big data and artificial intelligence platforms.

[0072] It should be noted that the video tag identification method in each embodiment of the present application can be executed by an electronic device, which can be the terminal device 110 or the server 120, i.e., the method can be executed by the terminal device 110 or the server 120 alone, or by the terminal device 110 and the server 120 together.

[0073] For example, when executed by the terminal device 110 and the server 120 together, a video tag identification related client can be installed on the terminal device 110, and an object can determine an identification task video based on the client, and send the identification task video to the server 120 through the terminal device 110. The server 120 side has pre-deployed a trained video tag identification model. Then, the server 120 uses the video tag identification method in the embodiments of the present application to identify the video tag of the identification task video based on the video tag identification model, and finally returns the identified video tag to the terminal device 110, which presents the video tag to the object through the client. The specific identification process on the server 120 side is as follows:

[0074] The video tag recognition model is used for respectively encoding each video frame in the recognition task video, to form a global feature set including global features of each video frame obtained by encoding, and a local feature set including local features of each video frame obtained by encoding; the global feature set is compressed based on feature similarities between global features in the global feature set, to obtain a global feature sequence with a preset storage quantity; the local feature set is compressed based on feature similarities between local features in the local feature set, to obtain a local feature sequence with a preset storage quantity; after the global query feature and the local query feature obtained by pre-training are spliced, a self-attention mechanism is used to extract a self-attention feature; the global query feature and the local query feature are obtained by training learnable query features based on sample videos; the self-attention feature fuses key information in the global query feature and the local query feature; a cross-attention mechanism is used to respectively extract a first cross-attention feature between the self-attention feature and each global feature in the global feature sequence, and a second cross-attention feature between the self-attention feature and each local feature in the local feature sequence; based on the obtained each first cross-attention feature and each second cross-attention feature, a video tag of the recognition task video is recognized.

[0075] The above embodiments are only simple examples, and other embodiments are also applicable to the present application, which will not be described one by one.

[0076] In an optional embodiment, the terminal device 110 and the server 120 can communicate through a communication network.

[0077] In an optional embodiment, the communication network is a wired network or a wireless network.

[0078] It should be noted that the number of terminal devices and servers is not limited in the present application, and is not limited in the present application.

[0079] In the present application, when the number of servers is multiple, the multiple servers can form a blockchain, and the servers are nodes on the blockchain; as disclosed in the present application, the video tag recognition method, wherein the video related data involved can be saved on the blockchain, for example, the video features of the recognition task video, the related features extracted for the recognition task video, the video tags, the model parameters of the video tag recognition model, etc.

[0080] In addition, the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, intelligent transportation, assisted driving, etc.

[0081] Specifically, video tagging technology uses deep learning and computer vision algorithms to analyze video content, extract key features, and label them, which allows unstructured video data to be structured and applied in various fields. For example, video tagging can be used in content classification, intelligent recommendation, video search and summary generation, etc., to improve the organization and utilization efficiency of video data. Here are some specific application scenarios:

[0082] (1) Content classification: Video platforms can use the video tagging method in this application to automatically add metadata tags to uploaded videos, helping users quickly find interesting content. For example, a video containing a football match may be tagged with "football", "sports event", "goal", etc., so that the video can be classified into the corresponding category.

[0083] (2) Intelligent recommendation system: Streaming service providers can use the video tagging method in this application to understand video content and recommend similar or related videos based on users' viewing history and preferences. For example, if a user often watches science fiction movies, the system will identify this preference and recommend more movies with "science fiction" tags to enhance the viewing experience, promote content consumption, increase user stickiness, and achieve personalized content distribution.

[0084] Referring to FIG. 2, it is a flowchart of a content distribution product side application in an embodiment of the present application. Video enters the content processing link from the content production link, and the corresponding content features are obtained through human-computer cooperation, and enters the downstream content distribution link. In the content processing process of the video, the video can be tagged based on the cooperation of machine tagging and manual tagging, i.e. adding corresponding video tags to the video. The video tagging method in this application belongs to the machine tagging link.

[0085] (3) Video search and summary generation: In a large-scale video database, users can search for video clips containing specific elements through keyword search. In addition, tag recognition can also be used to automatically generate video summaries, highlighting key scenes or events in the video, such as news highlights, movie highlights, etc.

[0086] In addition to the above scenarios, video tagging can also be applied to:

[0087] (4) Ad targeting: Media companies can use video tags to accurately place ads, ensuring that ads are related to video content and improving ad effectiveness. For example, sports brand ads can be played in sports event videos.

[0088] (V) Copyright protection and content review: Using the video tag identification method in this application, content platforms can help detect and prevent copyright infringement, such as identifying unauthorized use of copyrighted music or images. At the same time, it is also used to review video content, automatically filter inappropriate or illegal content, such as violence, gore, etc. These applications not only improve the efficiency of video data management and distribution, but also enhance the object experience and platform security.

[0089] It should be noted that the above-mentioned application scenarios are only simple examples, and in addition to this, other application scenarios are also applicable to the embodiments of the present application, which will not be repeated here.

[0090] In addition, it needs to be emphasized that in the specific embodiments of the present application, the video-related data involved, such as the identification task videos that need to be identified by the above-mentioned tag identification, the viewing history and preferences of the object, etc. When the above embodiments of the present application are applied to specific products or technologies, the permission or consent of the object needs to be obtained, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of the country and region.

[0091] The video tag identification method provided by the exemplary embodiments of the present application will be described below in conjunction with the above-described application scenarios and reference to the accompanying drawings. It should be noted that the above-mentioned application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this respect.

[0092] The scheme provided by the embodiments of the present application mainly involves the application of artificial intelligence in video tag identification. When identifying video tags, a deep learning-based artificial neural network model needs to be used, i.e. the video tag identification model obtained by the embodiments of the present application in combination with machine learning and video detection architecture-related methods. The query converter and LLM in the video tag identification model are trained based on machine learning methods for domain adaptation, and then in actual scenarios, the purpose of identifying the identification task video can be achieved through the video tag identification model.

[0093] Specifically, the video tag identification process in the embodiments of the present application can be divided into two parts, including a training part and an application part. The training part involves the technical field of machine learning, and in the training part, the embodiments of the present application adopt a phased training method. First, in the first training phase, the video text description information of the sample video is used as the sample label to train the query converter in the video tag identification model. In the second training phase, the video tag of the sample video is used as the sample label to jointly train the query converter and the LLM in the video tag identification model. The application part is used to use the trained video tag identification model to identify the labels of the identification task video and output the identified label list (containing one or more video tags). In addition, it should be noted that the artificial neural network model in the embodiments of the present application can be online training or offline training, which is not limited here. In this paper, offline training is taken as an example for illustration.

[0094] First, the training process of the video tag identification model of the embodiments of the present application is described below:

[0095] Before training the video tag identification model, a video-related training sample set needs to be constructed. The training sample set in the embodiments of the present application contains multiple training samples, and each training sample includes a sample video and its corresponding sample label.

[0096] It should be noted that the sample video in the model training stage and the identification task video in the model application stage in the embodiments of the present application can be any video, which can be various types of videos on a video platform, including live videos and non-live videos. According to the length of time, the non-live video can be any fixed-length short video, medium video and long video, and can also be any non-fixed-length video stream; according to the type, the non-live video can be a film and television video, an information video, a conference video, a teaching video and a life sharing video, etc.

[0097] Optionally, the sample label in the embodiments of the present application contains two parts of the video tag and the video text description information of the sample video.

[0098] Among them, the video tag is a keyword or phrase used to describe the content, features or context of the video, which can help the object quickly understand the theme and type of the video. A video can have one or more tags. Specifically, if a video has multiple tags, the multiple tags can cover different aspects.

[0099] For example, a tutorial video about baking a cake, its tags might include "baking", "cake", "recipe", "tutorial", and "food". These tags not only describe the specific content of the video (baking a cake), but also indicate its function (tutorial) and belonging category (food); for another example, a documentary about outdoor adventure, its tags might include "outdoor adventure", "natural scenery", "wildlife", "hiking", "photography skills", "survival skills", "travel guide", and "human geography"; and so on.

[0100] Among them, the video text description information can refer to any text information related to the video, which can provide additional context or directly describe the media content. In the embodiments of the present application, the text modalities of the video text description information include but are not limited to the following parts or all:

[0101] Video title, description text, key text extracted by Optical Character Recognition (OCR), subtitles or transcripts generated by Automatic Speech Recognition (ASR), comments and forum discussions, metadata information.

[0102] Among them, the description text refers to the detailed description below the video or on the video page, which usually contains the theme, purpose, key points, characters, locations, etc. of the video. The description text can be provided by the video producer, or it can be edited later or summarized by the audience's comments. Metadata information includes creator information, upload date, etc. These data, although not directly describing the video content, are crucial for the organization and retrieval of the video.

[0103] In addition, through OCR technology, the present application can identify and extract text information from images in the video, such as text on posters, text displayed on screens, book covers, menus, road signs, etc. These texts can provide additional information about the video scene, helping to understand the video content. Through ASR technology, the speech in the video can be converted into text, generating subtitles or voice transcripts. These texts not only include dialogue content, but also include narration, commentary, and description of background sounds, providing convenience for the hearing impaired, and also helping the video search engine to understand and index the video content.

[0104] In the embodiments of the present application, the information of the above-mentioned text modalities can be summarized as [video_txt], representing the video text description information of a video, which can greatly enrich the understanding and analysis of the video.

[0105] In an optional embodiment, the sample label can be obtained by manual labeling. Considering that manual labeling can be affected by subjective consciousness of the labeler, to reduce labeling errors, multiple manual labeling results can be obtained, and then the labeling results of multiple persons are integrated to obtain the final sample label.

[0106] In addition, the sample video can also be labeled according to historical labeling information (such as historical labels, titles, etc.) of the sample video, or video labels, video text description information, etc. detected by other trained models (such as LLM).

[0107] In the embodiments of the present application, after obtaining the training sample set, the video label model to be trained can be trained by cyclic iteration based on the training sample set to obtain a trained video label recognition model. The training process is briefly described as follows:

[0108] Referring to FIG. 3, an embodiment of a video label recognition method provided by the present application is shown, and the method is taken as an example with a server as the execution subject. The specific implementation process of the method is as follows S31-S34:

[0109] S31: Each video frame in the sample video is encoded by the video label recognition model to be trained, to form a global feature set including global features of each video frame obtained by encoding, and a local feature set including local features of each video frame obtained by encoding; the global feature set is compressed based on feature similarity between global features in the global feature set, to obtain a global feature sequence with a preset storage number; the local feature set is compressed based on feature similarity between local features in the local feature set, to obtain a local feature sequence with a preset storage number.

[0110] The video frame can be an image of the corresponding frame in the video, so the video frame can also be referred to as a video frame image, simply referred to as an image. For example, assuming that the frame rate of a video is 30 and the time length is 1 second, the video has a total of 30 frames of images, which are all video frames of the video.

[0111] The video tag identification model in the embodiments of the present application belongs to a multi-modal model. Taking the MLLM as an example, the MLLM needs a bridge connecting the non-text token and the LLM, which converts the input from different modalities into tokens that can be understood by the LLM. In the present application, the query converter realizes this function. Optionally, the video tag identification model in the embodiments of the present application is a retrieval-enhanced video tag identification model, which includes a visual encoder, a query converter and a large language model. The visual encoder is a component of the video tag identification model and is used to extract global features and local features of a video frame image. It can convert an input video frame image into visual features (column vectors). Specifically, the visual encoder can select a vision transformer (ViT), a convolutional neural network (CNN), a residual network (ResNet), an Inception network, a Swin transformer, a MobileNet, an EfficientNet or the like. The query converter (i.e., Querying transformer) is used for feature fusion and compression based on a self-attention mechanism and a cross-attention mechanism. In addition, the query converter includes a fully connected layer for realizing dimension alignment of visual features and text features. Finally, the video tag is generated by asking the large language model.

[0112] In the process, each local feature corresponds to an image block in the video frame. The extraction of the global feature and the local feature can be realized by the visual encoder in the video tag identification model. FIG. 4 is a structural schematic diagram of a video tag identification model in an embodiment of the present application. As shown in the visual encoder 401 in FIG. 4, the visual encoder is used to convert an input video frame image into visual features (column vectors). Specifically, the visual encoder can select a vision transformer (ViT), a convolutional neural network (CNN), or other encoding models such as a residual network (Residual Networks, ResNet), an Inception network (Inception Networks), a Swin transformer (Swin Transformers), a MobileNet, an EfficientNet or the like.

[0113] Specifically, regardless of the encoding model chosen, the extracted features can be divided into two categories: global features and local features.

[0114] Global features refer to the overall representation of a video frame image, typically containing information such as the macrostructure, color distribution, and overall style of the image. Global features are used to help the model understand the overall semantics of the image, such as identifying whether a picture is a landscape, a person, or an abstract art.

[0115] Local features refer to the features of local regions in a video frame image, such as the shape, texture, and location information of objects such as faces, vehicles, and trees. Local features are used to help the model identify and locate specific objects in the image. Image blocks refer to smaller, fixed-size regions in an image. In this application, a video frame image can be pre-divided into N image blocks, and local features are extracted at the image block level, with each image block corresponding to a local feature reflecting the details of the local region of the image.

[0116] Specifically, for ViT, the input image is first divided into multiple image blocks of fixed size, and each patch is flattened and linearly projected into an embedding vector, i.e., patch tokens. In addition, ViT introduces a special classification token (CLS token), which is at the front of the sequence and is input into the Transformer encoder together with the patch tokens. The CLS token does not correspond to any specific image patch, but interacts with all patch tokens through self-attention mechanisms, gradually gathering information about the entire image, and its embedding vector (emb) ultimately represents the global feature of the image, denoted as f glob , size 1*d. At the same time, the embedding vector (emb) of each patch token after passing through the Transformer encoder retains the local features of the corresponding image region, denoted as f loc,i , size N*d, N is the number of patches, i=1,2,…,N.

[0117] For CNN, the average pooling operation is usually located at the end of the network, which compresses each channel of the feature maps produced by the previous convolutional layer into a single value. The output vector after this operation aggregates the information of the entire input image, and is therefore considered as a global feature. The feature maps before average pooling retain the spatial structure of the image, with each point's value corresponding to the features of a local region of the image, so these feature maps correspond to local features.

[0118] The above is just an example of ViT and CNN to briefly describe the process of extracting global features and local features of video frame images. For other networks, the extraction method is related to the network structure, and this paper will not be repeated.

[0119] In the embodiments of the present application, the global features and local features of the video frame images are extracted at the same time, which can enable the model to understand the macro context of the image and capture detailed information, thereby achieving better performance in subsequent tasks.

[0120] Considering that ViT has good effect in processing image feature extraction, and in order to simplify the subsequent training process, a trained ViT is taken as an example to illustrate the visual feature encoder in the following, and its parameters are not updated in the subsequent training process. In addition, it should be noted that the visual encoder in this paper is an example of ViT, and other network structures can also be used, and this paper does not make specific limitations.

[0121] In addition, in order to solve the problem that the recognition accuracy and recall rate of video tags cannot be guaranteed due to sparse frame extraction for medium, long video or video stream (such as live streaming scene) in the related art, a video stream tag generation method based on memory enhanced multi-modal large model is proposed. Two memory units (i.e. MB module) are introduced to store and compress the global features and local features of the video frame sequence, as shown in the global memory unit 402 and the local memory unit 403 in FIG. 4. Therefore, an optional implementation is as follows:

[0122] Each time a video frame is encoded, the global features of the video frame are stored in the global memory unit, and the local features of the video frame are stored in the local memory unit; when the number of global features stored in the global memory unit reaches the preset storage number, based on the feature similarity between the global features of multiple groups of video frames in the global features stored in the global memory unit and the global features to be stored, the global features of at least two video frames are adjusted to adjust the global features stored in the global memory unit, so that the number of global features is less than or equal to the preset storage number; and for each image block, when the number of corresponding local features stored in the local memory unit reaches the preset storage number, based on the feature similarity between the local features in the local memory unit, the corresponding local features of at least two video frames in the local memory unit are fused, so that the number of local features stored in the local memory unit is less than or equal to the preset storage number.

[0123] In the embodiments of the present application, the effect of video frame sequence compression is realized by feature fusion of global features or local features. Feature fusion refers to the operation of merging and processing the global features of part of the video frames, and merging and processing the local features of part of the video frames. In the present application, when the global feature set is compressed based on the feature similarity between the global features in the global feature set, the global feature fusion will be performed; when the local feature set is compressed based on the feature similarity between the local features in the local feature set, the local feature fusion will be performed. The feature fusion mode includes averaging and pooling the target features of the most similar adjacent video frames based on the feature similarity, clustering the target features to be stored with the target features already stored in the memory unit and averaging and pooling the target features of each video frame that has been clustered, and averaging and pooling the target features of the longest continuous video frames, to obtain a global feature sequence and a local feature sequence with a preset storage quantity.

[0124] Among them, the feature fusion mode can be compression of adjacent video frames, clustering of multiple video frames, etc. For details, please refer to the relevant description of the model application stage below, such as compression mode one, compression mode two, compression mode three, etc.

[0125] Based on the feature similarity between the global features in the global feature set, the global feature set is compressed to obtain a global feature sequence with a preset storage quantity. The specific compression mode is: after encoding a video frame, the global features of the video frame are stored in the global memory unit; when the number of global features stored in the global memory unit reaches the preset storage quantity, based on the feature similarity between the global features stored in the global memory unit and the global features of the multiple groups of video frames to be stored, the compression can be performed in the following ways: one is to average and pool the global features of the most similar adjacent video frames in the global memory unit based on the feature similarity; two is to cluster the global features to be stored with the global features already stored in the global memory unit based on the feature similarity, and to average and pool the global features of each video frame that has been clustered; three is to average and pool the global features of the longest continuous video frames. This part will be described in detail in the subsequent content.

[0126] The local feature set is compressed based on the feature similarity between the local features in the local feature set, and a local feature sequence with a preset storage quantity is obtained. The specific compression method is as follows: for each encoded video frame, the local features of the video frame are stored in a local memory unit. For each image block, when the number of corresponding local features stored in the local memory unit reaches the preset storage quantity, based on the feature similarity between the local features stored in the local memory unit and the local features of the to-be-stored video frames at the image block, the compression can be performed in the following ways: first, based on the feature similarity, the local features of the most similar adjacent video frames at the image block in the local memory unit are averaged and pooled; second, based on the feature similarity, the local features of the to-be-stored video frames at the image block are clustered with the local features of the stored video frames at the image block in the local memory unit, and the local features of the clustered video frames at the image block are averaged and pooled; third, the local features of the video frames with the longest storage duration at the image block are averaged and pooled. This part of the content will be described in detail in the subsequent content.

[0127] The calculation method of the feature similarity is to calculate the similarity between the global features or the local features by using the cosine similarity, and the calculation formula of the cosine similarity is: where A and B represent two feature vectors, · represents the dot product of the vectors, and ||A|| and ||B|| represent the modulus of vectors A and B, respectively.

[0128] Then, the following steps S32 and S33 can be performed based on the query converter in the video label recognition model, such as the query converter 404 shown in FIG. 4:

[0129] S32: After splicing the first learnable query feature and the second learnable query feature, the self-attention mechanism is used to extract the self-attention feature; the self-attention feature integrates the key information in the first learnable query feature and the second learnable query feature.

[0130] The first learnable query feature is a learnable global feature query sequence for the global feature, which can be in the form of a vector and can be denoted as Q1. The second learnable query feature is a learnable local feature query sequence for the local feature, which can also be in the form of a vector and can be denoted as Q2. The first learnable query feature and the second learnable query feature can be understood as model parameters in the query converter.

[0131] The Q1 and Q2 can be randomly initialized as a learnable query before model training, and are learned and optimized during model training, wherein the Q1 represents attention to global features, and the Q2 represents attention to local features, and a splicing result of the Q1 and the Q2 is taken as an input of the self-attention mechanism, which allows the global feature query sequence to capture local details and allows the local feature query sequence to obtain global context, and finally generates a fused feature representation that is both detailed and broad in vision, thereby improving the understanding ability of the model to complex visual patterns, and the model learns how to effectively detect and locate global features and local features in the image without any prior knowledge, and the adaptability and expressiveness of the model are enhanced.

[0132] S33: A cross-attention mechanism is used to extract a first cross-attention feature between each global feature in the global feature sequence and the self-attention feature, and a second cross-attention feature between each local feature in the local feature sequence.

[0133] In the above S32, the first learnable query feature and the second learnable query feature are spliced, and after splicing, the key information in the first learnable query feature and the second learnable query feature can be fused through the self-attention mechanism, and on this basis, the cross-attention mechanism is used to extract a first cross-attention feature between each global feature in the global feature sequence and the self-attention feature, and a second cross-attention feature between each local feature in the local feature sequence.

[0134] Optionally, the self-attention feature obtained in the above S32 can be first split into two parts, denoted as a first self-attention feature and a second self-attention feature, and the splitting process is equivalent to the inverse process of splicing in S32. For example, if the first learnable query feature and the second learnable query feature are spliced in the row vector in the above S32, the splitting can be performed in the row vector, and so on.

[0135] On this basis, the cross-attention mechanism can be used to extract a first cross-attention feature between each global feature in the global feature sequence and the first self-attention feature, and a second cross-attention feature between each local feature in the local feature sequence and the second self-attention feature.

[0136] Optionally, the query converter includes a plurality of stacked decoders. For the first decoder, the input features include the global feature sequence and the local feature sequence output by the memory unit. For other decoders, the input features include the first cross-attention features and the second cross-attention features output by the previous decoder.

[0137] During the training process of the cross-attention mechanism, the propagation and optimization of the gradient are the key to ensure the learning effect of the model. After the first cross-attention feature and the second cross-attention feature are calculated, the error between the prediction result and the true label is calculated through the loss function. Then, the gradient of the error is propagated from the loss function to each parameter in the cross-attention mechanism, including the weight matrix, etc. (here, the matrix is not explicitly written in the original text, but can be inferred from the context to refer to the relevant weight matrix in the self-attention and cross-attention network). To optimize these parameters, we can use the Adam optimization algorithm. The Adam algorithm combines the advantages of AdaGrad and RMSProp algorithms, and can adaptively adjust the learning rate of each parameter. It dynamically adjusts the learning rate by calculating the first moment estimate (mean) and second moment estimate (variance) of the gradient, so that the model can converge faster during training and avoid falling into a local optimal solution. Specifically, the Adam algorithm updates the gradient mean and variance estimate at each iteration, and updates the parameters according to these estimates, as follows: m t = β1m t-1 + (1-β1)g t

[0138] where m t is the first moment estimate of the gradient (i.e., the mean of the gradient), which records the average of the gradient in the past iterations; v t is the second moment estimate of the gradient (i.e., the variance of the gradient), which reflects the variation of the gradient; β1 and β2 are the decay rates, usually β1 takes the value of 0.9 and β2 takes the value of 0.999, which are used to control the update speed of the first moment and second moment estimates; g t is the gradient of the current iteration, which represents the direction and size of the parameter change in the current step; and are the corrected first moment and second moment estimates, since m t and v t will be biased to 0 at the beginning of training, they need to be corrected; θ t is the parameter of the current iteration, i.e., the model parameter to be updated; α is the learning rate, which controls the step size of parameter update; ε is a very small constant, usually takes the value of 10 -8 , which is used to avoid division by zero and ensure the stability of the calculation.

[0139] As shown in FIG. 5, it is a structure schematic diagram of a query converter in an embodiment of the present application. The content structure of a Decoder is represented above the dashed line in FIG. 5, and a multi-layer stacked Decoder structure is represented below the dashed line.

[0140] Specifically, one Decoder includes one self-attention network, such as multi-head self-attention 501 in FIG. 5, two parallel cross-attention networks, which can be denoted as a first cross-attention network and a second cross-attention network, such as first multi-head cross-attention 502 (i.e., the first cross-attention network) and second multi-head cross-attention 503 (i.e., the second cross-attention network) in FIG. 5, and a Feed-Forward Network (FFN) after each cross-attention network, such as first Feed-Forward Network 504 and second Feed-Forward Network 505 in FIG. 5. Among them, the cross-attention network in each Decoder is divided into two parallel ones, where the first cross-attention network corresponds to processing global features, and the second cross-attention network corresponds to processing local features.

[0141] Wherein, the FFN can perform nonlinear transformation on the features output by the cross-attention network through the linear layer and the nonlinear activation function inside it, enhance the feature expression ability, and make it more suitable for subsequent task processing. Such conversion helps the model to learn more complex feature relationships and patterns.

[0142] It should be noted that the cross-attention feature in the embodiments of the present application can refer to the output of the cross-attention network, and on this basis, the output of the cross-attention network can be nonlinearly changed through the FFN. In order to simply describe and facilitate understanding, the features of the output of the cross-attention network nonlinearly changed through the FFN can still be referred to as cross-attention features.

[0143] It should be noted that each Decoder in the embodiments of the present application corresponds to a set of Q1 and Q2 as the input of the self-attention network in the Decoder, and the input of the cross-attention network in the Decoder is determined based on the above-mentioned manner. For example, the input of 502 in Decoder1 is each global feature output by the global memory unit, and the input of 503 in Decoder1 is each local feature output by the local memory unit; the input of 502 in Decoder2 is each first cross-attention feature output by 504 in Decoder1, and the input of 503 in Decoder2 is each second cross-attention feature output by 505 in Decoder1; the input of 502 in Decoder3 is each first cross-attention feature output by 504 in Decoder2, and the input of 503 in Decoder3 is each second cross-attention feature output by 505 in Decoder2; the subsequent Decoder is the same, and the same applies.

[0144] S34: According to the obtained first cross-attention features and the second cross-attention features, the difference between the predicted video label and video text description information of the sample video and the corresponding sample label is predicted, and the video label recognition model is adjusted in parameters.

[0145] The trained video label recognition model is used to extract the video label of the recognition task video.

[0146] In the case where the query converter contains multiple decoders, the first cross-attention features and the second cross-attention features output by the last decoder in S34 are specifically referred to, and then the first cross-attention features and the second cross-attention features output by the subsequent decoder can be used to predict the video label and the video text description information of the sample video in combination with the LLM.

[0147] The prediction process is specifically described as follows:

[0148] For the MLLM, the LLM receives two kinds of inputs, including visual tokens and text tokens. The visual tokens are obtained based on the visual encoder processing, and in order to align the input of the LLM, a query converter is connected behind the visual tokens, which is used to map the visual tokens to the input dimension of the LLM and convert them into tokens suitable for the LLM, i.e., the visual features in this paper, denoted as [video_emb]. For the text tokens, they include video titles, description texts, OCR, ASR, etc., denoted as [video_txt]. The text token refers to a word or subword in the text, which is the basic unit used by the model when processing the text. In the video label recognition model of the present application, the text token includes video titles, description texts, key texts extracted by optical character recognition (OCR), subtitles or transcripts generated by automatic speech recognition (ASR), comments and forum discussions, metadata information, etc., denoted as [video_txt], which is provided as input to the large language model together with the visual token to generate the video label.

[0149] Specifically, the first cross-attention features and the second cross-attention features output by the query converter can be used as visual tokens (belonging to non-text tokens), and combined with the text tokens of the video, the large language model 406 can be interrogated to obtain the corresponding video label. That is, the query converter in this paper serves as a bridge connecting non-text tokens and LLM, which is used to convert inputs from different modalities into tokens that can be understood by the LLM. The compressed and fused visual tokens are spliced with the text tokens of the text prompt to input the LLM for training.

[0150] In addition, considering that the dimensions of the visual tokens and the text tokens output by the query transformer are not necessarily the same, the dimension alignment of the features of different modalities can also be performed through a fully connected layer (FC). The FC can be divided into a part of the query transformer and connected to the output of the last Decoder in the query transformer. Of course, the FC can also be regarded as an independent part between the query transformer and the LLM, as shown in the fully connected layer 405 in FIG. 4. Regardless of the form, the FC is used to align the dimensions of the visual features and the dimensions of the text features.

[0151] The following takes the query transformer containing the fully connected layer as an example:

[0152] In the embodiments of the present application, in order to ensure that the visual tokens extracted by the query transformer can more accurately represent the features of the video, and then more accurately extract the video label, the training process of the video label identification model in the present application can include two main training stages, including the pre-training of the query transformer in the first training stage, and the instruction fine-tuning training of the query transformer and the LLM in the second training stage.

[0153] Referring to FIG. 6, which is a schematic diagram of a phased training process of a video label identification model in an embodiment of the present application. In the two training stages, the input of the LLM is different, the output is different, and the adjusted parameters are also different. In the first training stage, the input of the LLM is the visual features obtained based on the query transformer (which can be denoted as [video_emb]) and the first text instruction, and the output is the video text description information, and only the parameters of the query transformer are adjusted; in the second training stage, the input of the LLM is the visual features obtained based on the query transformer and the text features (which can be denoted as [video_txt]) and the second text instruction, and the output is the video label, and the parameters of the query transformer and the LLM are adjusted. The two training stages are described in detail as follows:

[0154] (I) First training stage: pre-training of the query transformer.

[0155] This stage only trains the Querying transformer, and the parameters of the visual encoder and the LLM are all fixed (i.e., frozen), without LoRA. The main goal is to align the visual tokens to the LLM.

[0156] The training data of this stage adopts a pair of <sample video, video description text information>, for example, a pair of <short video, video title> in an actual business scenario, to train the model aiming at generating a brief description of the video content. An optional training method is as follows:

[0157] In the first training stage, after splicing the first cross-attention features and the second cross-attention features, the visual features are converted to be aligned with the text feature dimension of the sample video; then, the first text instruction is taken as the question and answer prompt information, and the visual features are input into the large language model to generate the video text description information of the sample video; then, according to the difference between the predicted video text description information and the video text description information in the sample label, the parameters of the query converter are updated.

[0158] The text feature of the sample video refers to the coding of the video text description information of the sample video. For example, if the video text description information is a video title, the text feature is obtained by coding the video title. If the video text description information includes a video title, extracted key text, etc., the text feature is obtained by splicing the coding of these video text description information. The text encoder is a component used to convert text information into a vector representation that can be processed by a computer for subsequent fusion and processing with visual features. In this application, the text feature of the sample video is obtained by coding the video text description information of the sample video using a text encoder. Specifically, it can be generated by any text encoder, such as a Word2Vec network, BERT, and RoBERTa (Robustly Optimized BERT Pretraining Approach). During model training, the network parameters of this part do not need to be updated. Like the visual encoder, the network parameters of this part do not need to be updated during model training. Different text encoders have different bases:

[0159] Word2Vec network: It is a simple and effective text encoding method that maps words to a low-dimensional vector space, capturing the semantic relationships between words. If the video text description information is relatively simple and mainly focuses on semantic information at the word level, Word2Vec is a good choice, as it is computationally efficient and fast to train.

[0160] BERT: It is a pre-training model based on bidirectional Transformer, which can capture the context information of the text and has strong semantic understanding ability. If the video text description information is complex and contains rich semantic and contextual relationships, BERT can better extract text features and improve the performance of the model.

[0161] RoBERTa: It is an improved model based on BERT, which uses larger batch size and longer training time in the training process to learn more rich language representations. For large-scale video text description information, RoBERTa may perform better.

[0162] The first text instruction is used to prompt the large language model to output the video text description information of the sample video. Therefore, the first text instruction can adopt one of the following sentences:

[0163] "Describe the following video briefly.", "Provide a brief description of the given video segment.", "Explain the provided video segment concisely.", "Summarize the visual content of the following video.", "Provide a brief and clear explanation of the next video segment.", "Briefly explain the meaning of the provided video.", "Briefly describe the key features of the segment.", "Briefly describe the content of the video shown.", "Provide a clear and concise summary of the following video.", "Write an informative summary about the following video segment.", "Present the provided video in a concise narrative manner.", and the like.

[0164] It should be noted that the above listed sentences are only simple examples, and other sentences that can prompt the large language model to output the video text description information of the sample video are also applicable to the embodiments of the present application, and will not be repeated here.

[0165] Among them, the LLM can select an open-source Chinese LLM, such as chinese-llama, BigScience Large Open-science Open-access Multilingual (BLOOM), etc. The input prompt word template is:

[0166] "###Human: [video_emb][alignment instruction]###Assistant:"

[0167] Note that in the first training stage, [video_txt] is not input, and only [video_emb] is provided to the LLM to guide the Querying transformer to better train.

[0168] Among them, "###Human:" and "###Assistant:" are common templates for training, which are used to let the machine understand that "Human" is followed by instructions, and "Assistant" is used to start answering questions. The alignment instruction is a guide or command given to the LLM during training, telling the LLM what output to generate when given input. This alignment instruction can be diversified, such as using at least one of the sentences listed in the first text instruction above.

[0169] In the first training stage, when adjusting the model parameters, only the query transformer is adjusted, and the adjustment method is as follows:

[0170] According to the difference between the predicted video text description information and the video text description information in the sample label, a loss function is constructed, and the parameter of the query transformer is updated according to the loss function.

[0171] Wherein, the loss function can be Cross-Entropy Loss, Cosine Similarity Loss, BLEU Score (Bilingual Evaluation Understudy) and the like, which is not specifically limited herein. In combination with the output rules of the large language model, the design and optimization process of the loss function will consider the probability distribution of the generated sequence (i.e. the predicted video text description information) and the difference between the generated sequence and the target sequence (i.e. the video text description information in the sample label). By adjusting the model parameters, the consistency of the generated text description and the real label description can be gradually improved, thereby improving the accuracy of video content understanding and the quality of natural language description.

[0172] After constructing the loss function, the gradient can be calculated by the back propagation algorithm, and then the parameters of the query transformer are adjusted to minimize the loss.

[0173] (II) Second training phase: instruction fine-tuning training of querying transformer and LLM.

[0174] In this phase, only the parameters of the visual encoder are fixed, and the Querying transformer and LLM are trained.

[0175] Wherein, the training of LLM is divided into two ways, one is to directly adjust the parameters of LLM, and the other is to fix the parameters of LLM and add a low-rank adaptive matrix (Low-Rank Adaptation, LoRA) to train the parameters of LoRA.

[0176] In the case of adding LoRA, the parameters of the large language model can be updated in the following way:

[0177] Fix the original parameters of the large language model, and add a low-rank adaptive matrix to the large language model; update the parameters of the low-rank adaptive matrix to update the parameters of the large language model.

[0178] Wherein, LoRA is a technology for fine-tuning pre-trained models, its main goal is to adapt to specific tasks by introducing additional low-rank matrices without modifying the original model parameters, thereby reducing the resources and time required for fine-tuning. The specific implementation details are as follows:

[0179] Low-rank adaptive matrix is a technique for fine-tuning pre-trained models, which aims to adapt to specific tasks by introducing additional low-rank matrices without modifying the original model parameters, thereby reducing the resources and time required for fine-tuning. In large language models, assuming the weight matrix of a certain linear layer is W, two additional small matrices A and B are introduced in LoRA, by updating the parameters of matrices A and B, the behavior of the model is effectively adjusted to adapt to the specific task requirements without changing the original weight matrix W of the large language model.

[0180] Assuming the weight matrix of a certain linear layer in a large language model is where d in is the input dimension, d out is the output dimension. In LoRA, we introduce two additional small matrices and where r is the rank of the low-rank matrix, usually r << d in ,d out These two matrices A and B are the left and right matrices after low-rank decomposition.

[0181] During forward propagation, the output of the original linear layer is y original = Wx, where x is the input vector. After introducing LoRA, the output is y = Wx + BAx, that is, the result of BAx is added to the output of the original linear layer to form the final output. In this way, without changing the original weight matrix W of the large language model, by updating the parameters of matrices A and B, the behavior of the model can be effectively adjusted to adapt to the specific task requirements.

[0182] During training, only the parameters of matrices A and B need to be updated, while the original parameters W of the large language model remain unchanged. Since the dimensions of A and B are relatively small, the number of parameters to be updated is greatly reduced, thereby reducing the resources and time required for fine-tuning. At the same time, by adjusting the value of r, the expressive power and computational complexity of LoRA can be controlled.

[0183] In the above implementation, LoRA greatly reduces the memory and computing resources required for fine-tuning by adjusting only a small number of newly added parameters instead of all parameters of the entire large language model, while maintaining the performance of the original model and improving the efficiency of model training. Compared with the fine-tuning method of the complete model, the LoRA model occupies less storage space, also saving storage resources.

[0184] The training data at this stage is constructed based on the sample video (such as medium video, long video or video stream) in the actual business scenario, which is marked as <sample video, video label>, for example, <video stream i (t), the label list i (t) , where i is an index representing the i-th sample video in the training sample set, the superscript t represents the time t, and the label list i (t) , where i is an index representing the i-th sample video in the training sample set, the superscript t represents the time t, and the label list i , where i is an index representing the i-th sample video in the training sample set, the superscript t represents the time t, and the label list

[0185] It should be noted that the same video stream can be audited multiple times at different times to obtain different label results.

[0186] In this stage, the generated label list text is targeted for instruction fine-tuning. An optional training method is as follows:

[0187] In the second training stage, the first cross-attention feature and the second cross-attention feature are spliced and converted into visual features that are aligned with the text feature dimension of the sample video. The text feature and the second text instruction are used as question and answer prompts, and are combined with the visual features to input the large language model to generate the video label of the sample video. Then, in the second training stage, the parameters of the query converter and the large language model are updated according to the difference between the predicted video label and the video label in the sample label.

[0188] , where the text feature of the sample video is as described above and will not be repeated here. The second text instruction is used to prompt the large language model to output the video label of the sample video. For example, the second text instruction can use one of the following sentences:

[0189] "What tags can be used to summarize the above video?", "Please output some key content tags based on the content of the above video.", "Generate the main content tags based on the content of the above video.", etc.

[0190] It should be noted that the above-mentioned sentences are only simple examples, and other sentences that can prompt the large language model to output the video label of the sample video are also applicable to the embodiments of the present application, and will not be repeated here.

[0191] , where LLM can select an open-source Chinese LLM such as chinese-llama, BLOOM, etc. The input prompt template is:

[0192] "###Human:[video_emb][video_txt][instruct]###Assistant:"

[0193] , where [video_emb], [video_txt], and [instruct] are the visual token of the video, the text token of the video, and the token of the instruction text, respectively.

[0194] In the second training phase, [video_txt] and [video_emb] are input, and the LLM is provided with visual information and text information to train the Querying transformer and the LLM in combination with the instruct instruction.

[0195] Among them, "###Human:" and "###Assistant:" are two commonly used templates for training, which are used to let the machine understand that Human refers to the instruction, and Assistant is the beginning of answering the question. Similar to the alignment instruction in the first training phase, it is also a guide or command given to the LLM during training, telling the LLM what output it should generate when given input. The instruct instruction can also be designed with synonymous instructions to improve training effectiveness, such as using at least one of the sentences listed in the second text instruction above.

[0196] In the second training phase, when adjusting the model parameters, both the Querying transformer and the LLM need to be adjusted, and the adjustment method is as follows:

[0197] According to the difference between the predicted video tags and the video tags in the sample tags, a loss function is constructed; without LoRA, the Querying transformer and the LLM are updated according to the loss function. Or, with LoRA, the Querying transformer and LoRA are updated according to the loss function.

[0198] Among them, the type and construction method of the loss function are similar to the first training phase, and will not be repeated here.

[0199] After constructing the loss function, the gradient can be calculated through the backpropagation algorithm, and then the parameters of the Querying transformer are adjusted to minimize the loss.

[0200] It should be noted that the above is an example of the Querying transformer containing FC, if FC is independent of the Querying transformer, then both the first training phase and the second training phase also update the parameters of FC.

[0201] And this paper mentioned the update of the LLM parameters, there are two ways of not adding LoRA and adding LoRA, assuming that FC is independent of the Querying transformer, then the following will briefly explain the parameter update logic during model training combined with the attached drawings:

[0202] As shown in FIG. 7, which is a schematic diagram of model parameter updating in an embodiment of the present application. FIG. 7 lists a more specific network structure of a video tag identification model, wherein the visual encoder is used to encode each video frame in the input video stream in turn to generate global features and local features of each video frame, wherein the global features MB are global memory units in the present disclosure, used to store and compress global features, and the local features MB are local memory units in the present disclosure, used to store and compress local features.

[0203] When it is necessary to identify a video tag, the global feature sequence in the global feature MB is input into the multi-head cross attention 1 in the right query converter, and the local feature sequence in the local feature MB is input into the multi-head cross attention 2 in the right query converter, the query converter contains N stacked Decoders, after processing by the self-attention mechanism and the cross-attention mechanism, the output is dimensionally aligned by FC, and then combined with the prompt to input the LLM for prediction. The prompt is composed of video description text information and text instructions (such as the first text instruction and the second text instruction).

[0204] In FIG. 7, different patterns are used to represent whether different modules in the model need to be updated. The snowflake pattern represents parameter freezing and no update, and the flame pattern represents parameter updating. Therefore, FIG. 7 indicates that the visual encoder is not updated during the model training process, while the query converter, FC and LLM are updated. FC is used to dimensionally align the features output by the query converter to adapt to the input of the LLM, and in the manner without LoRA, the query converter and LLM are updated together.

[0205] As shown in FIG. 8, which is another schematic diagram of model parameter updating in an embodiment of the present application. The difference from FIG. 7 is that the original parameters of the LLM are frozen in FIG. 8, and the visual encoder and the LLM are not updated, while the query converter, FC and LoRA are updated. FC is used to dimensionally align the features output by the query converter to adapt to the input of the LLM, and in the manner with LoRA, the query converter and LoRA are updated together.

[0206] Of course, the FC layer can also be replaced by a linear projection module or a multi-layer perceptron (MLP), etc., which is not specifically limited in the present disclosure.

[0207] In the above embodiments, in the first training stage, the query converter is focused on training, and the query converter is adjusted based on the video text description information. This can improve the accuracy of the query converter in extracting visual features, and ensure that the extracted visual features can more accurately describe the main content of the video. Based on this, in the second training stage, the query converter and the LLM are adjusted simultaneously, and the query converter and the LLM are adjusted based on the video label. This ensures that the LLM can generate more accurate video labels based on the visual features extracted by the query converter, thereby improving the accuracy of the video labels extracted by the model.

[0208] In addition, it should be noted that when training the video label identification model, the first training stage and the second training stage can be alternately performed, i.e., the parameters of the query converter are updated once, then the parameters of the query converter and the LLM are updated simultaneously, then the parameters of the query converter are updated again, and then the parameters of the query converter and the LLM are updated simultaneously…

[0209] In addition, the second training stage can also be performed after the first training stage is completed, i.e., the parameters of the query converter are updated multiple times, and then the parameters of the query converter and the LLM are updated simultaneously multiple times.

[0210] Regardless of the training method described above or the training stage, the iteration stopping condition includes at least one of the following conditions:

[0211] (1) The number of iterations reaches a set number threshold.

[0212] (2) The model loss value is less than a set loss threshold.

[0213] In the embodiments of the present application, when the number of iterations does not exceed the pre-set number threshold, and the model loss value is not less than the set loss threshold, it is considered that the current model does not satisfy the iteration termination condition, and the model parameters need to be adjusted for further training. After adjusting the parameters, the next iteration training process is entered. When the number of iterations has exceeded the pre-set number threshold, or the model loss value is less than the set loss threshold, it is considered that the current model has satisfied the convergence condition, and the model training is ended.

[0214] It should be noted that the visual encoder and the text encoder in the present application can be replaced by other network structures; the query transformer is used to align the visual features to the input of the LLM, and thus can be understood as a feature alignment module. In addition to being designed as a Querying transformer, the feature alignment module can also be designed as other forms of adapter network, q-former network, etc., and is not limited to the Querying transformer designed in the present application. In addition, the memory unit can also be changed to other similar memory bank structures, and the main process of the present application can also replace the framework of the LLM with other generative models, etc. The simple modifications made to each unit of the video tag identification model listed in the present application, or the application of the core process of the present application in other scenarios, all fall within the protection scope of the present application.

[0215] After the model training is completed, it can be deployed online. Whether it is a short video, a medium video, a long video, or a video stream, it is input in the form of a video frame sequence. After compression by the MB, an audit can be triggered at any time (i.e., triggering video tag identification). Based on the Querying transformer, the compressed and fused visual tokens are calculated, and then the video text tokens and text instruction are input into the LLM to generate the predicted label results.

[0216] The above is mainly an introduction to the model training process. The following describes the model application process:

[0217] Referring to FIG. 9, an implementation flowchart of a video tag identification method provided by an embodiment of the present application is shown. Taking a server as an execution subject, the specific implementation process of the method is as follows: S91-S94.

[0218] S91: Encode each video frame in the identification task video by the video tag identification model to form a global feature set including global features of each video frame obtained by encoding, and a local feature set including local features of each video frame obtained by encoding; compress the global feature set based on the feature similarity between the global features in the global feature set to obtain a global feature sequence of a preset storage quantity; and compress the local feature set based on the feature similarity between the local features in the local feature set to obtain a local feature sequence of a preset storage quantity.

[0219] As described above, the identification task video in the embodiment of the present application can be any video, including but not limited to a short video, a medium video, a long video, and a video stream.

[0220] Among them, short video, medium video and long video refer to videos of different time lengths. For example, short video usually refers to video content with a length of 15 seconds to 10 minutes, medium video covers a range of 1 minute to 30 minutes, and long video refers to video with a length of more than 30 minutes, such as TV series, movies, etc.

[0221] In this step, the visual encoder in the video tag identification model can be used to encode each video frame in the identification task video respectively to obtain the global features of each video frame and the local features of each local region of each video frame. Each local feature corresponds to an image block in the video frame.

[0222] In an embodiment, before encoding the identification task video, the server can select a suitable video frame segmentation method according to the content complexity and feature distribution of the video. If the video content is relatively simple and the feature distribution is uniform, the server can segment the video frames in the identification task video into the same number of image blocks, which can ensure the consistency of feature extraction of each video frame and facilitate subsequent processing. For example, for some landscape videos, the picture elements change little, and the same number of image blocks can be used for efficient extraction of global and local features. If the video content is complex and the feature distribution is uneven, the server can segment the video frames in the identification task video according to a pyramid segmentation method to obtain corresponding image blocks, and the number of image blocks of different video frames is not the same. The pyramid segmentation can provide more detailed feature extraction in areas with large changes in video content and reduce unnecessary calculations in areas with relatively stable content. For example, for action videos containing fast moving objects and complex scene switching, the pyramid segmentation can better capture the key features in the video.

[0223] Among them, the number of image blocks of each video frame obtained according to the above segmentation method can be the same or in a geometric relationship. For example, the video frames in the identification task video are segmented according to the same segmentation method, so that the number of image blocks of each video frame is the same, such as each video frame is segmented according to a 3x3 method, as shown in FIG. 10.

[0224] In addition, the pyramid segmentation method can refer to the number of image blocks increasing according to a certain rule from the first frame of the video, for example, the number of image blocks of the next frame is twice that of the previous frame. The server can also segment the video frames in the video according to the pyramid segmentation method, such as the first video frame is segmented according to a 2x2 method to obtain 4 image blocks, the second video frame is segmented according to a 3x3 method to obtain 9 image blocks, and so on.

[0225] It should be noted that when segmenting, each image block can have overlapping content or can not have overlapping content, which is not limited in this paper.

[0226] The following takes the example of splitting each video frame in the recognition task video into the same split manner, assuming that the number of image blocks of each video frame is the same, N, then when the video frame is encoded by the visual encoder ViT, the global feature is denoted as f glob , size is 1*d, and the local feature is denoted as f loc,i , size is N*d, N is the number of patches, i = 1, 2, …, N.

[0227] In the embodiments of the present application, the global feature set contains at least one global feature, specifically the global features of each video frame in the recognition task video that has been encoded. As time goes on, more and more video frames are encoded, and the global features contained in the global feature set also increase.

[0228] Similarly, the local feature set contains at least one local feature, specifically the local features of each video frame in the recognition task video that has been encoded. Wherein, if each video frame can be divided into N image blocks, then each video frame corresponds to N local features.

[0229] In the embodiments of the present application, compression can be performed after all video frames in the recognition task video are encoded, or compression can be performed while encoding. In the case of encoding and compression at the same time, an optional implementation is as follows:

[0230] Each time a video frame is encoded, the global feature of the video frame is stored in the global memory unit, and the local feature of the video frame is stored in the local memory unit.

[0231] Specifically, the two memory units designed in the present application store and compress global features and local features, respectively. For the input video frame sequence, it can be represented as {img (t) |t = 1, 2, 3, …, T}, T is the total number of frames of the recognition task video, here T can be constantly increasing, and is not limited to processing a known length of video frame sequence. As described earlier, each video frame img (t) extracted by the visual encoder can obtain global features and local features, denoted as f (t) = {f glob (t) , f loc,i (t)}, which are input into the corresponding MB.

[0232] In addition, since the number of video tokens is large, the global memory unit and the local memory unit are each provided with a corresponding preset storage quantity, such as a maximum length in the global memory unit, that is, the preset storage quantity of the global memory unit is M, indicating that the global features of up to M frames of video are accommodated, and correspondingly, the preset storage quantity of the local memory unit is M*N, indicating that the local features of up to M frames of video are accommodated, and each frame of video corresponds to N local features.

[0233] Further, when the number of global features stored in the global memory unit reaches the preset storage quantity, based on the feature similarity between the global features of the plurality of groups of video frames stored in the global memory unit and the global features to be stored, the global features of at least two video frames are fused into the global features of one video frame to adjust the global features stored in the global memory unit, so that the number of global features is less than or equal to the preset storage quantity; and for each image block, when the number of corresponding local features stored in the local memory unit reaches the preset storage quantity, based on the feature similarity between the local features in the local memory unit, the corresponding local features of at least two video frames in the local memory unit are fused, so that the number of local features stored in the local memory unit is less than or equal to the preset storage quantity.

[0234] For example, for the global memory unit, when the data quantity in the MB reaches M, if the global features of a new video frame need to be added to the MB, compression needs to be performed once, so that the data in the MB remains no more than M.

[0235] Similarly, for the local memory unit, when the data quantity in the MB reaches M*N, if the local features of a new video frame need to be added to the MB, compression needs to be performed once, so that the data in the MB remains no more than M*N.

[0236] It should be noted that when the local features are compressed, the local features of each image block are fused in units of image blocks.

[0237] In addition, it should be noted that when calculating the similarity between features, the distance between feature vectors is usually calculated to achieve this, and the calculation method of the feature similarity in the embodiments of the present application has many methods, including but not limited to any one of the following:

[0238] Cosine similarity, Euclidean distance, Manhattan distance, Jaccard similarity, Pearson Correlation Coefficient.

[0239] In the above embodiments, the global memory unit can capture the macro information of the video, and the local memory unit focuses on the detail changes, and the two are complementary. By introducing two memory units to store and compress the global features and local features of the video frame sequence respectively, the video data can be processed more efficiently, the comprehensiveness of the information is ensured, the redundancy is reduced, and the storage efficiency and computing performance are improved.

[0240] In the following, the cosine similarity will be taken as an example for illustration.

[0241] Optionally, the global memory unit and the local memory unit are collectively referred to as a memory unit, and the global features and the local features are collectively referred to as target features. In the embodiments of the present application, the target features of at least two video frames can be fused in any of the following ways:

[0242] Compression mode one: based on the feature similarity, the target features of the most similar adjacent video frames in the memory unit are average-pooled.

[0243] Specifically, in the case where the storage capacity of the memory unit has reached the upper limit, each time a new target feature, i.e., a target feature to be stored, is to be stored, the target features of the most similar adjacent video frames in the memory unit can be average-pooled, and then the target feature to be stored can be stored in the memory unit.

[0244] For example, for the global memory unit: based on the feature similarity between the global features of multiple groups of video frames in the global features stored in the global memory unit and the global features to be stored, the global features of at least two video frames are fused into the global features of one video frame, including: based on the feature similarity between the global features of multiple groups of adjacent video frames in the global features stored in the global memory unit and the global features to be stored, the global features of the most similar at least one group of adjacent video frames are average-pooled by group to obtain the global features of one video frame in each group.

[0245] Specifically, the global feature with the maximum cosine similarity between adjacent video frames is calculated, and is compressed into one global feature in the form of average-pooling, denoted as a token, which can be represented as: k = argmax t (cos(f glob (t) ,f glob (t+1) )), t = 1, 2, …, M (Formula 1) f glob (k) = 0.5 * (f glob (k) +f glob (k+1) ) (Formula 2)

[0246] wherein k represents the previous frame in the most similar adjacent video frames, f glob (k) i.e. the global feature of the video frame; k+1 represents the next frame in the most similar adjacent video frames, f glob (k+1) i.e. the global feature of the video frame. After feature fusion of the two frames, the obtained global feature can still be denoted as f glob (k) .

[0247] Specifically, assuming that the global features of the first frame to the tenth frame are currently stored in the global memory unit, M=10, after the global feature of the eleventh frame of video frame is obtained by encoding, the data of the global memory unit reaches the upper limit, then the similarity between each adjacent two frames in the first frame to the tenth frame needs to be analyzed and calculated, and the global features of the most similar adjacent video frames are averaged and pooled.

[0248] As shown in FIG. 11, it is a global feature compression process schematic diagram in the embodiment of the present application. Assuming that several frames of global features are stored in the global memory unit from right to left in order, assuming that k=5, then the global features of the fifth frame and the sixth frame need to be averaged and pooled as the new global feature of the fifth frame, which can be denoted as 5', and then the global feature of the eleventh frame of video frame can be stored in the global memory unit.

[0249] Further, after the global feature of the twelfth frame of video frame is obtained by encoding, the data of the global memory unit also reaches the upper limit, then the similarity between each adjacent two frames in the first frame, the second frame, the third frame, the fourth frame, the compressed fifth frame, the seventh frame, the eighth frame, the ninth frame, the tenth frame and the eleventh frame needs to be analyzed and calculated, and the global features of the most similar adjacent video frames are averaged and pooled.

[0250] It needs to be emphasized that considering that each compression in the present application will affect the number of the video frame corresponding to the global feature, therefore, each adjacent two frames here refers to the frames which are adjacent in the global memory unit, not strictly adjacent. For example, the original fourth frame and the compressed fifth frame belong to adjacent two frames, the compressed fifth frame and the original seventh frame also belong to adjacent two frames, and so on.

[0251] Assuming that the global features of the eighth frame and the ninth frame need to be averaged and pooled as the new global feature of the eighth frame, denoted as 8', and then the global feature of the twelfth frame of video frame can be stored in the global memory unit.

[0252] Similarly, after encoding the global features of the 13th video frame, if the data of the global memory unit also reaches the upper limit, the similarity between each adjacent two frames of the 1st, 2nd, 3rd, 4th, compressed 5th, compressed 8th, 10th, 11th and 12th frames needs to be analyzed and calculated, and then the global features of the most similar adjacent video frames are averaged and pooled.

[0253] Suppose that the global features of the compressed 5th and 7th frames need to be averaged and pooled this time as the global features of the 5th frame after secondary compression, denoted as 5", and then the global features of the 13th video frame can be stored in the global memory unit.

[0254] When the global features of new video frames are stored subsequently, the same principle applies, which will not be repeated here.

[0255] Similarly, for the local memory unit: based on the feature similarity between the local features in the local memory unit, the corresponding local features of at least two video frames in the local memory unit are fused, including: based on the feature similarity between the local features in the local memory unit, the local features of the most similar at least one group of adjacent video frames in the local memory unit are averaged and pooled within the group.

[0256] Specifically, using a similar method, the local feature with the maximum cosine similarity between adjacent video frames (specifically, the local feature corresponding to the patch position) is calculated, compressed into a local feature in the form of average pooling, denoted as a token. It should be noted that, unlike global features, a video frame can be divided into N patches, each patch corresponding to a local feature, so when fusing local features between adjacent video frames, attention should be paid to fusing the corresponding patch positions, which can be specifically represented as: k i = argmax t (cos(f loc,i (t) ,f loc,i (t+1) )), t = 1, 2, …, M, i = 1, 2, …, N (Formula 3)

[0257] f loc,i (k) = 0.5 * (f loc,i (k) + f loc,i (k+1) ) (Formula 4)

[0258] where k represents the previous frame in the most similar adjacent video frames, f loc,i (k)i.e. the local feature of the i-th patch in the video frame; k+1 represents the next frame in the most similar adjacent video frame, f loc,i (k+1) i.e. the local feature of the i-th patch in the video frame. After feature fusion of the i-th patch in the two frames, the obtained local feature can still be denoted as f loc,i (k) .

[0259] It should be noted that in the embodiments of the present application, the compression of the global feature and the local feature is realized based on two memory units, and thus in each compression, the compression of the global feature and the local feature is not limited to the same two adjacent video frames.

[0260] In addition, considering that in each compression, there can be multiple groups of most similar adjacent video frames, in this case, when performing average pooling on the target features of the most similar adjacent video frames, an optional implementation is as follows:

[0261] The target features of the most similar adjacent video frames with the longest storage duration in the memory unit are subjected to average pooling, or the target features of each group of most similar adjacent video frames in the memory unit are subjected to average pooling.

[0262] The first mode indicates that if there are multiple groups of most similar adjacent video frames, the adjacent video frames with a relatively longer storage duration are compressed according to the storage duration, and of course, each group can also be compressed.

[0263] Specifically, if there are multiple groups of most similar global features of adjacent video frames in the global features of multiple groups of adjacent video frames, the global features of at least one group of most similar adjacent video frames are subjected to average pooling by group to obtain the global feature of one video frame in each group, including: the global features of the most similar group of adjacent video frames including the global feature with the longest storage duration are subjected to average pooling.

[0264] For example, the global memory unit currently stores the global features of the first frame to the tenth frame, M=10, and after encoding the global feature of the eleventh frame, the data of the global memory unit reaches the upper limit, and thus the similarity between each adjacent two frames from the first frame to the tenth frame needs to be analyzed and calculated, and then the global features of the most similar adjacent video frames are subjected to average pooling.

[0265] Suppose the most similar adjacent video frames calculated are [frame 5, frame 6] and [frame 8, frame 9], and the storage time of frame 5 is the longest, then the global features of frame 5 and frame 6 can be average-pooled to obtain the global feature of a new frame 5, and then the global feature of frame 11 can be stored in the global memory unit.

[0266] Of course, the global features of frame 5 and frame 6 can also be average-pooled to obtain the global feature of a new frame 5, and the global features of frame 8 and frame 9 can also be average-pooled to obtain the global feature of a new frame 8, and then the global feature of frame 11 can be stored in the global memory unit.

[0267] In addition, there is still one position in the global memory unit, so after the global feature of frame 12 is encoded, the data in the global memory unit has reached the upper limit, and thus the global feature of frame 12 can be directly stored in the global memory unit. Further, after the global feature of frame 13 is encoded, the data in the global memory unit has reached the upper limit, and thus compression can be performed again.

[0268] In the above embodiment, the features of the video frame with a longer storage time are compressed according to the storage time, which can not only reduce data redundancy and save space occupied by data, but also provide more space for new data and ensure the timeliness of the video tag recognition result.

[0269] Compression method two: based on feature similarity, clustering the target feature to be stored with the target feature already stored in the memory unit, and average-pooling the target features of the video frames that have been clustered.

[0270] The target feature is a global feature or a local feature, and the memory unit is a global memory unit or a local memory unit. When the storage capacity of the memory unit has reached the upper limit, a new target feature is stored each time, that is, the target feature to be stored, and then the target feature to be stored can be clustered with the most similar target feature in the memory unit, and the target features of the two video frames that have been clustered can be average-pooled to achieve the compression effect.

[0271] Specifically, based on the feature similarity between the global features stored in the global memory unit and the global features of the multiple groups of video frames in the global features to be stored, the global features of at least two video frames are fused into the global feature of one video frame, including: based on the feature similarity between the global features stored in the global memory unit and the global features of the multiple groups of video frames in the global features to be stored, at least part of the global features in the global features stored in the global memory unit and the global features to be stored are clustered, and the global features of at least two video frames clustered into one category are fused into the global feature of one video frame in the category.

[0272] Suppose that the global memory unit currently stores the global features of the first frame to the tenth frame, M = 10, and after the global feature of the eleventh frame is obtained by encoding the eleventh frame, the data of the global memory unit reaches the upper limit, then the first frame to the tenth frame that is most similar to the eleventh frame, such as the tenth frame, can be analyzed, and then the tenth frame and the eleventh frame can be subjected to average pooling operation, and the pooled global feature can be stored in the global memory unit.

[0273] Similarly, when clustering, if there are multiple target features in the memory unit that are most similar to the target feature to be stored, then the target feature with the longest storage time in the memory unit and the target feature to be stored can be clustered; or, the multiple target features in the memory unit that are most similar to the target feature to be stored and the target feature to be stored can be clustered.

[0274] Suppose that the global memory unit currently stores the global features of the first frame to the eleventh frame after compression, M = 10, and after the global feature of the twelfth frame is obtained by encoding the twelfth frame, the data of the global memory unit reaches the upper limit, then the global memory unit that is most similar to the twelfth frame, such as the second frame and the third frame, can be analyzed, and then the second frame and the twelfth frame can be subjected to average pooling operation, and the pooled global feature can be stored in the global memory unit. In this way, after the global feature of the thirteenth frame is obtained by encoding the thirteenth frame, the data of the global memory unit still reaches the upper limit, then the global memory unit that is most similar to the thirteenth frame can be analyzed, and average pooling can be performed.

[0275] Of course, the second frame, the third frame and the twelfth frame can also be subjected to average pooling operation, and the pooled global feature can be stored in the global memory unit. In this way, after the global feature of the thirteenth frame is obtained by encoding the thirteenth frame, the data of the global memory unit has not yet reached the upper limit, then the global feature of the thirteenth frame can be directly stored in the global memory unit.

[0276] Of course, in the case that the storage capacity of the memory unit has reached the upper limit, a new target feature can be directly stored each time, that is, when a target feature to be stored is to be stored, the target feature to be stored can be clustered with several similar target features in the memory unit, and the target features of the clustered video frames can be average-pooled to achieve the compression effect. The several similar target features refer to target features with a feature similarity greater than a certain threshold or top-ranked in feature similarity, and the like, which are not limited in the present disclosure.

[0277] Similarly, the compression method for the local memory unit is also similar, and the difference is that when the local features of at least two video frames are fused, attention should be paid to fusing the corresponding patch positions.

[0278] Specifically, in some embodiments, based on the feature similarity between the local features of the plurality of groups of video frames at the image block in the local features stored in the local memory unit and the local features to be stored, the local features of at least two video frames at the image block are fused into the local features of one video frame at the image block, including: based on the feature similarity between the local features of the plurality of groups of adjacent video frames at the image block in the local features stored in the local memory unit and the local features to be stored, the local features of at least one group of the most similar adjacent video frames at the image block are average-pooled by group to obtain the local features of one video frame in each group.

[0279] In some embodiments, if there are multiple groups of local features of the most similar adjacent video frames at the image block in the local features of the plurality of groups of adjacent video frames at the image block, the local features of at least one group of the most similar adjacent video frames at the image block are average-pooled by group, including: the local features of at least one group of the most similar adjacent video frames at the image block, including the longest stored local features and the most similar local features at the image block, are average-pooled by group.

[0280] In some embodiments, based on the feature similarity between the local features of the plurality of groups of video frames at the image block in the local features stored in the local memory unit and the local features to be stored, the local features of at least two video frames at the image block are fused into the local features of one video frame at the image block, including: based on the feature similarity between the local features of the plurality of groups of video frames at the image block in the local features stored in the local memory unit and the local features to be stored, at least a part of the local features in the local features stored in the local memory unit and the local features to be stored are clustered, and the local features of at least two video frames at the image block in the same class are fused into the local features of one video frame at the image block in the same class.

[0281] In the above embodiments, based on the feature similarity, the target features are clustered into different groups. If a new feature is similar enough to a certain feature in the memory unit, they will be classified into the same cluster, and similar video frame features can be summarized together, reducing repeated storage while preserving the key information of the video.

[0282] It should be noted that the above-mentioned several compression methods are based on feature similarity. In addition, feature compression can also be performed according to the storage duration, such as:

[0283] Compression method three: average pooling the target features of the longest continuous video frames to reduce inter-frame redundancy. Among them, compression method three can be used when neither compression method one nor compression method two is used, or it can be used in combination with at least one of compression method one and compression method two. For example, when the number of target features with a storage duration reaching a preset storage duration in a memory unit is less than or equal to 1, at least one of compression method one and compression method two is used; when the number of target features with a storage duration reaching a preset storage duration and being continuous in a memory unit is greater than or equal to 2, compression method three is used.

[0284] In the case where the storage capacity of the memory unit has reached the upper limit, each time a new target feature is to be stored, the target features of the longest storage duration and adjacent video frames in the memory unit can be average-pooled, and then the target feature to be stored can be stored in the memory unit.

[0285] In order to accurately measure the storage duration, we can record a timestamp for each feature in the memory unit, which indicates the time when the feature is stored in the memory unit. When feature compression is needed, the storage duration of the longest feature is determined by comparing the timestamps. At the same time, in order to better manage the storage duration of the memory unit, we can set a time threshold T th When the storage duration of a certain feature in the memory unit exceeds T th , it is preferred to consider compression processing. For example, for the global memory unit, when the number of global features with a storage duration exceeding T th reaches a certain proportion, the longest continuous video frame global feature is compressed by average pooling. Similarly, for the local memory unit, the storage duration of each image block corresponding to the local feature is processed accordingly. This can more effectively control the data update in the memory unit and ensure that the model can timely reflect the latest features of the video.

[0286] For example, each time a new target feature is to be stored, the target features of the first 2 video frames can be average-pooled in chronological order.

[0287] In addition, the sliding window can also be used for feature compression, and the size of the sliding window is at least 2 and at most M. For example, if a sliding window with a length of 2 is selected, when a new target feature is to be stored, the target features of the two video frames in the sliding window are averaged and pooled.

[0288] The sliding window is a method for feature compression. In the present application, when the storage capacity of the memory unit reaches the upper limit, the sliding window can be used for feature compression when a new target feature is to be stored. The size of the sliding window is at least 2 and at most a preset storage number M. For example, if a sliding window with a length of 2 is selected, when a new target feature is to be stored, the target features of the two video frames in the sliding window are averaged and pooled.

[0289] It should be noted that the above-mentioned several compression methods are only simple examples, and other compression methods are also applicable to the embodiments of the present application, which will not be described here.

[0290] In the above embodiments, the target features of two or more video frames are fused by averaging and pooling, which can avoid storing too many almost identical features, save storage space, and more accurately reflect the common properties of the video frames. In addition, the averaging and pooling operation can also reduce the influence of noise and improve the stability of the features.

[0291] In the embodiments of the present application, the identification task video is input in the form of a video frame sequence, and feature compression is performed by the memory unit. In order to avoid unnecessary resource waste, a specified identification time can be set to trigger the identification of the video label.

[0292] The specified identification time refers to a time set flexibly according to actual needs, which can be set according to the video length, type, etc. of the identification task video, and is used to trigger the identification of the video label. When the specified identification time is reached, the global feature sequence stored in the global memory unit and the local feature sequence stored in the local memory unit are read, and then the subsequent video label identification process is performed; if the specified identification time is not reached, the feature sequence stored in the memory unit is not read, and only the memory unit is used for feature compression.

[0293] Specifically, before the first cross-attention feature between each global feature in the global feature sequence and the self-attention feature is extracted, and the second cross-attention feature between each local feature in the local feature sequence and the self-attention feature is extracted, it is necessary to determine whether the specified identification time is reached, and if so, the audit is triggered. The specified identification time can be flexibly adjusted according to actual needs.

[0294] If it is determined that the specified recognition time is reached, the global feature sequence stored in the global memory unit and the local feature sequence stored in the local memory unit are read, and then subsequent processes are performed to realize the recognition of the video label.

[0295] If it is not determined that the specified recognition time is reached, the feature sequences stored in the global memory unit and the local memory unit are not read, and the subsequent label recognition process is not performed, and only feature compression based on the memory unit is needed.

[0296] The specified recognition time in the embodiments of the present application can be flexibly set according to actual needs, or can be set according to the video length, type, etc. of the recognition task video, such as: for medium video and long video, prediction can be performed after the input of the video frame sequence is completed; for the form of video stream such as live scene, prediction can be performed once every N minutes, etc.

[0297] That is, the review of the recognition task video can be triggered after the input of the recognition task video is completed, in which case all video frames of the recognition task video can be encoded and input into the MB, and one or more compressions are performed through the MB, and finally the features of each video frame in the recognition task video are compressed into a fixed-length feature sequence, which can be output. Correspondingly, the specified recognition time refers to the time when the feature compression is completed.

[0298] In addition, in the process of continuously inputting the recognition task video, the review of the recognition task video is triggered once every specified recognition time, in which case the feature sequence output by the MB during each review is obtained by continuing to compress the features of the video frames input during the time period based on the feature sequence output during the previous review.

[0299] It should be noted that the above-mentioned several review methods are only simple examples, and other review methods are also applicable to the embodiments of the present application, which will not be described one by one.

[0300] In the above embodiments, by setting the specified recognition time, the subsequent label recognition process is triggered when the specified recognition time is reached, which can avoid the model always performing video label recognition and cause unnecessary resource waste. In addition, the specified recognition time can be flexibly set by the object, so that the recognition of the video label is more flexible.

[0301] S92: After splicing the global query feature and the local query feature obtained by pre-training, a self-attention mechanism is used to extract a self-attention feature; the global query feature and the local query feature are obtained by training the learnable query feature based on the sample video; the self-attention feature fuses the key information in the global query feature and the local query feature.

[0302] wherein the global query feature is obtained by training the first learnable query feature, and the local query feature is obtained by training the second learnable query feature, and the specific training process is as described above, which is not repeated here.

[0303] The global query feature and the local query feature are obtained by training the learnable query feature based on the sample video. The specific training process is as follows: the sample video is input into the video label recognition model to be trained, each video frame in the sample video is encoded to form a global feature set and a local feature set, the global feature set and the local feature set are compressed based on feature similarity to obtain a global feature sequence and a local feature sequence with a preset storage quantity; after the first learnable query feature and the second learnable query feature to be trained are spliced, a self-attention mechanism is used to extract a self-attention feature; a cross-attention mechanism is used to extract a first cross-attention feature between the self-attention feature and each global feature in the global feature sequence, and a second cross-attention feature between the self-attention feature and each local feature in the local feature sequence; based on the obtained each first cross-attention feature and each second cross-attention feature, the difference between the predicted video label and video text description information of the sample video and the corresponding sample label is used to adjust the parameters of the first learnable query feature and the second learnable query feature, and after multiple iterations of training, the iteration stopping condition is met, and the global query feature and the local query feature are obtained.

[0304] In this step, the query converter in the video label recognition model can be used to realize the training process. The query converter includes a self-attention network and two parallel cross-attention networks. The spliced results of the global query feature and the local query feature are used as the input of the self-attention network in the query converter. This process allows the global feature sequence to capture local details, and at the same time allows the local feature sequence to obtain global context, so as to finally generate a fused feature representation that is both detailed and broad in view.

[0305] As shown in FIG. 12, it is a calculation process diagram of an attention mechanism in an embodiment of the present application. The attention mechanism allows the model to focus on different parts of the input sequence in order to better capture the context information. Specifically, the left part of FIG. 12 shows three matrix multiplication operations, which correspond to the Query, Key and Value matrices in the attention mechanism, respectively. The result of these matrix multiplications is to generate Q, K and V matrices, which are all the same size. These matrix multiplications are obtained by multiplying the input matrix X with the weight matrices Wq, Wk and Wv. Among them, the three weight matrices are trainable parameters used to extract features.

[0306] The right formula shows the calculation process of the attention mechanism. First, the Q and K matrices are dot-multiplied (inner product), then normalized by the softmax function to form the attention score matrix. Where d k Key dimension, usually used to scale the dot product attention score to help the gradient flow through the network. This normalized attention score matrix represents the similarity between the query vector of each position and the key vector of each position. Then, multiply the attention score matrix with the V matrix to get the final attention output Z. This output contains information from all positions of the input, but the weight is allocated according to the attention score, so the position of the key vector that is most relevant to the query vector will have a higher weight. This process can be repeated multiple times to form multi-head attention, where each head generates an attention output, allowing the model to focus on different aspects of the input, and then combining them to enhance the expressive power and generalization ability of the model. Multi-head attention is an extension of the attention mechanism that allows the model to focus on different aspects of the input. In this application, self-attention networks and cross-attention networks can use multi-head attention mechanisms. The specific process is to repeat the calculation process of the attention mechanism multiple times, each head generates an attention output, and then combines these outputs to enhance the expressive power and generalization ability of the model. In the self-attention mechanism, the input matrix is multiplied by trainable weight matrices to obtain query (Q), key (K) and value (V) matrices, and the attention output is obtained by a series of operations on these matrices. The combination of multiple such head outputs is the result of multi-head attention.

[0307] In the self-attention network, the input X is the concatenation of the global query feature and the local query feature, which can be represented as X = concat(Q1, Q2), Wq, Wk and Wv are the weight matrices of the self-attention network, and the output Z of the self-attention network is the self-attention feature in this application. These weight matrices Wq, Wk and Wv can be initialized using the Xavier initialization method. The core idea of Xavier initialization is to keep the variance of the input and output consistent, avoiding the problem of gradient vanishing or gradient explosion in the forward and backward propagation process of the neural network. Specifically, for a fully connected layer with n in input and n out output, each element w ij in the weight matrix can be randomly sampled from a uniform distribution In the self-attention mechanism, by initializing the weight matrix with Xavier, the model can more stably learn the relationship between input features in the early stages of training.

[0308] The calculation process of the attention mechanism is described as follows: first, the global query feature and the local query feature are spliced to obtain an input matrix X, i.e., X = concat(Q1, Q2), wherein Q1 is the global query feature and Q2 is the local query feature. Then, the input matrix X is multiplied by trainable weight matrices W q , W k , and W v , respectively, to obtain a query matrix Q, a key matrix K, and a value matrix V, i.e., Q = XW q , K = XW k , and V = XW v . Next, the Q and K matrices are dot-multiplied (inner product), and then normalized by a softmax function to form an attention score matrix. Wherein d k is the size of the key dimension, which is usually used to scale the dot-product attention score to help the gradient flow through the network. This normalized attention score matrix represents the similarity between the query vector of each position and the key vector of each position. The specific formula is Finally, the attention score matrix is multiplied by the V matrix to obtain the final attention output Z, which is the self-attention feature. This process can be repeated multiple times to form multi-head attention, wherein each head generates an attention output, allowing the model to focus on different aspects of the input, and then combining them to enhance the expressiveness and generalization ability of the model.

[0309] S93: A cross-attention mechanism is used to extract a first cross-attention feature between the self-attention feature and each global feature in the global feature sequence, and a second cross-attention feature between the self-attention feature and each local feature in the local feature sequence.

[0310] Since the self-attention feature obtained in step S92 is obtained on the basis of splicing the global query feature and the local query feature, in S93, the query converter in the video tag recognition model can also be used to achieve this, and the two cross-attention networks in the query converter can be denoted as: the first cross-attention network (corresponding to the global feature) and the second cross-attention network (corresponding to the local feature); S93 can be implemented according to the flowchart shown below, including the following steps S931-S933 (not shown in FIG. 9):

[0311] S931: The self-attention feature is split into a first self-attention feature corresponding to the global feature and a second self-attention feature corresponding to the local feature.

[0312] Similar to the training process, before extracting the cross-attention feature, the self-attention feature obtained in S92 is first split into two parts, denoted as a first self-attention feature and a second self-attention feature.

[0313] S932: Extract, through the first cross-attention network in the query converter, a first cross-attention feature between the first self-attention feature and each global feature in the global feature sequence based on the cross-attention mechanism.

[0314] Specifically, in the first cross-attention network, there are two inputs, one being the global feature and the other being the first self-attention feature. Corresponding to FIG. 12, X has two, wherein the X in the first matrix multiplication in the left part can be understood as the first cross-attention feature to generate the Q matrix; the X in the second and third matrix multiplications in the left part can be understood as the global feature to generate the K and V matrices. Based on the first self-attention feature that integrates the key information in the global query feature and the local query feature, relevant information is attempted to be extracted from the global feature, and the output Z is the first cross-attention feature in the embodiment of the present application at this time.

[0315] S933: Extract, through the second cross-attention network in the query converter, a second cross-attention feature between the second self-attention feature and each local feature in the local feature sequence based on the cross-attention mechanism.

[0316] Similar to S932, S933 and S932 are implemented through two parallel cross-attention networks, and the specific execution process is similar. The second cross-attention network also has two inputs, one being the local feature and the other being the second self-attention feature. Corresponding to FIG. 12, X has two, wherein the X in the second matrix multiplication in the left part can be understood as the second cross-attention feature to generate the Q matrix; the X in the second and third matrix multiplications in the left part can be understood as the local feature to generate the K and V matrices. Based on the second self-attention feature that integrates the key information in the global query feature and the local query feature, relevant information is attempted to be extracted from the local feature, and the output Z is the second cross-attention feature in the embodiment of the present application at this time.

[0317] In the above embodiments, in the self attention module, the global feature query sequence and the local feature query sequence can see each other, so that the two features can interact to obtain self-attention features that simultaneously fuse key information in the global query features and the local query features. On this basis, after the self-attention features are split, they are input into the corresponding cross attention modules, two parallel cross attention modules are used to process the global features and the local features respectively, so as to ensure that relevant information is extracted from the global features while the local features are focused on, and relevant information is extracted from the local features while the global features are focused on, so as to extract more perfect first cross-attention features and second cross-attention features. In addition, further compression through cross attention can obtain more compact visual token sequences.

[0318] The cross attention mechanism is used to extract the first cross-attention features between each global feature in the global feature sequence and the self-attention features, and to extract the second cross-attention features between each local feature in the local feature sequence and the self-attention features. The specific calculation process is as follows: first, the self-attention features are split into first self-attention features Q self1 corresponding to the global features and second self-attention features Q self2 corresponding to the local features. For extracting the first cross-attention features, the first self-attention features Q self1 are multiplied by the weight matrix W q1 to obtain the query matrix Q1, i.e. Q1 = Q self1 W q1 ; each global feature F glob in the global feature sequence is multiplied by the weight matrix W k1 and W v1 to obtain the key matrix K1 and the value matrix V1, i.e. K1 = F glob W k1 , V1 = F glob W v1 . Then, the Q1 and K1 matrices are dot multiplied (inner product), and then normalized by the softmax function to form the attention score matrix, the formula is where d k1 is the size of the key dimension, used to scale the dot product attention score. Then, the attention score matrix is multiplied by the V1 matrix to obtain the first cross-attention features Z1, i.e. Z1 = Attention score1 V1. For extracting the second cross-attention features, the second self-attention features Q self2 are multiplied by the weight matrix W q2The multiplication obtains a query matrix Q2, and each local feature F in the local feature sequence is multiplied by the weight matrix W loc k2 and W v2 to obtain a key matrix K2 and a value matrix V2, and the second cross-attention feature Z2 is obtained through dot product, normalization and multiplication operations.

[0319] In the embodiment of the present application, the Querying transformer serves as a bridge connecting non-text tokens and LLM, which can convert inputs from different modalities into tokens that can be understood by LLM, and then the prediction of video labels can be performed through LLM in combination with the tokens. The specific process is as follows:

[0320] S94: Based on the obtained first cross-attention features and second cross-attention features, video labels of the recognition task video are identified.

[0321] The MLLM in the embodiment of the present application uses two modalities, i.e., visual modality and text modality. As shown in FIG. 13, in the embodiment of the present application, each video frame in the recognition task video (video stream) can obtain two visual modality features, including global features and local features, after being encoded by a visual encoder. Then, the global features are compressed by a global feature MB, and the local features are compressed by a local feature MB. After that, the first cross-attention features and the second cross-attention features are obtained through further fusion and compression of multi-head self-attention and multi-head cross-attention in the Querying transformer. After splicing, the dimension conversion is performed through an FC layer to obtain the final visual token, denoted as [video_emb], which is the visual feature in the present application. For the specific implementation, refer to the above-mentioned embodiments, and the repeated parts will not be described herein. The features of the text modality can include description text, key text extracted in OCR / ASR, etc. in addition to the title of the video, denoted as [video_txt], which is the text feature in the present application.

[0322] Taking [video_txt] and the text instruction as a prompt, the LLM can generate the predicted label result by inputting [video_emb].

[0323] It should be noted that the video label recognition model listed in FIG. 13 is trained on the basis of the model shown in FIG. 8, and thus further includes a LoRA module.

[0324] In summary, one optional implementation of S94 is as follows:

[0325] ​After splicing the first cross-attention features and the second cross-attention features, the visual features are converted to be aligned with the text feature dimension of the identification task video; then, the text feature and the text instruction of the identification task video are taken as the question and answer prompt information, and are input into the large language model in combination with the visual features to generate the video label of the identification task video.

[0326] Specifically, the conversion of the feature dimension can be realized by a fully connected layer, which can be located in the query converter as the output of the query converter, or can be an independent feature dimension conversion layer, which is not specifically limited herein. As shown in FIG. 13, after the first cross-attention and the second cross-attention in the query converter, the [video_emb] and the [video_txt] are dimensionally aligned through the FC layer. After splicing the first cross-attention features and the second cross-attention features, the visual features are converted to be aligned with the text feature dimension of the identification task video through the fully connected layer.

[0327] The fully connected layer is a component for dimensionally aligning different modal features. In the video label identification model of the present application, since the dimensions of the visual token and the text token output by the query converter are not necessarily the same, the dimensions of the visual features and the text features can be aligned through the fully connected layer. The fully connected layer can be divided as part of the query converter, connected to the output of the last Decoder in the query converter; or it can be regarded as an independent part between the query converter and the large language model. The calculation formula of the fully connected layer is: y = Wx + b, where x represents the spliced first cross-attention features and the second cross-attention features, W represents the weight matrix of the fully connected layer, b represents the bias vector, and y represents the output visual features.

[0328] The text feature [video_txt] of the identification task video refers to the encoding of the video text description information of the identification task video, and the video text description information includes but is not limited to video title, description text, OCR extracted key text, ASR generated subtitles or transcripts, comments and forum discussions, metadata information, etc. For details, please refer to the relevant description in the model training part, which will not be repeated here.

[0329] After aligning the visual features and the text features, the large language model can be input, wherein the text feature and the text instruction are taken as the question and answer prompt information prompt of the large language model. Since it is desired that the LLM outputs the video label, the text instruction here can adopt one of the following sentences:

[0330] "Can you summarize the above video with tags?", "Please output some key content tags based on the content of the above video.", "Based on the content of the above video, generate the main content tags.", etc. This paper does not make specific limitations.

[0331] In the above embodiment, by aligning the feature dimensions, the representation spaces of different modalities can be unified to effectively fuse and understand cross-modal information, reduce the representation difference between modalities, enable the LLM to process diverse data in the same semantic space, promote knowledge transfer, enhance generalization ability, and thus improve the recognition effect of video tags.

[0332] Optionally, the query converter includes a multi-layer stacked decoder; for the first decoder, the input features include the global feature sequence and the local feature sequence; for other decoders, the input features include the first cross-attention features and the second cross-attention features output by the previous decoder, and the specific implementation is the same as the training process, which can be referred to the above training part for relevant description, which will not be repeated here. On this basis, one optional embodiment of S94 is to identify the video tags of the recognition task video based on the first cross-attention features and the second cross-attention features output by the last decoder.

[0333] Still taking the example shown in FIG. 5, the input of 502 in Decoder1 is the global features output by the global memory unit, and the input of 503 in Decoder1 is the local features output by the local memory unit; the input of 502 in Decoder2 is the first cross-attention features output by 504 in Decoder1, and the input of 503 in Decoder2 is the second cross-attention features output by 505 in Decoder1; …; the input of 502 in Decoder5 is the first cross-attention features output by 504 in Decoder4, and the input of 503 in Decoder5 is the second cross-attention features output by 505 in Decoder4. Finally, the outputs of 504 and 505 in Decoder5 are spliced and dimensionally aligned by FC, and then input into the LLM combined with the prompt to obtain the corresponding video tags.

[0334] In the above-mentioned multi-layer stacked Decoder structure, between each layer, the global features and the local features are fused by self-attention and further compressed by cross-attention, so that more compact visual token sequences can be obtained. On this basis, the extraction of video tags can further reduce the computational amount of LLM and improve the recognition efficiency.

[0335] Referring to FIG. 14, it is an interaction logic diagram between a terminal device and a server in an embodiment of the present application.

[0336] The terminal device can install a video tag identification related client. The object can determine the identification task video based on the client, such as the TV series in FIG. 14. The terminal device sends the identification task video to the server. The server 120 side predeploys a trained video tag identification model. Then, the server uses the video tag identification method in the embodiment of the application to identify the video tag of the identification task video based on the video tag identification model. For details of the specific identification process, refer to the above embodiment, and the repeated parts will not be described here. Finally, the server can return the identified video tag to the terminal device, and the terminal device presents the video tag to the object through the client.

[0337] In addition to long videos, the identification task video can also be a live video. In this way, video tag identification can be performed every 5 minutes during live streaming, and so on.

[0338] In summary, the method of the application is simple, suitable for mainstream multi-modal models, can effectively compress visual features of various videos, reduce information loss, save costs, and improve machine review efficiency.

[0339] Based on the same inventive concept, the embodiment of the application also provides a video tag identification device. As shown in FIG. 15, it is a structural schematic diagram of a video tag identification device 1500, which can include:

[0340] The encoding compression module 1501 is configured to encode each video frame in the identification task video through the video tag identification model to form a global feature set including global features of each video frame obtained by encoding and a local feature set including local features of each video frame obtained by encoding; compress the global feature set based on the feature similarity between the global features in the global feature set to obtain a global feature sequence of a preset storage quantity; and compress the local feature set based on the feature similarity between the local features in the local feature set to obtain a local feature sequence of a preset storage quantity.

[0341] The fusion module 1502 is configured to splice the pre-trained global query feature and the local query feature, and then extract a self-attention feature using a self-attention mechanism. The global query feature and the local query feature are obtained by training the learnable query feature based on the sample video. The self-attention feature fuses key information in the global query feature and the local query feature.

[0342] The cross compression module 1503 is configured to extract a first cross-attention feature between the self-attention feature and each global feature in the global feature sequence and a second cross-attention feature between the self-attention feature and each local feature in the local feature sequence using a cross-attention mechanism.

[0343] The tag identification module 1504 is configured to identify a video tag of the identification task video based on the obtained first cross-attention features and the second cross-attention features.

[0344] Optionally, the video tag identification model comprises a query converter; and the cross-compression module 1503 is specifically configured to:

[0345] split the self-attention features into first self-attention features corresponding to the global features and second self-attention features corresponding to the local features;

[0346] extract, by the first cross-attention network in the query converter, the first cross-attention features between each global feature in the global feature sequence and the first self-attention features based on the cross-attention mechanism;

[0347] extract, by the second cross-attention network in the query converter, the second cross-attention features between each local feature in the local feature sequence and the second self-attention features based on the cross-attention mechanism.

[0348] Optionally, the query converter comprises a plurality of stacked decoders; for the first decoder, the input features comprise the global feature sequence and the local feature sequence; and for other decoders, the input features comprise the first cross-attention features and the second cross-attention features output by a previous decoder.

[0349] The tag identification module 1504 is specifically configured to:

[0350] identify the video tag of the identification task video based on the first cross-attention features and the second cross-attention features output by the last decoder.

[0351] Optionally, the video tag identification model comprises a large language model; and the tag identification module 1504 is specifically configured to:

[0352] concatenate the first cross-attention features and the second cross-attention features, and convert the concatenated features into visual features aligned with a text feature dimension of the identification task video;

[0353] input the text features of the identification task video and the text instruction as question and answer prompt information, and input the visual features into the large language model to generate the video tag of the identification task video.

[0354] Optionally, each local feature corresponds to an image block in a video frame; and the encoding compression module 1501 is specifically configured to:

[0355] store the global features of the video frame into the global memory unit and store the local features of the video frame into the local memory unit after encoding each video frame;

[0356] When the number of global features stored in the global memory unit reaches the preset storage number, the global features of at least two video frames are fused into the global feature of one video frame based on the feature similarity between the global features of the multiple groups of video frames in the global features to be stored and the global features stored in the global memory unit, so as to adjust the global features stored in the global memory unit, so that the number of global features is less than or equal to the preset storage number.

[0357] And, for each image block, when the number of corresponding local features stored in the local memory unit reaches the preset storage number, the corresponding local features of at least two video frames in the local memory unit are fused based on the feature similarity between the local features in the local memory unit, so that the number of local features stored in the local memory unit is less than or equal to the preset storage number.

[0358] Optionally, the device further comprises:

[0359] The reading module 1505 is configured to, before the cross compression module 1503 adopts the cross attention mechanism to extract the first cross attention feature between each global feature in the self-attention feature and the global feature sequence, and extract the second cross attention feature between each local feature in the self-attention feature and the local feature sequence, respectively, determine that the specified recognition time is reached, read the global feature sequence stored in the global memory unit, and read the local feature sequence stored in the local memory unit.

[0360] Optionally, the encoding compression module 1501 is further configured to fuse the target features of at least two video frames by any one of the following ways:

[0361] Based on the feature similarity, the most similar adjacent video frames of the target features in the memory unit are average-pooled;

[0362] Based on the feature similarity, the target features to be stored and the target features stored in the memory unit are clustered, and the target features of the clustered video frames are average-pooled;

[0363] Wherein, the target feature is a global feature or a local feature, and correspondingly, the memory unit is a global memory unit or a local memory unit.

[0364] Optionally, if there are multiple groups of the most similar adjacent video frames, the encoding compression module 1501 is specifically configured to:

[0365] The target features of the adjacent video frames with the longest storage time and the most similarity in the memory unit are average-pooled; or

[0366] The target features of each group of the most similar adjacent video frames in the memory unit are average-pooled, respectively.

[0367] Based on the same inventive concept, the embodiments of the present application also provide a training device of a video tag identification model. As shown in FIG. 16, which is a structural schematic diagram of the training device 1600 of the video tag identification model, the device can include:

[0368] The first feature extraction module 1601 is configured to encode each video frame in the sample video by the video tag identification model to be trained, to form a global feature set including global features of each video frame obtained by encoding, and a local feature set including local features of each video frame obtained by encoding; compress the global feature set based on feature similarities between the global features in the global feature set, to obtain a global feature sequence of a preset storage quantity; and compress the local feature set based on feature similarities between the local features in the local feature set, to obtain a local feature sequence of a preset storage quantity.

[0369] The second feature extraction module 1602 is configured to splice the first learnable query feature and the second learnable query feature, and extract a self-attention feature by using a self-attention mechanism; the self-attention feature fuses key information in the first learnable query feature and the second learnable query feature.

[0370] The third feature extraction module 1603 is configured to extract, by using a cross-attention mechanism, a first cross-attention feature between the self-attention feature and each global feature in the global feature sequence, and a second cross-attention feature between the self-attention feature and each local feature in the local feature sequence.

[0371] The parameter adjustment module 1604 is configured to adjust parameters of the video tag identification model according to differences between video tags and video text description information of the sample video predicted based on the obtained first cross-attention features and second cross-attention features, and corresponding sample tags, and to train the video tag identification model. The trained video tag identification model is used to extract video tags of a recognition task video.

[0372] Optionally, the video tag identification model includes a large language model and a query converter implemented based on a self-attention mechanism and a cross-attention mechanism, and the query converter is used to extract the first cross-attention feature and the second cross-attention feature; the device further includes:

[0373] The prediction module 1605 is configured to splice the first cross-attention features and the second cross-attention features in the first training stage, and convert them into visual features aligned with a text feature dimension of the sample video; input the first text instruction as a question and answer prompt information, and combine the visual features to input the large language model, to generate video text description information of the sample video.

[0374] In the second training stage, the first cross-attention feature and the second cross-attention feature are spliced and converted into visual features aligned with the text feature dimension of the sample video; the text feature and the second text instruction are taken as question and answer prompt information, and are input into the large language model in combination with the visual features to generate the video label of the sample video.

[0375] Optionally, the parameter adjustment module 1604 is specifically configured to:

[0376] In the first training stage, the query converter is updated in parameters according to the difference between the predicted video text description information and the video text description information in the sample label;

[0377] In the second training stage, the query converter and the large language model are updated in parameters according to the difference between the predicted video label and the video label in the sample label.

[0378] Optionally, the parameter adjustment module 1604 is further configured to update the parameters of the large language model in the following manner:

[0379] The original parameters of the large language model are fixed, and a low-rank adaptive matrix is added in the large language model;

[0380] The parameters of the low-rank adaptive matrix are updated to update the parameters of the large language model.

[0381] Thanks to the present application, when the visual features of the recognition task video are extracted by the video label recognition model, the video frames in the recognition task video are not needed to be extracted, but are coded. After coding, a global feature set composed of global features of each video frame and a local feature set composed of local features of each video frame can be obtained. On this basis, based on feature similarity, the obtained global feature set and local feature set are compressed respectively, specifically, the global features of part of the video frames are fused, and the local features of part of the video frames are fused, to obtain a global feature sequence and a local feature sequence with a preset storage quantity. Through the above operation, it can be ensured that the global feature sequence and the local feature sequence at this time can cover most of the content of the recognition task video. On this basis, the key information in the global query feature and the local query feature is fused through the self-attention mechanism, and then further compressed through the cross-attention mechanism to obtain a more compact visual feature sequence.

[0382] The above method can effectively compress the visual features of medium videos and long videos, reduce information loss, and improve the label recognition efficiency through compressed features, so as to realize efficient and accurate recognition of the video labels of medium videos and long videos.

[0383] For ease of description, the above parts are divided into modules (or units) according to functions and are described respectively. Of course, functions of the modules (or units) can be implemented in one or more software or hardware in the implementation of the present application.

[0384] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an integral module or unit that includes the functions of the module or unit.

[0385] After introducing the video tag identification method and device of the exemplary embodiments of the present application, next, the electronic device according to another exemplary embodiment of the present application is introduced.

[0386] Those skilled in the art can understand that each aspect of the present application can be implemented as a system, a method or a program product. Therefore, each aspect of the present application can be specifically implemented as follows: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system".

[0387] Based on the same inventive concept as the above method embodiments, an electronic device is also provided in the embodiments of the present application. In an embodiment, the electronic device can be a server, such as the server 120 shown in FIG. 1. In this embodiment, the structure of the electronic device can be as shown in FIG. 17, including a memory 1701, a communication module 1703 and one or more processors 1702.

[0388] The memory 1701 is used to store computer programs executed by the processor 1702. The memory 1701 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, programs required for running instant messaging functions and the like; the data storage area can store various instant messaging information and operation instruction sets and the like.

[0389] The memory 1701 can be a volatile memory, such as a random-access memory (RAM); the memory 1701 can also be a non-volatile memory, such as a read-only memory, a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); or the memory 1701 can be any other medium capable of storing and carrying desired computer programs in the form of instructions or data structures and capable of being accessed by a computer, but not limited to the above. The memory 1701 can be a combination of the above memories.

[0390] The processor 1702 can include one or more central processing units (CPUs) or digital processing units, etc. The processor 1702 is used to implement the above-mentioned video tag identification method when calling the computer program stored in the memory 1701.

[0391] The communication module 1703 is used to communicate with terminal devices and other servers.

[0392] The specific connection medium between the above-mentioned memory 1701, communication module 1703, and processor 1702 is not limited in the embodiments of the present application. In FIG. 17, the memory 1701 and the processor 1702 are connected through the bus 1704, which is described by a thick line in FIG. 17. The connection mode between other components is only schematically described, and is not limited. The bus 1704 can be divided into an address bus, a data bus, a control bus, etc. For the convenience of description, only one thick line is used to describe the bus 1704 in FIG. 17, but it is not described that there is only one bus or only one type of bus.

[0393] The memory 1701 stores a computer storage medium, and the computer storage medium stores computer executable instructions. The computer executable instructions are used to implement the video tag identification method or the training method of the video tag identification model according to the embodiments of the present application. The processor 1702 is used to execute the above-mentioned video tag identification method or the training method of the video tag identification model, as shown in FIG. 3 or FIG. 9.

[0394] In another embodiment, the electronic device can also be other electronic devices, such as the terminal device 110 shown in FIG. 1. In this embodiment, the structure of the electronic device can be as shown in FIG. 18, which includes a communication component 1810, a memory 1820, a display unit 1830, a camera 1840, a sensor 1850, an audio circuit 1860, a Bluetooth module 1870, a processor 1880, and the like.

[0395] The communication component 1810 is configured to communicate with a server. In some embodiments, a wireless fidelity (WiFi) module can be included, which belongs to a short-range wireless transmission technology. The electronic device can help the user to send and receive information through the WiFi module.

[0396] The memory 1820 can be used to store software programs and data. The processor 1880 executes various functions and data processing of the terminal device 110 by running the software programs or data stored in the memory 1820. The memory 1820 can include a high-speed random access memory, and can further include a non-volatile memory such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. The memory 1820 stores an operating system that enables the terminal device 110 to operate. In this application, the memory 1820 can store an operating system and various application programs, and can also store a computer program for executing the video tag identification method of the embodiment of the application.

[0397] The display unit 1830 can also be used to display information input by the user or information provided to the user, as well as the graphical user interface (GUI) of various menus of the terminal device 110. Specifically, the display unit 1830 can include a display screen 1832 arranged on the front of the terminal device 110. The display screen 1832 can be configured in the form of a liquid crystal display, a light-emitting diode, etc. The display unit 1830 can be used to display the video interface and the like in the embodiment of the application.

[0398] The display unit 1830 can also be used to receive input digital or character information, and generate signal input related to user settings and function control of the terminal device 110. Specifically, the display unit 1830 can include a touch screen 1831 arranged on the front of the terminal device 110, which can collect touch operations of the user thereon or therearound, such as clicking buttons, dragging scroll boxes, etc.

[0399] The touch screen 1831 can be overlaid on the display screen 1832, or the touch screen 1831 and the display screen 1832 can be integrated to realize the input and output functions of the terminal device 110. After integration, it can be referred to as a touch display screen. In this application, the display unit 1830 can display application programs and corresponding operation steps.

[0400] The camera 1840 can be used to capture still images, which a user can post through an application using the images taken by the camera 1840. The camera 1840 can be one or more. An object generates an optical image through a lens and projects the optical image onto a photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, which is then transmitted to the processor 1880 to convert into a digital image signal.

[0401] The terminal device can also include at least one sensor 1850, such as an acceleration sensor 1851, a distance sensor 1852, a fingerprint sensor 1853, a temperature sensor 1854. The terminal device can also be configured with a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, a light sensor, a motion sensor, and other sensors.

[0402] The audio circuit 1860, the speaker 1861, and the microphone 1862 can provide an audio interface between the user and the terminal device 110. The audio circuit 1860 can convert received audio data into an electrical signal, which is transmitted to the speaker 1861 and converted into a sound signal by the speaker 1861. The terminal device 110 can also be configured with a volume button for adjusting the volume of the sound signal. On the other hand, the microphone 1862 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1860 and converted into audio data, which is then output to the communication component 1810 for transmission to another terminal device 110, for example, or to the memory 1820 for further processing.

[0403] The Bluetooth module 1870 is used to interact with other Bluetooth devices with Bluetooth modules through Bluetooth protocols. For example, the terminal device can establish a Bluetooth connection with a wearable electronic device (e.g., a smart watch) that also has a Bluetooth module through the Bluetooth module 1870, thereby enabling data interaction.

[0404] The processor 1880 is a control center of the terminal device, which connects all parts of the terminal through various interfaces and lines, and performs various functions of the terminal device and processes data by running or executing software programs stored in the memory 1820 and calling data stored in the memory 1820. In some embodiments, the processor 1880 can include one or more processing units; the processor 1880 can also integrate an application processor and a baseband processor, wherein the application processor mainly processes operating systems, user interfaces, and application programs, and the baseband processor mainly processes wireless communication. It can be understood that the above-mentioned baseband processor can also not be integrated into the processor 1880. In the present application, the processor 1880 can run an operating system, an application program, a user interface display and a touch response, and a video tag identification method or a training method of a video tag identification model of the embodiments of the present application. In addition, the processor 1880 is coupled with the display unit 1830.

[0405] In some optional embodiments, various aspects of the video tag identification method provided by the present application can also be implemented in the form of a program product, which includes a computer program for causing an electronic device to perform the steps of the video tag identification method according to various exemplary embodiments of the present application described above when the program product is run on the electronic device, for example, the electronic device can perform the steps as shown in FIG. 3 or FIG. 9.

[0406] The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0407] The program product of the embodiments of the present application can adopt a portable compact disk read-only memory (CD-ROM) and include a computer program, and can be run on an electronic device. However, the program product of the present application is not limited to this, and in this document, the readable storage medium can be any tangible medium containing or storing a program, which can be used by or in conjunction with a command execution system, device or apparatus.

[0408] A readable signal medium can include a data signal traveling in baseband or propagated by a carrier wave appropriate for a communication network. Such a propagated signal can take a wide variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A readable medium can also be any medium that can be read by a machine, including, but not limited to, storage media or communication media. The communication media can include, but is not limited to, wired media such as twisted pair, coaxial, or fiber optics, and / or wireless media such as acoustic, RF, infrared, or other suitable combinations.

[0409] The computer program can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device. The computer-readable medium can be a computer- readable storage medium or a computer-readable signal medium.

[0410] The computer program can be implemented in any of a variety of programming languages, including object-oriented programming languages such as Java, C++, and the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer program can be executed in whole or in part on the user electronic device, as a stand-alone software package, partly on the user electronic device and partly on a remote electronic device, or entirely on the remote electronic device or server. In the latter scenario, the remote electronic device can be connected to the user electronic device through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external electronic device, such as through the Internet using an Internet Service Provider (ISP).

[0411] In summary, the application provides a video tag identification method, device, equipment, computer readable storage medium and computer program product. Each video frame in the identification task video is encoded by a video tag identification model to form a global feature set and a local feature set. The traditional sparse frame extraction method is difficult to cover most of the content for medium and long videos, while the method encodes each frame to comprehensively capture video information. Then, the two feature sets are compressed based on feature similarity to obtain a global feature sequence and a local feature sequence with a preset storage quantity. This process reduces data redundancy, avoids storing a large number of similar features, saves storage space and computing resources. After splicing the global query feature and the local query feature obtained by pre-training, a self-attention mechanism is used to extract a self-attention feature. The self-attention feature fuses key information in the global query feature and the local query feature, so that the model can focus on the global and capture local details, and the understanding ability of complex video features is enhanced. Then, a cross-attention mechanism is used to extract cross-attention features between each feature in the self-attention feature and the global feature sequence and the local feature sequence, respectively, to further compress the features and obtain a more compact visual feature sequence. Finally, the cross-attention features are used to identify the video tags of the identification task video. This method effectively compresses the visual features of medium and long videos, reduces information loss, improves tag identification efficiency, and realizes efficient and accurate identification of video tags of medium and long videos.

[0412] Further, if the video tag identification model includes a query converter, when the cross-attention mechanism is used to extract the cross-attention features, the self-attention feature is first split into a first self-attention feature corresponding to the global feature and a second self-attention feature corresponding to the local feature, and then the first cross-attention network and the second cross-attention network in the query converter are used to extract the first cross-attention feature and the second cross-attention feature based on the cross-attention mechanism. This splitting process allows the model to focus on video features from global and local perspectives respectively, avoids confusion of information, more accurately captures different levels of video features, improves the accuracy and relevance of feature extraction, and further improves the effect of video tag identification.

[0413] When the query converter comprises a multi-layer stacked decoder, the input features of the first decoder comprise a global feature sequence and a local feature sequence, and the input features of other decoders comprise first cross-attention features and second cross-attention features output by the previous decoder. The video tags of the identification task video are identified based on the first cross-attention features and the second cross-attention features output by the last decoder. The multi-layer stacked decoder structure enables the global features and the local features to be fused between each layer through a self-attention mechanism and further compressed through a cross-attention mechanism, so that the features are constantly refined to obtain a more compact visual token sequence. This reduces the computational load of the large language model because the input features are more concise and effective, and the large language model can process these features more quickly, thereby improving the identification efficiency.

[0414] If the video tag identification model comprises a large language model, when identifying the video tags, the first cross-attention features and the second cross-attention features are spliced and then converted into visual features that are aligned with the text feature dimension of the identification task video, and then the text features of the identification task video and the text instructions are taken as question and answer prompts, and the visual features are input into the large language model to generate the video tags. The feature dimensions of different modalities may be different, and the dimensions are aligned through a fully connected layer, which unifies the representation spaces of different modalities and reduces the representation differences between modalities. In this way, the large language model can process diverse data in the same semantic space, promote knowledge transfer, enhance the generalization ability of the model, and thus improve the identification effect of the video tags.

[0415] When each local feature corresponds to an image block in a video frame, when compressing the global feature set and the local feature set, the global features of a video frame are stored in the global memory unit and the local features are stored in the local memory unit every time a video frame is encoded. When the number of global features stored in the global memory unit reaches the preset storage number, at least two global features of video frames are fused into a global feature of a video frame based on feature similarity; for each image block, when the number of corresponding local features stored in the local memory unit reaches the preset storage number, at least two local features of video frames at the image block are fused into a local feature of a video frame at the image block based on feature similarity. The introduction of two memory units to store and compress the global features and the local features of the video frame sequence enables the global memory unit to capture macro information of the video and the local memory unit to focus on detail changes, and the two complement each other to ensure the comprehensiveness of the information and reduce redundancy, thereby improving storage efficiency and computational efficiency.

[0416] Before extracting the cross-attention feature by using the cross-attention mechanism, it is determined whether the specified recognition time is reached, and the global feature sequence stored in the global memory unit and the local feature sequence stored in the local memory unit are read. By setting the specified recognition time, unnecessary resource waste caused by the model continuously identifying the video label is avoided. For example, for medium and long videos, prediction can be performed after the input of the video frame sequence is completed, and for live scenes, prediction can be performed once every N minutes. The specified recognition time can be flexibly set according to actual needs, so that the identification of the video label is more flexible.

[0417] Based on the feature similarity between the global features of multiple groups of adjacent video frames in the global features stored in the global memory unit and the global features to be stored, the global features of at least one group of most similar adjacent video frames are averaged and pooled by group to obtain the global features of one video frame in each group. This average pooling method based on feature similarity combines similar global features, reduces data redundancy, and saves the space occupied by data. At the same time, the average pooling operation can also reduce the influence of noise and improve the stability of the features, because it integrates the information of multiple similar features, so that the features can better represent the overall features of the video.

[0418] If there are multiple groups of global features of the most similar adjacent video frames in the global features of multiple groups of adjacent video frames, the global features of the most similar adjacent video frames in the multiple groups of global features are averaged and pooled, including the global feature with the longest storage duration and the most similar group of adjacent video frames. According to the storage duration, the features of the video frames with longer storage duration are compressed, which can not only reduce data redundancy and save space occupied by data, but also provide more space for new data. Because the features with long storage time may not be able to well reflect the latest state of the video, timely compression of these features can ensure the timeliness of the video label recognition result, so that the model can process new video frame information more timely.

[0419] Based on the feature similarity between the global features of multiple groups of video frames in the global features stored in the global memory unit and the global features to be stored, at least part of the global features in the global features stored in the global memory unit and the global features to be stored are clustered, and the global features of at least two video frames in the same category are fused into the global features of one video frame in the same category. Based on the feature similarity, the global features are clustered, which can group similar video frame features and reduce redundant storage. The clustered features are more representative and can retain key information of the video, while reducing unnecessary calculations and improving storage and processing efficiency.

[0420] Based on the feature similarity between the local features stored in the local memory unit and the local features to be stored in the local feature of the plurality of groups of adjacent video frames at the image block, the local features of at least one group of the most similar adjacent video frames at the image block are averaged and pooled by group to obtain the local features of one video frame in each group. This way, the local features are compressed, reducing the redundancy of the local features and improving the storage efficiency. At the same time, the local features after average pooling can better represent the features of the image block, which helps the model to more accurately identify the local details in the video, thereby improving the accuracy of video label recognition.

[0421] If there are multiple groups of local features of the most similar adjacent video frames at the image block in the local features of the plurality of groups of adjacent video frames at the image block, the local features of the plurality of groups of the most similar adjacent video frames at the image block are averaged and pooled by group, including the local features with the longest storage duration and the local features of at least one group of the most similar adjacent video frames at the image block. According to the storage duration, the local features are compressed to free up space for new local features, ensuring the timeliness of video label recognition. At the same time, the redundancy of the local features is reduced, so that the model can focus more on valuable local features, improving the ability to identify local details in the video.

[0422] Based on the feature similarity between the local features stored in the local memory unit and the local features to be stored in the local feature of the plurality of groups of adjacent video frames at the image block, the local features stored in the local memory unit and at least part of the local features to be stored are clustered, and the local features of at least two video frames at the image block under the same category are fused into the local features of one video frame at the image block under the same category. By clustering local features, similar local features can be grouped together, reducing repeated storage and improving storage efficiency. The clustered local features can better reflect the common features of the local area of the video, which helps the model to more accurately identify the local objects and scenes in the video, and improves the effect of video label recognition.

[0423] For the training method, device, equipment, computer readable storage medium and computer program product of the video tag identification model, the electronic device encodes each video frame in the sample video through the to-be-trained video tag identification model to form a global feature set and a local feature set, compresses the two feature sets based on feature similarity, and obtains a global feature sequence and a local feature sequence with a preset storage quantity. After splicing the first learnable query feature and the second learnable query feature, the self-attention mechanism is used to extract a self-attention feature, and then the cross-attention mechanism is used to extract cross-attention features. Finally, according to the difference between the video tag and the video text description information of the sample video predicted based on the cross-attention features, the parameters of the video tag identification model are adjusted. This phased training method focuses on the pre-training of the query converter in the first phase, so that the query converter can better extract visual features. In the second phase, the query converter and the large language model are adjusted synchronously, and the model is trained based on the video tag, so that the model can better learn the features of the video and improve the performance of the model, thereby more accurately identifying the video tag in actual application.

[0424] In the first training phase, after splicing the first cross-attention features and the second cross-attention features, the visual features are converted to align with the text feature dimension of the sample video. The first text instruction is used as the question and answer prompt information, and the visual features are input into the large language model to generate the video text description information of the sample video. This phase focuses on training the query converter, and the parameters of the query converter are adjusted based on the video text description information, which can improve the accuracy of the query converter in extracting visual features and ensure that the extracted visual features can more accurately describe the main content of the video. Because the query converter continuously learns how to associate visual features with text description information in this phase, it can more accurately extract features related to the content of the video.

[0425] In the second training phase, after splicing the first cross-attention features and the second cross-attention features, the visual features are converted to align with the text feature dimension of the sample video. The text features and the second text instruction are used as the question and answer prompt information, and the visual features are input into the large language model to generate the video tag of the sample video. In this phase, the query converter and the large language model are adjusted synchronously, and the parameters of the query converter and the large language model are adjusted based on the video tag, so that the large language model can generate more accurate video tags based on the visual features extracted by the query converter, to improve the accuracy of the video tags extracted by the model. Through the training in this phase, the model can better combine visual features and text features, thereby more accurately identifying video tags.

[0426] In the first training stage, the query converter is updated according to the difference between the predicted video text description information and the video text description information in the sample label; in the second training stage, the query converter and the large language model are updated according to the difference between the predicted video label and the video label in the sample label. By updating the model parameters according to the difference between the different prediction results and the sample labels in stages, the model can be gradually optimized. In the first stage, the query converter is continuously adjusted to better extract visual features, and in the second stage, the query converter and the large language model are optimized together, so that the model can more accurately identify the video label, improving the performance of the model and the accuracy of video label recognition.

[0427] In addition, before encoding the recognition task video, the electronic device can select a suitable video frame segmentation method according to the content complexity and feature distribution of the video. If the video content is relatively simple and the feature distribution is relatively uniform, the electronic device can segment the video frames in the recognition task video into the same number of image blocks, which can ensure the consistency of feature extraction of each video frame and facilitate subsequent processing. For example, for some landscape videos, the picture elements change little, and the same number of image block segmentation methods can efficiently extract global and local features, reduce unnecessary calculations, and improve processing efficiency. If the video content is complex and the feature distribution is uneven, the electronic device can segment the video frames in the recognition task video according to a pyramid segmentation method to obtain corresponding image blocks, and the number of image blocks of different video frames is not the same. The pyramid segmentation can provide more detailed feature extraction in areas with large changes in video content and reduce unnecessary calculations in areas with relatively stable content. For example, for action videos containing fast moving objects and complex scene switching, the pyramid segmentation can better capture key features in the video, improve the accuracy of feature extraction, and thus improve the effect of video label recognition.

[0428] In training the video tag identification model, for the training of large language models, the low rank adaptive matrix (LoRA) method is adopted. LoRA is a technology for fine-tuning pre-trained models, and its main goal is to adapt to specific tasks without modifying the original model parameters by introducing additional low-rank matrices, thereby reducing the resources and time required for fine-tuning. In large language models, assuming that the weight matrix of a certain linear layer is, two additional small matrices and are introduced in LoRA, by updating the parameters of the matrices and, without changing the original weight matrix of the large language model, the behavior of the model is effectively adjusted to adapt to the specific task requirements. Since the dimensions of and are relatively small, the number of parameters that need to be updated is greatly reduced, thereby reducing the resources and time required for fine-tuning. At the same time, by adjusting the value of, the expressive power and computational complexity of LoRA can be controlled. Compared with the fine-tuning method of the complete model, the LoRA model occupies less storage space, saves storage resources, and improves the efficiency of model training.

[0429] It should be noted that although several units or sub-units of the apparatus are mentioned in the foregoing detailed description, such division is merely exemplary and not mandatory. Indeed, according to an implementation of the present application, the features and functionalities of two or more units described above can be embodied in one unit. Conversely, the features and functionalities of one unit described above can be further divided into units embodied by several units.

[0430] Furthermore, although the operations of the methods of the present application are described in a particular, sequential order, this order is not meant to be a limitation and is not intended to imply that the described operations are to be performed in the same order, or that all described operations are to be performed, in order to realize the benefits of the present application. Additionally or alternatively, certain of the steps can be rendered in inoperative, combined into a single step, and / or further divided into sub-steps.

[0431] Those skilled in the art will appreciate that embodiments of the present application can be supplied as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product embodied in one or more computer-usable storage media (including, but not limited to, disk memory, CD-ROMs, optical storage media, etc.) having computer usable program code embodied therein.

[0432] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0433] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks.

[0434] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks.

[0435] The technical features of the above embodiments can be combined in any manner. For the sake of brevity, not all possible combinations of the technical features described above are explicitly described, but it should be understood that the scope of the specification includes all possible combinations.

[0436] The above embodiments only express several implementation manners of the present application, and the description is specific and detailed, but it should not be understood as a limitation on the patent scope of the present application. It should be pointed out that, for those skilled in the art, some modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the patent protection scope of the present application should be subject to the appended claims.

Claims

1. A video tag identification method, executed by an electronic device, comprising: encoding each video frame in an identification task video by a video tag identification model to form a global feature set comprising global features of the encoded video frames and a local feature set comprising local features of the encoded video frames; compressing the global feature set based on feature similarities between global features in the global feature set to obtain a global feature sequence of a preset storage number, and compressing the local feature set based on feature similarities between local features in the local feature set to obtain a local feature sequence of a preset storage number; after splicing a global query feature and a local query feature obtained by pre-training, extracting a self-attention feature using a self-attention mechanism, wherein the global query feature and the local query feature are obtained by training learnable query features based on sample videos, and the self-attention feature fuses key information in the global query feature and the local query feature; extracting, using a cross-attention mechanism, a first cross-attention feature between the self-attention feature and each global feature in the global feature sequence, and extracting, using the cross-attention mechanism, a second cross-attention feature between the self-attention feature and each local feature in the local feature sequence; and identifying a video tag of the identification task video based on the obtained first cross-attention features and second cross-attention features.

2. The method of claim 1, wherein the video tag identification model comprises a query converter, and the extracting, using a cross-attention mechanism, a first cross-attention feature between the self-attention feature and each global feature in the global feature sequence, and the extracting, using the cross-attention mechanism, a second cross-attention feature between the self-attention feature and each local feature in the local feature sequence, comprises: splitting the self-attention feature into a first self-attention feature corresponding to the global features and a second self-attention feature corresponding to the local features; extracting, using a first cross-attention network in the query converter, a first cross-attention feature between the first self-attention feature and each global feature in the global feature sequence based on a cross-attention mechanism; extracting, using a second cross-attention network in the query converter, a second cross-attention feature between the second self-attention feature and each local feature in the local feature sequence based on a cross-attention mechanism.

3. The method of claim 2, wherein the query converter comprises a plurality of stacked decoders, input features for a first decoder comprise the global feature sequence and the local feature sequence, and input features for other decoders comprise the first cross-attention features and the second cross-attention features output by a previous decoder; the identifying a video tag of the identification task video based on the obtained first cross-attention features and second cross-attention features comprises: identifying the video tag of the identification task video based on the first cross-attention features and the second cross-attention features output by a last decoder. ​ 4. The method of any one of claims 1 to 3, wherein the video tag identification model comprises a large language model; and wherein identifying the video tag of the identification task video based on the obtained first cross-attention features and the obtained second cross-attention features comprises: concatenating the first cross-attention features and the second cross-attention features, and converting the concatenated first cross-attention features and the concatenated second cross-attention features into visual features that are aligned with a text feature dimension of the identification task video; and inputting the text features of the identification task video and the text instruction as question and answer prompt information into the large language model in combination with the visual features to generate the video tag of the identification task video.

5. The method of any one of claims 1 to 4, wherein each local feature corresponds to an image patch in a video frame; and wherein compressing the global feature set based on the feature similarity between the global features in the global feature set, and obtaining the pre-set storage number of global feature sequences, and compressing the local feature set based on the feature similarity between the local features in the local feature set, and obtaining the pre-set storage number of local feature sequences, comprises: storing the global features of each video frame in a global memory unit and storing the local features of each video frame in a local memory unit as each video frame is encoded; and when the number of global features stored in the global memory unit reaches the pre-set storage number, fusing the global features of at least two video frames into the global features of one video frame based on the feature similarity between the global features of the at least two video frames and the global features of the video frames to be stored, so as to adjust the global features stored in the global memory unit, so that the number of global features is less than or equal to the pre-set storage number; and for each image patch, when the number of corresponding local features stored in the local memory unit reaches the pre-set storage number, fusing the local features of at least two video frames at the image patch into the local features of one video frame at the image patch based on the feature similarity between the local features of the at least two video frames at the image patch and the local features of the video frames to be stored, so as to adjust the local features stored in the local memory unit, so that the number of local features is less than or equal to the pre-set storage number.

6. The method of claim 5, wherein before the first cross-attention features between the self-attention features and each global feature in the global feature sequence and the second cross-attention features between the self-attention features and each local feature in the local feature sequence are extracted using the cross-attention mechanism, the method further comprises: determining that a specified identification time is reached, and reading the global feature sequence stored in the global memory unit and reading the local feature sequence stored in the local memory unit.

7. The method of claim 5 or 6, wherein fusing the global features of at least two video frames into the global features of one video frame based on the feature similarity between the global features of the at least two video frames and the global features of the video frames to be stored comprises: ​ ​ ​ ​ ​ ​ based on the feature similarity between the global features of the plurality of groups of adjacent video frames in the global features stored in the global memory unit and the global features to be stored, performing average pooling on the global features of the at least one group of the most similar adjacent video frames to obtain the global feature of one video frame in each group.

8. The method of claim 7, if there are a plurality of groups of the most similar adjacent video frames in the global features of the plurality of groups of adjacent video frames, the performing average pooling on the global features of the at least one group of the most similar adjacent video frames to obtain the global feature of one video frame in each group comprises: performing average pooling on the global features of the at least one group of the most similar adjacent video frames including the longest stored global feature and the most similar adjacent video frames.

9. The method of claim 5 or 6, the fusing the global features of at least two video frames into the global feature of one video frame based on the feature similarity between the global features of the plurality of groups of video frames in the global features stored in the global memory unit and the global features to be stored comprises: based on the feature similarity between the global features of the plurality of groups of video frames in the global features stored in the global memory unit and the global features to be stored, clustering at least part of the global features in the global features stored in the global memory unit and the global features to be stored, and fusing the global features of at least two video frames clustered into one category into the global feature of one video frame in the one category.

10. The method of any one of claims 5 to 9, the fusing the local features of at least two video frames at the image block into the local feature of one video frame at the image block based on the feature similarity between the local features of the plurality of groups of video frames at the image block in the local features stored in the local memory unit and the local features to be stored comprises: based on the feature similarity between the local features of the plurality of groups of adjacent video frames at the image block in the local features stored in the local memory unit and the local features to be stored, performing average pooling on the local features of the at least one group of the most similar adjacent video frames at the image block to obtain the local feature of one video frame in each group.

11. The method of claim 10, if there are a plurality of groups of the most similar adjacent video frames at the image block in the local features of the plurality of groups of adjacent video frames at the image block, the performing average pooling on the local features of the at least one group of the most similar adjacent video frames at the image block comprises: performing average pooling on the local features of the at least one group of the most similar adjacent video frames at the image block including the longest stored local feature and the most similar adjacent video frames at the image block.

12. The method of any one of claims 5 to 9, wherein the fusing of the local features of at least two video frames at the image block into the local features of one video frame at the image block based on the feature similarities between the local features of the multiple groups of video frames at the image block in the stored local features and the to-be-stored local features comprises: clustering at least some of the stored local features and the to-be-stored local features in the local memory unit based on the feature similarities between the local features of the multiple groups of video frames at the image block in the stored local features and the to-be-stored local features, and fusing the local features of at least two video frames at the image block that are clustered into one category into the local features of one video frame at the image block in the one category.

13. A method for training a video label recognition model, implemented by an electronic device, the method comprising: encoding each video frame in a sample video by a video label recognition model to be trained, to form a global feature set comprising global features of the encoded video frames, and a local feature set comprising local features of the encoded video frames; compressing the global feature set based on feature similarities between the global features in the global feature set, to obtain a global feature sequence of a preset storage quantity; and compressing the local feature set based on feature similarities between the local features in the local feature set, to obtain a local feature sequence of a preset storage quantity; after splicing the first learnable query feature and the second learnable query feature, extracting a self-attention feature using a self-attention mechanism; the self-attention feature fuses key information in the first learnable query feature and the second learnable query feature; extracting, using a cross-attention mechanism, a first cross-attention feature between each global feature in the global feature sequence and the self-attention feature, and a second cross-attention feature between each local feature in the local feature sequence and the self-attention feature; and based on the differences between the predicted video label and video text description information of the sample video and the corresponding sample label, adjusting parameters of the video label recognition model, and using the trained video label recognition model to extract video labels of a recognition task video.

14. The method of claim 13, wherein the video label recognition model comprises a large language model and a query converter implemented based on a self-attention mechanism and a cross-attention mechanism, the query converter being configured to extract the first cross-attention feature and the second cross-attention feature; and wherein the predicting of the video label and the video text description information of the sample video based on the first cross-attention feature and the second cross-attention feature comprises: ​ In the first training stage, after splicing the first cross-attention features and the second cross-attention features, the visual features are converted to be aligned with the text feature dimension of the sample video; the first text instruction is taken as the question and answer prompt information, and the visual features are input into the large language model to generate the video text description information of the sample video; In the second training stage, after splicing the first cross-attention features and the second cross-attention features, the visual features are converted to be aligned with the text feature dimension of the sample video; the text features and the second text instruction are taken as the question and answer prompt information, and the visual features are input into the large language model to generate the video label of the sample video.

15. The method of claim 14, wherein the parameter adjustment of the video label identification model according to the difference between the predicted video label and video text description information of the sample video based on the obtained first cross-attention features and second cross-attention features comprises: In the first training stage, the query converter is updated according to the difference between the predicted video text description information and the video text description information in the sample label; In the second training stage, the query converter and the large language model are updated according to the difference between the predicted video label and the video label in the sample label.

16. A video label identification device, comprising: An encoding and compression module configured to encode each video frame in an identification task video by a video label identification model to form a global feature set comprising global features of each encoded video frame and a local feature set comprising local features of each encoded video frame; compress the global feature set based on feature similarity between global features in the global feature set to obtain a global feature sequence of a preset storage quantity; and compress the local feature set based on feature similarity between local features in the local feature set to obtain a local feature sequence of a preset storage quantity. A fusion module configured to splice a pre-trained global query feature and a local query feature to extract a self-attention feature using a self-attention mechanism; the global query feature and the local query feature are obtained by training a learnable query feature based on a sample video; the self-attention feature fuses key information in the global query feature and the local query feature. A cross-compression module configured to extract a first cross-attention feature between the self-attention feature and each global feature in the global feature sequence and a second cross-attention feature between the self-attention feature and each local feature in the local feature sequence using a cross-attention mechanism. And A label identification module configured to identify a video label of the identification task video based on the obtained first cross-attention features and second cross-attention features.

17. A video label identification device, comprising: The first feature extraction module is configured to encode each video frame in the sample video by using the video label recognition model to be trained, to form a global feature set including global features of each video frame obtained by encoding and a local feature set including local features of each video frame obtained by encoding; compress the global feature set based on feature similarities between global features in the global feature set, to obtain a global feature sequence with a preset storage quantity; and compress the local feature set based on feature similarities between local features in the local feature set, to obtain a local feature sequence with a preset storage quantity. The second feature extraction module is configured to extract a self-attention feature by using a self-attention mechanism after splicing the first learnable query feature and the second learnable query feature; the self-attention feature fuses key information in the first learnable query feature and the second learnable query feature. The third feature extraction module is configured to extract, by using a cross-attention mechanism, a first cross-attention feature between the self-attention feature and each global feature in the global feature sequence and a second cross-attention feature between the self-attention feature and each local feature in the local feature sequence. The parameter adjustment module is configured to adjust parameters of the video label recognition model based on differences between video label and video text description information of the sample video predicted based on each first cross-attention feature and each second cross-attention feature, and a corresponding sample label, and to train the video label recognition model to extract video labels of recognition task videos. The memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method in any one of claims 1-15.

18. An electronic device comprising a processor and a memory, wherein, 19.A computer readable storage medium comprising a computer program, the computer program being configured to cause an electronic device to perform the steps of the method in any one of claims 1-15 when the computer program is run on the electronic device. 20.A computer program product comprising a computer program stored in a computer readable storage medium, and when a processor of an electronic device reads the computer program from the computer readable storage medium, the processor executes the computer program to cause the electronic device to perform the steps of the method in any one of claims 1-15. ​

Images