Video tag recognition model training method, video tag recognition method and device

Abstract

Provided are a video label recognition model training method, a video label recognition method and a device, belonging to the field of artificial intelligence technology, comprising: obtaining a first training sample set, each training sample comprising a sample video, a label of the sample video at time t and text description information of the sample video, determining for each training sample the visual features of the sample video of the training sample, the visual features of the sample video being obtained by clustering and updating K class center features according to the visual features of the sample video from time t0 to time t, the K class center features being obtained by clustering the visual features of the sample video before time t0, training a video label recognition model according to the visual features of the sample video, the text description information of the sample video, the first predicted label of the sample video and a first preset paradigm instruction, the video label recognition model comprising a visual feature encoder, a storage queue, a pre-trained alignment module and a recognition model.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a video label recognition model training method, a video label recognition method and a device. Background Art

[0002] With the development of network technology, multimedia data such as videos have become the main body of big data. Video tag recognition is an important part of video content features. By automatically generating tags for massive videos, machines can provide video content features of different granularities for downstream content distribution links, improve the efficiency of content distribution, and reduce manual review costs.

[0003] At present, a label recognition model is used to perform video label recognition. Specifically, a short video is sparsely extracted. For example, N frames are extracted at fixed intervals to obtain a video frame sequence. Then, the visual features of each video frame in the video frame sequence are extracted. The extracted visual features of multiple video frames are spliced ​​together and input into the label recognition model to output the label of the video segment.

[0004] However, the short videos to which the above method can be applied are generally no longer than 30 seconds. For medium and long videos, the video frame sequence obtained by sparse frame extraction cannot cover most of the content of the video, which will cause information loss and thus lead to low accuracy and recall rate of video tag recognition. For video media such as video streams (such as live broadcast scenes), the length of the video stream is unpredictable before the end, making it difficult to use the above method. Summary of the invention

[0005] The embodiments of the present application provide a video tag recognition model training method, a video tag recognition method and a device, which can reduce information loss and improve the accuracy of video tag recognition.

[0006] In a first aspect, an embodiment of the present application provides a video tag recognition model training method, comprising:

[0007] Acquire a first training sample set, where each training sample in the first training sample set includes a sample video, a label of the sample video at time t, and text description information of the sample video, where the time t is less than or equal to the duration of the sample video;

[0008] For each training sample, determine the visual features of the sample video of the training sample, wherein the visual features of the sample video are obtained by clustering and updating K class center features stored in a storage queue according to the visual features of the sample video from time t0 to the time t, and the K class center features are obtained by clustering the visual features of the sample video before the time t0, wherein K is a positive integer, the time t0 is a preset value, and t0 is less than t;

[0009] According to the visual features of the sample video, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction, the video label recognition model is trained until the training stop condition is met to obtain a trained video label recognition model, the video label recognition model includes a visual feature encoder, the storage queue, a pre-trained alignment module and a recognition model, and the first predicted label of the sample video is the result of the last label recognition performed by the video label recognition model on the sample video at time t.

[0010] In a second aspect, an embodiment of the present application provides a video tag recognition method, including:

[0011] Acquire a video to be identified and text description information of the video to be identified, wherein the video to be identified is input in the form of a video frame sequence;

[0012] Inputting the video to be identified before time t into a visual feature encoder, and outputting a first visual feature;

[0013] Clustering the first visual features by using a clustering algorithm to obtain K class center features, and writing the K class center features into a storage queue;

[0014] Determine whether the label recognition condition is met, and determine the predicted label of the video to be recognized based on the K class center features stored in the storage queue, the text description information of the video to be recognized, the first preset paradigm instruction and the trained video label recognition model, wherein the video label recognition model includes the visual feature encoder, the storage queue, the alignment module and the recognition model.

[0015] In a third aspect, an embodiment of the present application provides a video tag recognition model training device, comprising:

[0016] An acquisition module, configured to acquire a first training sample set, wherein each training sample in the first training sample set includes a sample video, a label of the sample video at time t, and text description information of the sample video, wherein the time t is less than or equal to the duration of the sample video;

[0017] A determination module, for determining, for each training sample, a visual feature of a sample video of the training sample, wherein the visual feature of the sample video is obtained by clustering and updating K class center features stored in a storage queue according to the visual features of the sample video from time t0 to the time t, wherein the K class center features are obtained by clustering the visual features of the sample video before the time t0, wherein K is a positive integer, the time t0 is a preset value, and t0 is less than t;

[0018] A training module is used to train a video label recognition model according to the visual features of the sample video, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction until the training stop condition is met to obtain a trained video label recognition model, wherein the video label recognition model includes a visual feature encoder, the storage queue, a pre-trained alignment module and a recognition model, and the first predicted label of the sample video is the result of the last label recognition performed by the video label recognition model on the sample video at time t.

[0019] In one embodiment, the determining module is used to:

[0020] Inputting the video of the sample video before the time t0 into the visual feature encoder, and outputting a first visual feature;

[0021] Clustering the first visual features by using a clustering algorithm to obtain K class center features, and writing the K class center features into the storage queue;

[0022] Input the video of the sample video from time t0 to time t into the visual feature encoder, and output the visual features of the video of the sample video from time t0 to time t;

[0023] According to the visual features of the sample video from time t0 to time t, the K class center features are clustered and updated, and the K class center features obtained after the update are determined as the visual features of the sample video of the training sample.

[0024] In one embodiment, the determining module is used to:

[0025] According to the video features of the sample video at time t0, clustering and updating the K class center features, obtaining the updated K class center features and writing them into the storage queue;

[0026] The K class center features in the storage queue are clustered and updated according to the video features of the sample video at each moment between time t0 and time t and the visual features of the video at time t, and the updated K class center features are obtained and written into the storage queue.

[0027] In one embodiment, the video features of the sample video at time t0 include image block features of N video frames, where N is the number of image blocks, and the determination module is specifically configured to:

[0028] For each image block feature of a video frame in the video features of the sample video at time t0, determine a corresponding target class center feature, wherein the target class center feature is a class center feature having the smallest Euclidean distance with the image block feature of the video frame;

[0029] The target class center features are clustered and updated according to preset class center update parameters and the image block features of the video frame.

[0030] In one embodiment, the determining module is specifically used to:

[0031] The sum of the target class center feature and the first result is determined as the updated target class center feature, where the first result is the product of the difference between the image block feature of the video frame and the target class center feature and the class center update parameter.

[0032] In one embodiment, the training module is used to: input the visual features of the sample video into the alignment module, and output the aligned visual features;

[0033] Inputting the aligned visual features, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction into the recognition model, and outputting the predicted label of the sample video at time t;

[0034] According to the label of the sample video at time t and the predicted label of the sample video at time t, the parameters of the alignment module and the parameters of the recognition model are adjusted until the training stop condition is met.

[0035] In one embodiment, the training module is used to:

[0036] Constructing a loss function according to the label of the sample video at time t and the predicted label of the sample video at time t;

[0037] According to the loss function, back propagation is used to adjust the parameters of the alignment module and the parameters of the recognition model until a training stop condition is met.

[0038] In one embodiment, the video tag recognition model further includes a low-order adaptive LoRA module of the model, and the training module is used to:

[0039] Inputting the visual features of the sample video into the alignment module, and outputting the aligned visual features;

[0040] Inputting the aligned visual features, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction into the recognition model, and outputting the predicted label of the sample video at time t;

[0041] The parameters of the recognition model are fixed, and the parameters of the alignment module and the parameters of the LoRA module are adjusted according to the label of the sample video at time t and the predicted label of the sample video at time t until the training stop condition is met.

[0042] In one embodiment, the training module is used to:

[0043] Constructing a loss function according to the label of the sample video at time t and the predicted label of the sample video at time t;

[0044] According to the loss function, back propagation adjusts the parameters of the alignment module and the parameters of the LoRA module until the training stop condition is met.

[0045] In one embodiment, the training module is further used to:

[0046] Acquire a second training sample set, wherein each training sample in the second training sample set includes a sample video and text description information of the sample video;

[0047] The parameters of the visual feature encoder and the parameters of the recognition model are fixed, and an alignment module is trained according to the second training sample set to obtain the pre-trained alignment module.

[0048] In one embodiment, the training module is used to:

[0049] For each training sample, inputting a sample video of the training sample into the visual feature encoder, and outputting visual features of the sample video;

[0050] Clustering the visual features of the sample video using a clustering algorithm to obtain K cluster center features, where K is a positive integer;

[0051] Input the K class center features into an alignment module, and output aligned visual features;

[0052] Inputting the aligned visual features and the second preset paradigm instruction into the recognition module, and outputting the predicted description information of the sample video;

[0053] According to the text description information of the sample video and the predicted description information of the sample video, the parameters of the alignment module are adjusted until a training stop condition is met to obtain the alignment module.

[0054] In a fourth aspect, an embodiment of the present application provides a video tag recognition device, including:

[0055] An acquisition module, used for acquiring a video to be identified and text description information of the video to be identified, wherein the video to be identified is input in the form of a video frame sequence;

[0056] A processing module, used to determine whether a label recognition condition is satisfied, and obtain visual features of the video to be recognized, wherein the visual features of the video to be recognized are obtained by clustering and updating K class center features stored in a storage queue according to the visual features of the video to be recognized from time t to time t1, and the K class center features are obtained by clustering the visual features of the video to be recognized before the time t, wherein K is a positive integer, and the time t1 is the duration of the video to be recognized when the video to be recognized is recognized, and t1 is less than or equal to the duration of the video to be recognized;

[0057] The recognition module is used to determine whether the label recognition conditions are met, and determine the predicted label of the video to be identified according to the visual features of the video to be identified, the text description information of the video to be identified, the first preset paradigm instruction and the trained video label recognition model, wherein the video label recognition model includes a visual feature encoder, the storage queue, the alignment module and the recognition model.

[0058] In one embodiment, the identification module is used to:

[0059] If a tag recognition instruction is received which is triggered after the video frame sequence input of the video to be recognized is completed, it is determined that the tag recognition condition is met; or,

[0060] Starting from the time t, it is determined that the periodic tag identification time has been reached, and then it is determined that the tag identification condition is met.

[0061] In one embodiment, the processing module is further configured to:

[0062] Inputting the video to be identified before time t into a visual feature encoder, and outputting a first visual feature;

[0063] The first visual features are clustered by a clustering algorithm to obtain K class center features, and the K class center features are written into the storage queue.

[0064] In one embodiment, the processing module is used to:

[0065] Input the video of the video to be identified from time t to time t1 into the visual feature encoder, and output the visual features of the video to be identified from time t to time t1;

[0066] According to the visual features of the video to be identified from time t to time t1, the K class center features are clustered and updated, and the K class center features obtained after the update are determined as the visual features of the video to be identified.

[0067] In one embodiment, the processing module is used to:

[0068] According to the video features of the video to be identified at time t, clustering and updating the K class center features, obtaining the updated K class center features and writing them into the storage queue;

[0069] The K class center features in the storage queue are clustered and updated according to the video features of the video to be identified at each moment between time t and time t1 and the visual features of the video at time t1, and the updated K class center features are obtained and written into the storage queue.

[0070] In one embodiment, the video features of the video to be identified at time t include image block features of N video frames, where N is the number of image blocks, and the processing module is specifically configured to:

[0071] For each image block feature of a video frame in the video features of the video to be identified at time t, determine a corresponding target class center feature, wherein the target class center feature is a class center feature having the smallest Euclidean distance with the image block feature of the video frame;

[0072] The target class center features are clustered and updated according to preset class center update parameters and the image block features of the video frame.

[0073] In one embodiment, the processing module is specifically used for:

[0074] The sum of the target class center feature and the first result is determined as the updated target class center feature, where the first result is the product of the difference between the image block feature of the video frame and the target class center feature and the class center update parameter.

[0075] In one embodiment, the identification module is used to:

[0076] Inputting the visual features of the video to be identified into the alignment module, and outputting the aligned visual features;

[0077] The aligned visual features, the text description information of the video to be identified, the result of the previous label identification of the video to be identified, and the first preset paradigm instruction are input into the recognition model, and the predicted label of the video to be identified is output.

[0078] In a fifth aspect, an embodiment of the present application provides a computer device, comprising: a processor and a memory, the memory being used to store a computer program, the processor being used to call and run the computer program stored in the memory to execute the method of the first aspect or the second aspect.

[0079] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer program, enable the computer to execute a method as in the first aspect or the second aspect.

[0080] In a seventh aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the method of the first aspect or the second aspect.

[0081] In summary, in an embodiment of the present application, a first training sample set is obtained, each training sample in the first training sample set includes a sample video, a label of the sample video at time t, and a text description information of the sample video. For each training sample, the visual features of the sample video of the training sample are determined, and the visual features of the sample video are obtained by clustering and updating the K class center features stored in the storage queue according to the visual features of the sample video from time t0 to time t, and the K class center features are obtained by clustering the visual features of the sample video before time t0. Then, according to the visual features of the sample video, the text description information of the sample video, the first predicted label of the sample video, and the first preset paradigm instruction, the video label recognition model is trained until the training stop condition is met. Thus, the trained video tag recognition model can first cluster the visual features of the video to be recognized before time t to obtain K class center features and write them into the storage queue during the process of video tag recognition. When obtaining the visual features of the video to be recognized, the visual features of the video to be recognized from time t to time t1 are used to cluster and update the K class center features stored in the storage queue to obtain the visual features of the video to be recognized. In this way, regardless of the length of the video to be recognized, the visual features of the video to be recognized always maintain a fixed length and always include K class center features, greatly compressing redundant information, thereby reducing information loss caused by the sparse frame extraction method and improving the accuracy and recall rate of video tag recognition. On the other hand, since the method of this embodiment does not limit the length of the video to be recognized when used, it can support the need to recognize the video to be recognized at any time point in the actual application scenario, and is also applicable to the video to be recognized whose length of the video stream is unpredictable before the end, and has high flexibility in use. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] Figure 1 A schematic diagram of a process for applying a video tag recognition method provided in an embodiment of the present application;

[0083] Figure 2 A schematic diagram of an implementation scenario of a video tag recognition method provided in an embodiment of the present application;

[0084] Figure 3 A schematic diagram of a system architecture involved in an embodiment of the present application;

[0085] Figure 4 A flowchart of a video tag recognition model training method provided in an embodiment of the present application;

[0086] Figure 5 A schematic diagram of a visual feature before and after clustering provided in an embodiment of the present application;

[0087] Figure 6 A flowchart of a video tag recognition model training method provided in an embodiment of the present application;

[0088] Figure 7 A schematic diagram of the structure of a video tag recognition model provided in an embodiment of the present application;

[0089] Figure 8 A flowchart of a video tag recognition method provided in an embodiment of the present application;

[0090] Fig. 9 A schematic diagram of the structure of a video tag recognition model training device provided in an embodiment of the present application;

[0091] Fig.10 A schematic diagram of the structure of a video tag recognition device provided in an embodiment of the present application;

[0092] Fig.11 It is a schematic block diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0093] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the embodiments of the present application.

[0094] It should be noted that the terms "first", "second", etc. in the specification and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or server comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0095] First, the relevant terms involved in the embodiments of the present application are introduced.

[0096] 1. Pre-training model: also known as cornerstone model or large model, it refers to a deep neural network (DNN) with large parameters. It is trained on massive unlabeled data, and the function approximation ability of large-parameter DNN is used to enable PTM to extract common features from the data. After fine tuning, parameter efficient fine tuning (PEFT), prompt-tuning and other technologies, it is suitable for downstream tasks. Therefore, the pre-training model can achieve ideal results in few-shot or zero-shot scenarios. PTM can be divided into language models (ELMO, BERT, GPT), visual models (swin-transformer, ViT, V-MOE), speech models (VALL-E), multimodal models (ViBERT, CLIP, Flamingo, Gato), etc. according to the data modality processed. Among them, the multimodal model refers to a model that establishes two or more data modality feature representations. The pre-training model is an important tool for outputting artificial intelligence generated content (AIGC), and can also be used as a general interface to connect multiple specific task models.

[0097] 2. Large Language Model (LLM): By training a large amount of text data to learn the statistical laws and semantic information of the language, it can predict the next word or sentence. LLM has a wide range of applications in the field of natural language processing (NLP), such as machine translation, speech recognition, text generation, etc. LLM can adopt different training strategies and model structures, such as pre-training and fine-tuning. Pre-training refers to large-scale training of the model under unsupervised or self-supervised conditions, so that it has a general understanding and expression ability of language data. Fine-tuning is to optimize the model for specific application scenarios or tasks through supervised learning based on the pre-trained model. LLM has achieved remarkable results in tasks such as language understanding, generation and translation, and its performance and application scope are constantly expanding with the continuous expansion of the model scale and the increase of training data.

[0098] 3. Multimodal large language model (MLLM): Based on LLM, media data of other modalities (such as images, videos, audio, etc.) are integrated, so that the model can process information of different modalities at the same time, better understand and express semantics, thereby improving the effect and accuracy of the application.

[0099] 4. Optical Character Recognition (OCR): a technology that detects and recognizes text content from images.

[0100] 5. Automatic Speech Recognition (ASR) is a technology that converts human speech into text.

[0101] 6. Token: In LLM, a token represents the smallest unit of meaning that the model can understand and generate, and is the basic unit used by the model when processing text. Depending on the specific tokenization scheme used, a token can represent a word, part of a word, or even just a character. Tokens are assigned numerical values ​​or identifiers, arranged in sequences or vectors, and are input or output from the model. They are the language building blocks of the model.

[0102] In the related technology, for medium and long videos, the video frame sequence obtained by sparse frame extraction cannot cover most of the content of the video, which will cause information loss and thus lead to low accuracy and recall rate of video tag recognition; and for video media such as video streams (such as live broadcast scenes), the length of the video stream cannot be predicted before the end, making it difficult to use this method.

[0103] To solve this problem, the embodiment of the present application obtains the video to be identified and the text description information of the video to be identified when performing video label identification on the video to be identified. The visual features of the video to be identified before time t are first clustered to obtain K class center features and write them into a storage queue. When it is determined that the label identification conditions are met, the K class center features stored in the storage queue are clustered and updated according to the visual features of the video to be identified from time t to time t1 to obtain the visual features of the video to be identified. Time t1 is the length of the video to be identified when identifying the video to be identified. Then, according to the visual features of the video to be identified, the text description information of the video to be identified, the first preset paradigm instruction and the trained video label recognition model, the predicted label of the video to be identified is determined. Since the visual features of the video to be identified before time t are first clustered to obtain K class center features and written into the storage queue, when obtaining the visual features of the video to be identified, the visual features of the video to be identified from time t to time t1 are used to cluster and update the K class center features stored in the storage queue to obtain the visual features of the video to be identified. In this way, regardless of the length of the video to be identified, the visual features of the video to be identified always maintain a fixed length and always include K class center features, which greatly compresses redundant information, thereby reducing information loss caused by sparse frame extraction and improving the accuracy and recall rate of video tag recognition. On the other hand, since the method of this embodiment does not limit the length of the video to be identified, it can support the need to identify the video to be identified at any time point in actual application scenarios, and is also applicable to videos to be identified whose length of the video stream is unpredictable before the end, and has high flexibility in use.

[0104] The embodiments of the present application can be applied to various scenarios, including but not limited to video tag prediction scenarios, video classification scenarios, video recommendation, live broadcast scenarios, and so on. For example, for a large number of videos on a video website or video application, the method provided by the embodiments of the present application can be used to parse the video content offline or online to obtain the video tags corresponding to the videos. Based on the video tags corresponding to the videos, the large number of videos can be classified. For another example, based on the video tags corresponding to the videos, videos of interest can be recommended to users.

[0105] For example, Figure 1 A schematic diagram of a flow chart of the application of the video tag recognition method provided in the embodiment of the present application, such as Figure 1 As shown, the video enters the content processing link from the content production link, obtains the corresponding label through human-machine collaboration (machine labeling and manual labeling), and then enters the downstream content distribution link. The video label recognition method provided in the embodiment of the present application belongs to the machine labeling link, and obtains the label of the video.

[0106] It should be noted that the application scenarios described above are only used to illustrate the embodiments of the present application and are not intended to be limiting. In specific implementation, the technical solutions provided in the embodiments of the present application can be flexibly applied according to actual needs.

[0107] For example, Figure 2 A schematic diagram of an implementation scenario of a video tag recognition method provided in an embodiment of the present application, such as Figure 2 As shown, the implementation scenario of the embodiment of the present application involves a server 1 and a terminal device 2, and the terminal device 2 can communicate data with the server 1 through a communication network. The communication network can be a wireless or wired network such as an intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, and a call network.

[0108] Among them, in some possible implementations, the terminal device 2 refers to a type of device that has rich human-computer interaction methods, has the ability to access the Internet, is usually equipped with various operating systems, and has strong processing capabilities. The terminal device can be a terminal device such as a smart phone, a tablet computer, a portable laptop, a desktop computer, or a phone watch, etc., but is not limited thereto. Optionally, in the embodiment of the present application, various applications are installed in the terminal device 2, such as video applications, instant messaging applications, etc.

[0109] Among them, in some possible implementations, the terminal device 2 includes but is not limited to mobile phones, computers, intelligent voice interaction devices, smart home appliances, car terminals, etc.

[0110] Figure 2 The server 1 in the example can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. This embodiment of the application does not limit this. In this embodiment of the application, the server 1 can be a background server of an application installed in the terminal device 2.

[0111] In some possible implementations, Figure 2 One terminal device and one server are shown as an example, but other numbers of terminal devices and servers may actually be included, and the embodiments of the present application do not limit this.

[0112] In some embodiments, when video tag recognition is required, server 1 can use the method provided in the embodiment of the present application to first train a video tag recognition model, which can be specifically: obtaining a first training sample set, each training sample in the first training sample set includes a sample video, a label of the sample video at time t, and text description information of the sample video, and time t is less than or equal to the duration of the sample video. For each training sample, determining the visual features of the sample video of the training sample, the visual features of the sample video are obtained by clustering and updating the K class center features stored in the storage queue according to the visual features of the sample video from time t0 to time t, and the K class center features are The feature is obtained by clustering the visual features of the sample video before time t0, K is a positive integer, time t0 is a preset value, t0 is less than t, and the video label recognition model is trained according to the visual features of the sample video, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction until the stop training condition is met to obtain a trained video label recognition model. The video label recognition model includes a visual feature encoder, a storage queue, a pre-trained alignment module and a recognition model. The first predicted label of the sample video is the result of the last label recognition performed by the video label recognition model on the sample video at time t.

[0113] After obtaining the trained video tag recognition model, video tag recognition may be performed, specifically, the video tag recognition method described below in this application may be executed.

[0114] It is understandable that in the specific implementation of the present application, related data such as user information (such as sample videos and videos to be identified) are involved. When the method of the embodiment of the present application is applied to a specific product or technology, it is necessary to obtain user permission or consent, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0115] Figure 3 A schematic diagram of a system architecture involved in an embodiment of the present application.

[0116] like Figure 3 As shown, the system architecture may include a user device 101 , a data collection device 102 , a training device 103 , an execution device 104 , a database 105 and a content library 106 .

[0117] The data collection device 102 is used to read the training data from the content library 106 and store the read training data in the database 105. The training data involved in the embodiment of the present application includes a first training sample set and a second training sample set.

[0118] The training device 103 trains the video tag recognition model based on the training data maintained in the database 105. Optionally, the video tag recognition model can be further connected to other downstream models. The model obtained by the training device 103 can be applied to different systems or devices.

[0119] Optional, reference Figure 2 The execution device 104 is configured with an I / O interface 107 to exchange data with an external device, such as the user device 101. The computing module 109 in the execution device 104 uses the trained video tag recognition model to perform video tag recognition on the acquired video data to obtain the tag of the video data. Optionally, the execution device 104 can send the tag of the corresponding video data to the user device 101 through the I / O interface.

[0120] The user device 101 may include a mobile phone, a computer, an intelligent voice interaction device, an intelligent home appliance, a vehicle terminal, an aircraft or other terminal devices, which is not limited in the present embodiment. The user device 101 has a video APP installed therein.

[0121] The execution device 104 may be a server. Exemplarily, the server may be a computing device such as a rack server, a blade server, a tower server, or a cabinet server. The server may be an independent server or a server cluster composed of multiple servers, which is not limited in the embodiments of the present application.

[0122] In this embodiment, the execution device 104 is connected to the user device 101 through a network.

[0123] It should be noted that Figure 3 It is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. In some embodiments, the data acquisition device 102 and the user device 101, the training device 103 and the execution device 104 can be the same device. The database 105 can be distributed on one server or on multiple servers, and the content library 106 can be distributed on one server or on multiple servers.

[0124] The technical solution of the embodiment of the present application will be described in detail below:

[0125] Figure 4 A flowchart of a video tag recognition model training method provided in an embodiment of the present application. The execution subject of the embodiment of the present application is a device with a model training function, and the model training device can be, for example, a server, such as Figure 4 As shown, the method may include:

[0126] S101. Obtain a first training sample set, where each training sample in the first training sample set includes a sample video, a label of the sample video at time t, and text description information of the sample video, where time t is less than or equal to the duration of the sample video.

[0127] Specifically, the video tag recognition model training may be an iterative training. In any iterative training process, the operations of S101 to S103 may be performed until a training stop condition is met to obtain a trained video tag recognition model.

[0128] Obtain the first training sample set. In any iterative training process, multiple (for example, I, where I is a positive integer) training samples can be randomly selected from the original training sample set to form the first training sample set. Each training sample includes a sample video, a label of the sample video at time t, and text description information of the sample video. The label of the sample video at time t is the real label of the sample video at time t, which can be obtained by manual labeling, that is, it can be a human review label. The sample video is a video stream. The label of the video may be different at different times (time points). For example, the length of the video is 1 minute, and the corresponding labels may be different at the three times of 15 seconds, 30 seconds or 1 minute. The same video stream can be reviewed multiple times at different times to obtain different review results. Among them, the text description information of the sample video is used to describe the video content of the sample video, for example, it can be the title of the sample video and / or the key text extracted from the OCR / ASR recognition results of the sample video.

[0129] Optionally, in one embodiment, an original training sample set is constructed in advance in this embodiment. The training samples in the original training sample set can come from multiple video streams. One video stream can obtain multiple training samples. For example, the original training sample set includes 6 training samples. The 6 training samples come from two video streams, of which 3 training samples come from video stream 1. Training sample 1 includes a video stream 1, a label of the video stream 1 at time t1, and text description information of the video stream 1. Training sample 2 includes a video stream 1, a label of the video stream 1 at time t2, and text description information of the video stream 1. 1, training sample 3 includes a video stream 1, the label of the video stream 1 at time t3 and the text description information of the video stream 1; there are 3 training samples from video stream 2, training sample 4 includes a video stream 2, the label of the video stream 2 at time t4 and the text description information of the video stream 2, training sample 5 includes a video stream 2, the label of the video stream 2 at time t5 and the text description information of the video stream 2, and training sample 6 includes a video stream 2, the label of the video stream 2 at time t6 and the text description information of the video stream 2. The label of each training sample can be manually reviewed and annotated.

[0130] In one embodiment, the original training sample set may be, for example, < video stream i (t) , Label i (t) , text description information i (t) >, where i represents the index of the i-th video stream in the original training sample set, and the superscript t represents the time t. The same video stream can be reviewed multiple times at different times and obtain different label results.

[0131] S102. For each training sample, determine the visual features of the sample video of the training sample. The visual features of the sample video are obtained by clustering and updating the K class center features stored in the storage queue according to the visual features of the sample video from time t0 to time t. The K class center features are obtained by clustering the visual features of the sample video before time t0. K is a positive integer, time t0 is a preset value, and t0 is less than t.

[0132] Specifically, after obtaining the training sample set, for each training sample, the visual features of the sample video of the training sample are determined, and the visual features of the sample video of the training sample are obtained by clustering and updating the K class center features stored in the storage queue according to the visual features of the sample video from time t0 to time t, and the K class center features are obtained by clustering the visual features of the sample video before time t0. Among them, time t0 is a preset value, and optionally, time t0 can be preset according to the scene category to which the sample video belongs. The process of clustering the visual features of the sample video before time t0 to obtain K class center features and storing them can be performed offline.

[0133] Exemplarily, the process of generating visual features of a sample video of each training sample is shown below.

[0134] Optionally, as an implementable manner, the visual features of the sample video of the training sample determined in S102 may specifically be:

[0135] S1021. Input the sample video before time t0 into a visual feature encoder, and output a first visual feature.

[0136] Specifically, the function of the visual feature encoder in the embodiment of the present application is to convert an input video frame image into a sequence of visual features (column vectors). Optionally, the visual feature encoder in the present embodiment may be ViT or CNN, or an encoder of other structures, which is not limited in the present embodiment. For ViT, the size of the visual feature may be N*d, where N is the number of image blocks (patches) and d is the feature dimension; and for a structure such as CNN, the size of the visual feature may be H*W*d, where H*W is equivalent to the number of patches N of ViT. In one embodiment, a ViT trained by the CLIP method may be used as a visual feature encoder. Specifically, the video of the sample video before time t0 is input into the visual feature encoder, and the visual features of the video of the sample video before time t0 are output, and the visual features of the video of the sample video before time t0 are called the first visual features. For the convenience of explanation, for example, for the video frame received at time T= t0, the extracted visual features are recorded as: F (t) ={f (t) j}, j = 1, 2, ..., N, ignoring the time superscript without introducing ambiguity, can be abbreviated as F, where N is the number of image patches. In actual business scenarios, the model is usually not required to predict the results when a small number of video frames are received. In this embodiment, assuming that the model will not be allowed to reason before time T = t0, the features corresponding to all video frames received before time t0 can be used to initialize the storage queue. In order to prevent the visual token from being too long, the maximum length of the storage queue can be set to K.

[0137] S1022: Cluster the first visual feature using a clustering algorithm to obtain K cluster center features, and write the K cluster center features into a storage queue.

[0138] Specifically, the clustering algorithm may be a preset clustering algorithm, such as a K-means algorithm, or other clustering algorithms, which are not limited in this embodiment. The first visual feature is clustered by a clustering algorithm to obtain K cluster center features, that is, the visual features of the sample video before time t0 are represented by the K cluster center features. For example, the K cluster center features are recorded as M= {m1,m2, …, m K},m K is the K-th class center feature. Figure 5 A schematic diagram of a visual feature before and after clustering provided in an embodiment of the present application, such as Figure 5 As shown, the first visual feature is clustered by the K-means algorithm to obtain K class center features, such as Figure 5 After clustering, three cluster center features are obtained. S1021-S1022 can be called the process of initializing the storage queue.

[0139] S1023. Input the sample video from time t0 to time t into a visual feature encoder, and output the visual features of the sample video from time t0 to time t.

[0140] Specifically, t0 is less than t, t is the moment when label prediction is performed on the sample video, the video of the sample video from moment t0 to moment t includes the video of the sample video at moment t0, the video between moment t0 and moment t, and the video of the sample video at moment t, the video of the sample video from moment t0 to moment t is input into the visual feature encoder for feature extraction, and the visual features of the video of the sample video from moment t0 to moment t are output.

[0141] S1024. Cluster and update the K class center features according to the visual features of the sample video from time t0 to time t, and determine the K class center features obtained after the update as the visual features of the sample video of the training sample.

[0142] Specifically, based on the visual features of the sample video from time t0 to time t, the K class center features are clustered and updated, and the number of class center features obtained after the update is still K. The K class center features obtained after the update are the visual features of the sample video of the training sample.

[0143] Optionally, in an implementable manner, S1024 may specifically be:

[0144] S10241. Cluster and update the K class center features according to the video features of the sample video at time t0, obtain the updated K class center features and write them into the storage queue.

[0145] Specifically, compared with before time t0, the sample video has a new video frame at time t0, and according to the video features of the sample video at time t0, the K class center features are clustered and updated.

[0146] S10242. Cluster and update the K class center features in the storage queue according to the video features of the sample video at each moment between time t0 and time t and the visual features of the video at time t, obtain the updated K class center features and write them into the storage queue.

[0147] Specifically, after the K class center features are clustered and updated according to the video features of the sample video at time t0, the same updating method is used to cluster and update the K class center features in the storage queue according to the video features of the video at each moment between time t0 and time t and the visual features of the video at time t, and the updated K class center features are obtained and written into the storage queue.

[0148] In one embodiment, the video features of the sample video at time t0 include image block features of N video frames, where N is the number of image blocks. In S10241, clustering and updating of K cluster center features are performed according to the video features of the sample video at time t0, which may be specifically:

[0149] S1. For the image block features of each video frame in the video features of the sample video at time t0, determine the corresponding target class center feature, where the target class center feature is the class center feature with the smallest Euclidean distance with the image block features of the video frame.

[0150] S2. Clustering and updating the target class center features according to preset class center update parameters and image block features of the video frame.

[0151] Optionally, as an implementable manner, in S2, the target class center features are clustered and updated according to the preset class center update parameters and the image block features of the video frame, which may be specifically:

[0152] The sum of the target class center feature and the first result is determined as the updated target class center feature, and the first result is the product of the difference between the image block feature of the video frame and the target class center feature and the class center update parameter.

[0153] Specifically, for example, for the video frame of the sample video at time t0, the extracted visual features are recorded as: F (t) = {f (t) j}, j = 1, 2, ..., N, ignoring the time superscript without introducing ambiguity, it can be abbreviated as F, where N is the number of image patches. For each video frame in F, the image patch feature f (t) j , j = 1, 2, ..., N, determine f (t) j The target class center feature is K class center features M= {m1,m2, …, m K} and f (t) j The class center feature with the smallest Euclidean distance between them. For example, the target class center feature is m i , the class center update parameter is η, the sum of the target class center feature and the first result is determined as the updated target class center feature, and the first result is the product of the difference between the image block feature of the video frame and the target class center feature and the class center update parameter. That is: m i = m i + η(f - m i ).

[0154] A k-means clustering method is exemplarily given below to cluster and update K cluster center features.

[0155] Parameters: τ: number of online k-means iterations; η: class center update parameter

[0156] Input: The current data M in the storage queue t-1 , the received new visual features {f (t) j}

[0157] Output: Updated storage queue data M t

[0158] Explanation of symbols:

[0159] L2_distance() is the L2 norm

[0160] for x = 1: τ do # Number of online clustering iterations

[0161] index = []

[0162] for f∈F

[0163] i = argmin j L2_distance(m j , f) # Find the class center feature with the smallest Euclidean distance for each f

[0164] index.append((f, i))# Save the result

[0165] end for

[0166] for f,i in index:

[0167] m i = m i + η(f - m i )# Update the corresponding class center feature

[0168] end for

[0169] end for

[0170] Updated M t There are still K features and are not affected by the length of the video stream.

[0171] S103. Train the video label recognition model according to the visual features of the sample video, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction until the training stop condition is met to obtain a trained video label recognition model. The video label recognition model includes a visual feature encoder, a storage queue, a pre-trained alignment module and a recognition model. The first predicted label of the sample video is the result of the last label recognition performed by the video label recognition model on the sample video at time t.

[0172] Specifically, after obtaining the visual features of the sample video, the video label recognition model can be trained according to the visual features of the sample video, the text description information of the sample video, the first predicted label of the sample video, and the first preset paradigm instruction. Among them, the first predicted label of the sample video is the result of the last label recognition of the sample video by the video label recognition model at time t. The result of the last label recognition here can be the result of label recognition of the sample video at time tn, and n can be a preset value. For a sample video, for example, a sample video with a duration of 5 minutes, label recognition can be performed once every 1 minute, 2 minutes, or 5 minutes, and the time intervals of label recognition can be the same or different.

[0173] The specific video tag recognition model training process is shown below.

[0174] Optionally, in an implementable manner, in S103, the video label recognition model is trained according to the visual features of the sample video, the text description information of the sample video, the first predicted label of the sample video, and the first preset paradigm instruction, which may be:

[0175] S1031. Input the visual features of the sample video into an alignment module, and output the aligned visual features.

[0176] S1032, input the aligned visual features, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction into the recognition model, and output the predicted label of the sample video at time t.

[0177] Among them, the alignment module is a trained alignment module, the alignment module contains a learnable parameter matrix, the alignment module is mainly used to map the visual features to the input dimension of the recognition model, and at the same time convert the visual features into text unit features that are adapted to the recognition model. The alignment module can convert inputs from different modalities into tokens that can be understood by the recognition model. Specifically, the visual features can be aligned and dimensionally converted to obtain aligned visual features. Optionally, the alignment module can use a fully connected layer prediction (projection) form, can also use other forms of adapter networks, can also be a trained linear projection module or a multilayer perceptron (Multilayer Perceptron, MLP). This embodiment does not limit this.

[0178] The text description information of the sample video may include the title of the sample video, the description text of the sample video, and key text extracted by optical character recognition (OCR) / automatic speech recognition (ASR) of the sample video, etc., wherein OCR detects and recognizes text content from video frames, and ASR converts speech into text. It should be noted that for the sample video that is first subjected to label recognition, the first predicted label of the sample video is empty.

[0179] The first preset paradigm instruction may be, for example, any one of the following three examples:

[0180] "What tags can be used to summarize the above video?";

[0181] "Please output some key content tags based on the content of the above video.";

[0182] "Based on the above video content, generate the main content tags."

[0183] Optionally, the aligned visual features, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction may be used to form a prompt word, for example, the prompt word may be formed in the form of [aligned visual features] [text description information of the sample video] [first predicted label of the sample video] [first preset paradigm instruction], the prompt word may be input into the recognition model, and the predicted label of the sample video may be output. Optionally, the recognition model may be a multimodal model. Specifically, for example, it may be a large language model (LLM) or a multimodal large language model (MLLM), or it may be chinese-llama, BLOOM, etc. For the introduction of LLM and MLLM, please refer to the introduction of the aforementioned terms, which will not be repeated here.

[0184] S1033. According to the label of the sample video at time t and the predicted label of the sample video at time t, the parameters of the alignment module and the parameters of the recognition model are adjusted until the training stop condition is met.

[0185] Specifically, the label of the sample video at time t is the true label. According to the true label of the sample video at time t and the predicted label of the sample video at time t, the parameters of the alignment module and the parameters of the recognition model can be adjusted until the stopping training condition is met, and the trained video label recognition model can be obtained.

[0186] Optionally, as an implementable manner, in S1033, according to the label of the sample video at time t and the predicted label of the sample video at time t, the parameters of the alignment module and the parameters of the recognition model are adjusted until the training stop condition is met, which may be:

[0187] S10331. Construct a loss function based on the label of the sample video at time t and the predicted label of the sample video at time t.

[0188] S10332. According to the loss function, back-propagation is used to adjust the parameters of the alignment module and the parameters of the recognition model until the training stop condition is met.

[0189] The training stop condition may be preset, such as reaching a preset number of training times, or other preset conditions, which are not limited in this embodiment.

[0190] Through the training of S101-S103 above, a trained video tag recognition model is obtained, and the trained video tag recognition model includes a visual feature encoder, a storage queue, a pre-trained alignment module and a recognition model.

[0191] In another practicable manner, the video tag recognition model further includes a low-rank adaptation of large language models (LoRA) module of the model. In S103, the video tag recognition model is trained according to the visual features of the sample video, the text description information of the sample video, the first predicted tag of the sample video, and the first preset paradigm instruction. Specifically, it can be:

[0192] S1031', input the visual features of the sample video into the alignment module, and output the aligned visual features.

[0193] S1032', input the aligned visual features, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction into the recognition model, and output the predicted label of the sample video at time t.

[0194] S1033', fix the parameters of the recognition model, and adjust the parameters of the alignment module and the parameters of the LoRA module according to the label of the sample video at time t and the predicted label of the sample video at time t, until the training stop condition is met.

[0195] Optionally, as an implementable manner, in S1033', according to the label of the sample video at time t and the predicted label of the sample video at time t, the parameters of the alignment module and the parameters of the LoRA module are adjusted until the training stop condition is met, which may be:

[0196] S11. Construct a loss function according to the label of the sample video at time t and the predicted label of the sample video at time t.

[0197] S12. According to the loss function, back propagation is used to adjust the parameters of the alignment module and the parameters of the LoRA module until the training stop condition is met.

[0198] By adding the LoRA module, only the parameters of the LoRA module need to be trained, which can reduce the complexity of the model and the computing resources required for fine-tuning.

[0199] In this embodiment, the alignment module is pre-trained. The method of this implementation may further include a training process of the alignment module before S101, which will be described in detail below in conjunction with S104-S105.

[0200] In an practicable manner, further, before S101, the method of this embodiment may further include:

[0201] S104: Obtain a second training sample set, where each training sample in the second training sample set includes a sample video and text description information of the sample video.

[0202] Specifically, the second training sample set here is a different training sample set from the first training sample set in S101. The second training sample set here includes multiple training samples, each of which includes a sample video and text description information of the sample video. The text description information of the sample video is used to describe the video content of the sample video, for example, the title of the sample video and / or key text extracted from the OCR / ASR recognition results of the sample video. The main goal of training at this stage is to align visual features to the recognition model.

[0203] S105 , fixing the parameters of the visual feature encoder and the parameters of the recognition model, and training the alignment module according to the second training sample set to obtain a pre-trained alignment module.

[0204] Specifically, the video tag recognition model to be trained in this embodiment includes a visual feature encoder, a storage queue, an alignment module and a recognition model. In this embodiment, the parameters of the visual feature encoder and the recognition model are fixed first, and the storage queue has no parameters. The alignment module is first trained according to the training sample set to obtain a pre-trained alignment module. The main goal of the pre-trained alignment module is to align the visual features to the recognition model.

[0205] Optionally, in S105, the alignment module is trained according to the second training sample set to obtain a pre-trained alignment module, which may specifically be:

[0206] S1051. For each training sample, input the sample video of the training sample into a visual feature encoder, and output the visual features of the sample video.

[0207] S1052: Cluster the visual features of the sample video using a clustering algorithm to obtain K cluster center features, where K is a positive integer.

[0208] Specifically, the clustering process is similar to the clustering process in the above training process, and will not be repeated here.

[0209] S1053, input the K class center features into the alignment module, and output the aligned visual features.

[0210] Specifically, the alignment module includes a learnable parameter matrix for mapping visual features to the input dimension of the recognition model and converting them into text unit features adapted to the recognition model.

[0211] S1054, input the aligned visual features and the second preset paradigm instruction into the recognition module, and output the predicted description information of the sample video.

[0212] Specifically, the second preset paradigm instruction may be, for example, any one of the following examples:

[0213] "Briefly describe the following video.";

[0214] "Provides a short description of a given video clip.";

[0215] "Concisely explain the video clips provided.";

[0216] "An overview of the visual content of the following video.";

[0217] "Give a brief and clear explanation of the following video clip.";

[0218] "Briefly describe the meaning of the video provided.";

[0219] "Briefly describe the key features of the fragment.";

[0220] "Briefly describe the content of the video being shown.";

[0221] "Provide a clear and concise summary of the following video.";

[0222] "Write an informative summary of the following video clip.";

[0223] "Present the provided videos in a concise narrative.".

[0224] Optionally, the aligned visual features and the second preset paradigm instructions may be combined into a prompt word, for example, a prompt word may be combined into the form of [aligned visual features] [second preset paradigm instructions], the prompt word may be input into the recognition model, and the predicted description information of the sample video may be output.

[0225] S1055. According to the text description information of the sample video and the predicted description information of the sample video, the parameters of the alignment module are adjusted until the training stop condition is met to obtain the alignment module.

[0226] Optionally, the parameters of the alignment module are adjusted according to the text description information of the sample video and the predicted description information of the sample video. A loss function may be constructed according to the text description information of the sample video and the predicted description information of the sample video, and the parameters of the alignment module are adjusted by back propagation according to the loss function until the stop training condition is met to obtain a pre-trained alignment module. The stop training condition here may be a preset stop training condition, such as reaching a preset number of iterative training times or other conditions, which is not limited in this embodiment.

[0227] The video label recognition model training method provided in the present embodiment obtains a first training sample set, wherein each training sample in the first training sample set includes a sample video, a label of the sample video at time t, and text description information of the sample video. For each training sample, the visual features of the sample video of the training sample are determined, wherein the visual features of the sample video are obtained by clustering and updating K class center features stored in a storage queue based on the visual features of the sample video from time t0 to time t, wherein the K class center features are obtained by clustering the visual features of the sample video before time t0. Then, according to the visual features of the sample video, the text description information of the sample video, the first predicted label of the sample video, and the first preset paradigm instruction, the video label recognition model is trained until a training stop condition is met. Thus, the trained video tag recognition model can first cluster the visual features of the video to be recognized before time t to obtain K class center features and write them into the storage queue during the process of video tag recognition. When obtaining the visual features of the video to be recognized, the visual features of the video to be recognized from time t to time t1 are used to cluster and update the K class center features stored in the storage queue to obtain the visual features of the video to be recognized. In this way, regardless of the length of the video to be recognized, the visual features of the video to be recognized always maintain a fixed length and always include K class center features, greatly compressing redundant information, thereby reducing information loss caused by the sparse frame extraction method and improving the accuracy and recall rate of video tag recognition. On the other hand, since the method of this embodiment does not limit the length of the video to be recognized when used, it can support the need to recognize the video to be recognized at any time point in the actual application scenario, and is also applicable to the video to be recognized whose length of the video stream is unpredictable before the end, and has high flexibility in use.

[0228] Combine the following Figure 6 and Figure 7 , a specific implementation example is used to describe in detail the process of video tag recognition model training.

[0229] Figure 6 A flowchart of a video tag recognition model training method provided in an embodiment of the present application. The execution subject of the embodiment of the present application is a device with a model training function, and the model training device can be, for example, a server, such as Figure 6 As shown, the method may include:

[0230] S201: Obtain a second training sample set, where each training sample in the second training sample set includes a sample video and text description information of the sample video.

[0231] Specifically, the text description information of the sample video is used to describe the video content of the sample video, for example, it may be the title of the sample video and / or key text extracted from the OCR / ASR recognition result of the sample video.

[0232] S202 , fixing the parameters of the visual feature encoder and the parameters of the recognition model, and training the alignment module according to the second training sample set to obtain a pre-trained alignment module.

[0233] Specifically, Figure 7 A schematic diagram of the structure of a video tag recognition model provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the video tag recognition model to be trained in this embodiment includes a visual feature encoder 110, a storage queue 120, an alignment module 130 and a recognition model 140. The training of the video tag recognition model in this embodiment is divided into two stages. S201-S202 is stage one, in which the parameters of the visual feature encoder and the parameters of the recognition model are fixed first, and the storage queue has no parameters. The alignment module is trained according to the second training sample set to obtain a pre-trained alignment module. The main goal of the pre-trained alignment module is to align the visual features to the recognition model. S203-S207 is stage two, in which the recognition model and the alignment module are trained to finally obtain a trained video tag recognition model.

[0234] Among them, the training samples can use the <video, video title> pairs in the business scenario as sample videos and text description information of the sample videos. The video description text in this embodiment does not need to be manually labeled, and the video description text can be directly used to train the model with the goal of generating a brief description of the video content.

[0235] Optionally, in S202, the alignment module is trained according to the second training sample set to obtain a pre-trained alignment module, which may specifically be:

[0236] S2021. For each training sample, input the sample video of the training sample into a visual feature encoder, and output the visual features of the sample video.

[0237] S2022. Cluster the visual features of the sample video using a clustering algorithm to obtain K cluster center features, where K is a positive integer.

[0238] For details, please refer to Figure 4 The specific description of S1022 in the illustrated embodiment will not be repeated here.

[0239] S2023. Input the K class center features into the alignment module and output the aligned visual features.

[0240] Specifically, the alignment module includes a learnable parameter matrix for mapping visual features to the input dimension of the recognition model and converting them into text unit features adapted to the recognition model.

[0241] S2024. Input the aligned visual features and the second preset paradigm instructions into the recognition module, and output the predicted description information of the sample video.

[0242] Specifically, the second preset paradigm instruction may be, for example, any one of the following examples:

[0243] "Briefly describe the following video.";

[0244] "Provides a short description of a given video clip.";

[0245] "Concisely explain the video clips provided.";

[0246] "An overview of the visual content of the following video.";

[0247] "Give a brief and clear explanation of the following video clip.";

[0248] "Briefly describe the meaning of the video provided.";

[0249] "Briefly describe the key features of the fragment.";

[0250] "Briefly describe the content of the video being shown.";

[0251] "Provide a clear and concise summary of the following video.";

[0252] "Write an informative summary of the following video clip.";

[0253] "Present the provided videos in a concise narrative.".

[0254] Optionally, the aligned visual features and the second preset paradigm instructions may be combined into a prompt word, for example, a prompt word may be combined into the form of [aligned visual features] [second preset paradigm instructions], the prompt word may be input into the recognition model, and the predicted description information of the sample video may be output.

[0255] S2025. According to the text description information of the sample video and the predicted description information of the sample video, the parameters of the alignment module are adjusted until the training stop condition is met to obtain the alignment module.

[0256] Optionally, the parameters of the alignment module are adjusted according to the text description information of the sample video and the predicted description information of the sample video. A loss function may be constructed according to the text description information of the sample video and the predicted description information of the sample video, and the parameters of the alignment module are adjusted by back propagation according to the loss function until the stop training condition is met to obtain a pre-trained alignment module. The stop training condition here may be a preset stop training condition, such as reaching a preset number of iterative training times or other conditions, which is not limited in this embodiment.

[0257] After obtaining the pre-trained alignment module, the second stage of training is carried out.

[0258] S203: Obtain a first training sample set, where each training sample in the first training sample set includes a sample video, a label of the sample video at time t, and text description information of the sample video, where time t is less than or equal to the duration of the sample video.

[0259] Specifically, in an implementable manner, training samples can be obtained based on videos in actual business scenarios and manually reviewed labels of the videos. The label of the sample video at time t is the real label of the sample video at time t, which can be obtained through manual labeling, that is, it can be a human review label. The sample video is a video stream. The labels of this video may be different at different moments (time points). For example, if the length of this video is 1 minute, the corresponding labels may be different at 15 seconds, 30 seconds or 1 minute. The same video stream can be reviewed multiple times at different times to obtain different review results. Among them, the text description information of the sample video is used to describe the video content of the sample video, for example, it can be the title of the sample video and / or key text extracted from the OCR / ASR recognition results of the sample video.

[0260] In one embodiment, the first training sample set may be, for example, < video stream i (t) , Label i (t) , text description information i (t) >, where i represents the index of the i-th video stream in the original training sample set, and the superscript t represents the time t. The same video stream can be reviewed multiple times at different times and obtain different label results.

[0261] S204. For each training sample, determine the visual features of the sample video of the training sample. The visual features of the sample video are obtained by clustering and updating the K class center features stored in the storage queue according to the visual features of the sample video from time t0 to time t. The K class center features are obtained by clustering the visual features of the sample video before time t0. K is a positive integer, time t0 is a preset value, and t0 is less than t.

[0262] Specifically, the process of determining the visual features of the sample video of the training sample can be seen in Figure 4 The description of S1021-S1024 in the illustrated embodiment will not be repeated here. Figure 7 The central feature M of the class at time t-1 t-1 There are four square blocks. After receiving the visual authentication of the new video frame at time t, the four new class center features M are obtained after online clustering. tis the feature M located at the center of the four classes shown in Figure 7 t-1 The diamond blocks on top.

[0263] S205: Input the visual features of the sample video into an alignment module, and output the aligned visual features.

[0264] S206, input the aligned visual features, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction into the recognition model, and output the predicted label of the sample video at time t.

[0265] The first preset paradigm instruction may be, for example, any one of the following three examples:

[0266] "What tags can be used to summarize the above video?";

[0267] "Please output some key content tags based on the content of the above video.";

[0268] "Based on the above video content, generate the main content tags."

[0269] Optionally, the aligned visual features, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction may be used to form a prompt word, for example, the prompt word may be formed in the form of [aligned visual features] [text description information of the sample video] [first predicted label of the sample video] [first preset paradigm instruction], and the prompt word may be input into the recognition model to output the predicted label of the sample video. Figure 7 As shown in , the prompt word can also be input into the text processing module (tokenizer) for processing and then input into the recognition model. The text processing module is used to convert text data into a token sequence that can be input into the model.

[0270] Optionally, the recognition model may be a multimodal model. Specifically, for example, it may be a large language model (LLM) or a multimodal large language model (MLLM), or may be Chinese-llama, BLOOM, etc. For an introduction to LLM and MLLM, please refer to the introduction to the aforementioned terms, which will not be repeated here.

[0271] S207. According to the label of the sample video at time t and the predicted label of the sample video at time t, the parameters of the alignment module and the parameters of the recognition model are adjusted until the training stop condition is met to obtain a trained video label recognition model.

[0272] Specifically, the label of the sample video at time t is the true label. According to the true label of the sample video at time t and the predicted label of the sample video at time t, the parameters of the alignment module and the parameters of the recognition model can be adjusted until the stopping training condition is met, and the trained video label recognition model can be obtained.

[0273] Optionally, as an implementable manner, in S207, according to the label of the sample video at time t and the predicted label of the sample video at time t, the parameters of the alignment module and the parameters of the recognition model are adjusted until the training stop condition is met, which may be:

[0274] S2071. Construct a loss function according to the label of the sample video at time t and the predicted label of the sample video at time t.

[0275] S2072. According to the loss function, back-propagation is used to adjust the parameters of the alignment module and the parameters of the recognition model until the training stop condition is met.

[0276] Optionally, in another practicable manner, the video tag recognition model further includes a low-rank adaptation of large language models (LoRA) module. In this case, S207 may specifically be:

[0277] S207', fixing the parameters of the recognition model, and adjusting the parameters of the alignment module and the parameters of the LoRA module according to the label of the sample video at time t and the predicted label of the sample video at time t, until the training stop condition is met.

[0278] Optionally, as an implementable manner, in S207', according to the label of the sample video at time t and the predicted label of the sample video at time t, the parameters of the alignment module and the parameters of the LoRA module are adjusted until the training stop condition is met, which may be:

[0279] S2071', construct a loss function according to the label of the sample video at time t and the predicted label of the sample video at time t.

[0280] S2072', according to the loss function, adjust the parameters of the alignment module and the parameters of the LoRA module by back propagation until the training stop condition is met.

[0281] By adding the LoRA module, only the parameters of the LoRA module need to be trained, which can reduce the complexity of the model and the computing resources required for fine-tuning.

[0282] Through the above training, a trained video label recognition model is obtained.

[0283] The video tag recognition model training method provided in this embodiment can make the trained video tag recognition model cluster the visual features of the video to be recognized before time t to obtain K class center features and write them into the storage queue during the process of video tag recognition. When obtaining the visual features of the video to be recognized, the visual features of the video to be recognized from time t to time t1 are used to cluster and update the K class center features stored in the storage queue to obtain the visual features of the video to be recognized. In this way, regardless of the length of the video to be recognized, the visual features of the video to be recognized always maintain a fixed length and always include K class center features, greatly compressing redundant information, thereby reducing information loss caused by the sparse frame extraction method and improving the accuracy and recall rate of video tag recognition. On the other hand, since the method of this embodiment does not limit the length of the video to be recognized when used, it can support the need to recognize the video to be recognized at any time point in the actual application scenario, and is also applicable to the video to be recognized whose length of the video stream is unpredictable before the end, and has high flexibility in use.

[0284] Figure 8 A flowchart of a video tag recognition method provided in an embodiment of the present application, the execution subject of the method may be a server or a terminal device, such as Figure 8 As shown, the method may include:

[0285] S301: Obtain a video to be identified and text description information of the video to be identified. The video to be identified is input in the form of a video frame sequence.

[0286] Specifically, in this embodiment, the video to be identified can be a long video, a medium video or a short video, or a live video stream, all of which are input in the form of a video frame sequence. After online clustering, a fixed-length feature sequence is obtained, which can trigger a tag recognition at any time. For example, for a medium-long video, a prediction can be made after the video frame sequence input is completed. For a live scene, such as a video stream, a tag recognition can be triggered periodically, for example, every N minutes.

[0287] S302, determine whether the label recognition condition is met, and obtain the visual features of the video to be recognized. The visual features of the video to be recognized are obtained by clustering and updating the K class center features stored in the storage queue according to the visual features of the video to be recognized from time t to time t1. The K class center features are obtained by clustering the visual features of the video to be recognized before time t. K is a positive integer. Time t1 is the duration of the video to be recognized when identifying the video to be recognized. t1 is less than or equal to the duration of the video to be recognized.

[0288] S303, determining a predicted label of the video to be identified based on the visual features of the video to be identified, the text description information of the video to be identified, the first preset paradigm instruction and the trained video label recognition model, wherein the video label recognition model includes a visual feature encoder, a storage queue, an alignment module and a recognition model.

[0289] The video tag recognition model is based on Figure 4 The method of the illustrated embodiment is trained.

[0290] Optionally, the determination in S302 that the tag identification condition is met may specifically be:

[0291] If a tag recognition instruction is received after the video frame sequence input of the video to be recognized is completed, it is determined that the tag recognition condition is met; or

[0292] Starting from time t, it is determined that the periodic tag identification time has been reached, and it is determined that the tag identification condition is met.

[0293] Specifically, in this embodiment, in one practicable manner, the process of initializing the storage queue to obtain K class center features can be performed in an offline manner. For the input video to be identified, the visual features of the video to be identified before time t can be clustered to obtain K class center features after receiving the video before time t, and the subsequent input video frames are received at the same time. In another practicable manner, it can also be performed in an online manner, which can be specifically the following process S304-S305.

[0294] Optionally, the method of this embodiment may further include:

[0295] S304: Input the video to be identified before time t into a visual feature encoder, and output a first visual feature.

[0296] S305 , clustering the first visual feature by using a clustering algorithm to obtain K cluster center features, and writing the K cluster center features into a storage queue.

[0297] Specifically, the clustering algorithm may be a preset clustering algorithm, such as a K-means algorithm, or other clustering algorithms, which are not limited in this embodiment. The first visual feature is clustered by a clustering algorithm to obtain K cluster center features, that is, the visual features of the video to be identified before time t are represented by the K cluster center features. For example, the K cluster center features are recorded as M= {m1,m2, …, m K},m K is the K-th class center feature.

[0298] Optionally, in an implementable manner, obtaining the visual features of the video to be identified in S302 may specifically be:

[0299] S3021, input the video to be identified from time t to time t1 into a visual feature encoder, and output the visual features of the video to be identified from time t to time t1.

[0300] S3022. Cluster and update the K class center features according to the visual features of the video to be identified from time t to time t1, and determine the K class center features obtained after the update as the visual features of the video to be identified.

[0301] In one embodiment, S3022 may specifically be:

[0302] S30221. Based on the video features of the video to be identified at time t, cluster and update the K class center features, obtain the updated K class center features and write them into the storage queue.

[0303] S30222. Cluster and update the K class center features in the storage queue according to the video features of the video to be identified at each moment between time t and time t1 and the visual features of the video at time t1, obtain the updated K class center features and write them into the storage queue.

[0304] In one embodiment, the video features of the video to be identified at time t include image block features of N video frames, where N is the number of image blocks. In S30221, clustering and updating of K class center features are performed according to the video features of the video to be identified at time t, which may be specifically:

[0305] S111. For each image block feature of a video frame in the video features of the video to be identified at time t, determine the corresponding target class center feature, where the target class center feature is the class center feature with the smallest Euclidean distance with the image block feature of the video frame.

[0306] S112: performing clustering update on the target class center features according to preset class center update parameters and image block features of the video frame.

[0307] Optionally, in an implementable manner, S112 may specifically be:

[0308] The sum of the target class center feature and the first result is determined as the updated target class center feature, and the first result is the product of the difference between the image block feature of the video frame and the target class center feature and the class center update parameter.

[0309] Specifically, for example, for the video frame to be identified at time t, the extracted visual features are recorded as: F (t) = {f (t) j}, j = 1, 2, ..., N, ignoring the time superscript without introducing ambiguity, it can be abbreviated as F, where N is the number of image patches. For each video frame in F, the image patch feature f (t) j , j = 1, 2, ..., N, determine f (t) j The target class center feature is K class center features M= {m1,m2, …, m K} and f (t) j The class center feature with the smallest Euclidean distance between them. For example, the target class center feature is m i , the class center update parameter is η, the sum of the target class center feature and the first result is determined as the updated target class center feature, and the first result is the product of the difference between the image block feature of the video frame and the target class center feature and the class center update parameter. That is: m i = m i + η(f - m i ).

[0310] Optionally, in an implementable manner, S303 may specifically be:

[0311] S3031. Input the visual features of the video to be identified into an alignment module, and output the aligned visual features.

[0312] S3032, input the aligned visual features, the text description information of the video to be identified, the result of the previous label identification of the video to be identified, and the first preset paradigm instruction into the recognition model, and output the predicted label of the video to be identified.

[0313] Specifically, the first preset paradigm instruction may be the same as the first preset paradigm instruction used in the video tag recognition model training. It should be noted that when tag recognition is performed for the first time on the video to be recognized, the result of the last tag recognition of the video to be recognized is empty.

[0314] In one embodiment, in combination Figure 7 The video tag recognition model shown can be a recognition model that inputs the text description information of the video to be recognized, the result of the previous tag recognition of the video to be recognized, and the first preset paradigm instruction into a text processing module (tokenizer) for processing, and then inputs it into the recognition model. At the same time, the aligned visual features are also input into the recognition model. The text processing module is used to convert text data into a token sequence that can be input by the model.

[0315] The video tag recognition method provided in this embodiment first clusters the visual features of the video to be recognized before time t to obtain K class center features and writes them into a storage queue. When obtaining the visual features of the video to be recognized, the visual features of the video to be recognized from time t to time t1 are used to cluster and update the K class center features stored in the storage queue to obtain the visual features of the video to be recognized. In this way, regardless of the length of the video to be recognized, the visual features of the video to be recognized always maintain a fixed length, always include K class center features, and greatly compress redundant information, thereby reducing the information loss caused by the sparse frame extraction method, and improving the accuracy and recall rate of video tag recognition. On the other hand, since the method of this embodiment does not limit the length of the video to be recognized when used, it can support the need to recognize the video to be recognized at any time point in the actual application scenario, and is also applicable to the video to be recognized whose length of the video stream is unpredictable before the end, and has high flexibility in use. It can also save manual review costs and improve the efficiency of video tag recognition.

[0316] Fig. 9 A schematic diagram of the structure of a video tag recognition model training device provided in an embodiment of the present application is shown in FIG. Fig. 9 As shown, the device may include: an acquisition module 11, a determination module 12 and a training module 13.

[0317] The acquisition module 11 is used to acquire a first training sample set, each training sample in the first training sample set includes a sample video, a label of the sample video at time t and text description information of the sample video, and time t is less than or equal to the duration of the sample video.

[0318] The determination module 12 is used to determine the visual features of the sample video of the training sample for each training sample. The visual features of the sample video are obtained by clustering and updating the K class center features stored in the storage queue based on the visual features of the sample video from time t0 to time t. The K class center features are obtained by clustering the visual features of the sample video before time t0. K is a positive integer, time t0 is a preset value, and t0 is less than t.

[0319] The training module 13 is used to train the video label recognition model according to the visual features of the sample video, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction until the training stop condition is met to obtain a trained video label recognition model. The video label recognition model includes a visual feature encoder, a storage queue, a pre-trained alignment module and a recognition model. The first predicted label of the sample video is the result of the last label recognition performed by the video label recognition model on the sample video at time t.

[0320] In one embodiment, the determination module 12 is used to:

[0321] Input the video of the sample video before time t0 into the visual feature encoder, and output the first visual feature;

[0322] Clustering the first visual feature by a clustering algorithm to obtain K class center features, and writing the K class center features into a storage queue;

[0323] Input the sample video from time t0 to time t into the visual feature encoder, and output the visual features of the sample video from time t0 to time t;

[0324] According to the visual features of the sample video from time t0 to time t, the K class center features are clustered and updated, and the K class center features obtained after the update are determined as the visual features of the sample video of the training sample.

[0325] In one embodiment, the determination module 12 is used to:

[0326] According to the video features of the sample video at time t0, the K class center features are clustered and updated to obtain the updated K class center features and write them into the storage queue;

[0327] According to the video features of the sample video at each moment between time t0 and time t and the visual features of the video at time t, the K class center features in the storage queue are clustered and updated to obtain the updated K class center features and write them into the storage queue.

[0328] In one embodiment, the video features of the sample video at time t0 include image block features of N video frames, where N is the number of image blocks, and the determination module is specifically configured to:

[0329] For each image block feature of a video frame in the video features of the sample video at time t0, determine the corresponding target class center feature, where the target class center feature is the class center feature with the smallest Euclidean distance with the image block feature of the video frame;

[0330] According to the preset class center update parameters and the image block features of the video frame, the target class center features are clustered and updated.

[0331] In one embodiment, the determination module 12 is specifically configured to:

[0332] The sum of the target class center feature and the first result is determined as the updated target class center feature, and the first result is the product of the difference between the image block feature of the video frame and the target class center feature and the class center update parameter.

[0333] In one embodiment, the training module 13 is used to: input the visual features of the sample video into the alignment module, and output the aligned visual features;

[0334] Input the aligned visual features, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction into the recognition model, and output the predicted label of the sample video at time t;

[0335] According to the label of the sample video at time t and the predicted label of the sample video at time t, the parameters of the alignment module and the parameters of the recognition model are adjusted until the training stop condition is met.

[0336] In one embodiment, the training module 13 is used to:

[0337] Construct a loss function based on the label of the sample video at time t and the predicted label of the sample video at time t;

[0338] According to the loss function, back propagation adjusts the parameters of the alignment module and the parameters of the recognition model until the training stop condition is met.

[0339] In one embodiment, the video tag recognition model further includes a low-order adaptive LoRA module of the model, and the training module 13 is used to:

[0340] Input the visual features of the sample video into the alignment module, and output the aligned visual features;

[0341] Input the aligned visual features, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction into the recognition model, and output the predicted label of the sample video at time t;

[0342] The parameters of the recognition model are fixed, and the parameters of the alignment module and the parameters of the LoRA module are adjusted according to the label of the sample video at time t and the predicted label of the sample video at time t until the training stop condition is met.

[0343] In one embodiment, the training module 13 is used to:

[0344] Construct a loss function based on the label of the sample video at time t and the predicted label of the sample video at time t;

[0345] According to the loss function, back propagation adjusts the parameters of the alignment module and the parameters of the LoRA module until the training stop condition is met.

[0346] In one embodiment, the training module 13 is further configured to:

[0347] Acquire a second training sample set, where each training sample in the second training sample set includes a sample video and text description information of the sample video;

[0348] The parameters of the visual feature encoder and the parameters of the recognition model are fixed, and the alignment module is trained according to the second training sample set to obtain a pre-trained alignment module.

[0349] In one embodiment, the training module 13 is used to:

[0350] For each training sample, input the sample video of the training sample into a visual feature encoder, and output the visual features of the sample video;

[0351] Cluster the visual features of the sample video using a clustering algorithm to obtain K cluster center features, where K is a positive integer;

[0352] Input the K class center features into the alignment module and output the aligned visual features;

[0353] Input the aligned visual features and the second preset paradigm instruction into the recognition module, and output the predicted description information of the sample video;

[0354] According to the text description information of the sample video and the predicted description information of the sample video, the parameters of the alignment module are adjusted until the training stop condition is met to obtain the alignment module.

[0355] Fig.10 A schematic diagram of the structure of a video tag recognition device provided in an embodiment of the present application is shown in FIG. Fig.10 As shown, the device may include: an acquisition module 21, a processing module 22 and an identification module 23.

[0356] The acquisition module 21 is used to acquire the video to be identified and the text description information of the video to be identified, and the video to be identified is input in the form of a video frame sequence;

[0357] The processing module 22 is used to input the video to be identified before time t into the visual feature encoder, and output the first visual feature;

[0358] The processing module 22 is further used to: cluster the first visual feature by a clustering algorithm to obtain K cluster center features, and write the K cluster center features into a storage queue;

[0359] The recognition module 23 is used to determine whether the label recognition condition is met, and determine the predicted label of the video to be recognized according to the K class center features stored in the storage queue, the text description information of the video to be recognized, the first preset paradigm instruction and the trained video label recognition model. Figure 4 The video label recognition model trained by the method shown includes a visual feature encoder, a storage queue, an alignment module and a recognition model.

[0360] In one embodiment, the identification module 23 is used to:

[0361] If a tag recognition instruction is received after the video frame sequence input of the video to be recognized is completed, it is determined that the tag recognition condition is met; or

[0362] Starting from time t, it is determined that the periodic tag identification time has been reached, and it is determined that the tag identification condition is met.

[0363] In one embodiment, the processing module 22 is further configured to:

[0364] Input the video to be identified before time t into the visual feature encoder, and output the first visual feature;

[0365] The first visual feature is clustered by a clustering algorithm to obtain K class center features, and the K class center features are written into a storage queue.

[0366] In one embodiment, the processing module 22 is used to:

[0367] Input the video to be identified from time t to time t1 into the visual feature encoder, and output the visual features of the video to be identified from time t to time t1;

[0368] According to the visual features of the video to be identified from time t to time t1, K class center features are clustered and updated, and the K class center features obtained after the update are determined as the visual features of the video to be identified.

[0369] In one embodiment, the processing module 22 is used to:

[0370] According to the video features of the video to be identified at time t, the K class center features are clustered and updated to obtain the updated K class center features and write them into the storage queue;

[0371] According to the video features of the video to be identified at each moment between time t and time t1 and the visual features of the video at time t1, the K class center features in the storage queue are clustered and updated to obtain the updated K class center features and write them into the storage queue.

[0372] In one embodiment, the video features of the video to be identified at time t include image block features of N video frames, where N is the number of image blocks. The processing module 22 is specifically configured to:

[0373] For each image block feature of a video frame in the video features of the video to be identified at time t, determine the corresponding target class center feature, where the target class center feature is the class center feature with the smallest Euclidean distance with the image block feature of the video frame;

[0374] According to the preset class center update parameters and the image block features of the video frame, the target class center features are clustered and updated.

[0375] In one embodiment, the processing module 22 is specifically used for:

[0376] The sum of the target class center feature and the first result is determined as the updated target class center feature, and the first result is the product of the difference between the image block feature of the video frame and the target class center feature and the class center update parameter.

[0377] In one embodiment, the identification module 23 is used to:

[0378] Input the visual features of the video to be identified into the alignment module, and output the aligned visual features;

[0379] The aligned visual features, the text description information of the video to be identified, the result of the previous label recognition of the video to be identified, and the first preset paradigm instruction are input into the recognition model, and the predicted label of the video to be identified is output.

[0380] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0381] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Fig. 9 The video label recognition model training device shown or Fig.10 The video tag identification device shown can execute the method embodiment corresponding to the computer device, and the aforementioned and other operations and / or functions of each module in the device are respectively for implementing the method embodiment corresponding to the computer device, which will not be repeated here for the sake of brevity.

[0382] The video tag recognition model training device and the video tag recognition device of the embodiment of the present application are described above from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or a combination of hardware and software modules in the decoding processor to execute. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0383] Fig.11 It is a schematic block diagram of a computer device 300 provided in an embodiment of the present application.

[0384] like Fig.11 As shown, the computer device 300 may include:

[0385] The memory 310 and the processor 320, the memory 310 is used to store the computer program and transmit the program code to the processor 320. In other words, the processor 320 can call and run the computer program from the memory 310 to implement the method in the embodiment of the present application.

[0386] For example, the processor 320 may be configured to execute the above method embodiments according to instructions in the computer program.

[0387] In some embodiments of the present application, the processor 320 may include but is not limited to:

[0388] General-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0389] In some embodiments of the present application, the memory 310 includes but is not limited to:

[0390] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct memory bus random access memory (Direct Rambus RAM, DR RAM).

[0391] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 310 and executed by the processor 320 to complete the method provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0392] like Fig.11 As shown, the computer device may also include:

[0393] The transceiver 330 may be connected to the processor 320 or the memory 310 .

[0394] The processor 320 may control the transceiver 330 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 330 may include a transmitter and a receiver. The transceiver 330 may further include an antenna, and the number of antennas may be one or more.

[0395] It should be understood that the various components in the electronic device are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0396] The embodiment of the present application also provides a computer storage medium on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the embodiment of the present application also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0397] When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0398] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present application.

[0399] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0400] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0401] The above contents are only specific implementation methods of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be included in the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be based on the protection scope of the claims.

Claims

1. A video tag recognition model training method, characterized in that: include: Acquire a first training sample set, where each training sample in the first training sample set includes a sample video, a label of the sample video at time t, and text description information of the sample video, where the time t is less than or equal to the duration of the sample video; For each training sample, determine the visual features of the sample video of the training sample, wherein the visual features of the sample video are obtained by clustering and updating K class center features stored in a storage queue according to the visual features of the sample video from time t0 to the time t, and the K class center features are obtained by clustering the visual features of the sample video before the time t0, wherein K is a positive integer, the time t0 is a preset value, and t0 is less than t; According to the visual features of the sample video, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction, the video label recognition model is trained until the training stop condition is met to obtain a trained video label recognition model, the video label recognition model includes a visual feature encoder, the storage queue, a pre-trained alignment module and a recognition model, and the first predicted label of the sample video is the result of the last label recognition performed by the video label recognition model on the sample video at time t; Among them, the visual feature encoder is used to convert the input video frames into visual features, the alignment module is used to align and dimensionally transform the visual features, map the visual features to the input dimension of the recognition model, and obtain aligned visual features, and the recognition model is used to take the aligned visual features, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction as input, and output the predicted label of the sample video at the time t.

2. The method according to claim 1, characterized in that The step of determining visual features of the sample video of the training sample comprises: Inputting the video of the sample video before the time t0 into the visual feature encoder, and outputting a first visual feature; Clustering the first visual features by a clustering algorithm to obtain K class center features, and writing the K class center features into the storage queue; Input the video of the sample video from time t0 to time t into the visual feature encoder, and output the visual features of the video of the sample video from time t0 to time t; According to the visual features of the sample video from time t0 to time t, the K class center features are clustered and updated, and the K class center features obtained after the update are determined as the visual features of the sample video of the training sample.

3. The method according to claim 2, characterized in that The clustering and updating of the K cluster center features according to the visual features of the sample video from time t0 to time t includes: According to the video features of the sample video at time t0, clustering and updating the K class center features, obtaining the updated K class center features and writing them into the storage queue; The K class center features in the storage queue are clustered and updated according to the video features of the sample video at each moment between time t0 and time t and the visual features of the video at time t, and the updated K class center features are obtained and written into the storage queue.

4. The method according to claim 3, characterized in that The video features of the sample video at time t0 include image block features of N video frames, where N is the number of image blocks, and clustering and updating the K class center features according to the video features of the sample video at time t0 includes: For each image block feature of a video frame in the video features of the sample video at time t0, determine a corresponding target class center feature, wherein the target class center feature is a class center feature having the smallest Euclidean distance with the image block feature of the video frame; The target class center features are clustered and updated according to preset class center update parameters and the image block features of the video frame.

5. The method according to claim 4, characterized in that The clustering and updating of the target class center features according to the preset class center update parameters and the image block features of the video frame includes: The sum of the target class center feature and the first result is determined as the updated target class center feature, where the first result is the product of the difference between the image block feature of the video frame and the target class center feature and the class center update parameter.

6. The method according to any one of claims 1 to 5, characterized in that: The step of training the video label recognition model according to the visual features of the sample video, the text description information of the sample video, the first predicted label of the sample video, and the first preset paradigm instruction includes: Inputting the visual features of the sample video into the alignment module, and outputting the aligned visual features; Inputting the aligned visual features, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction into the recognition model, and outputting the predicted label of the sample video at time t; According to the label of the sample video at time t and the predicted label of the sample video at time t, the parameters of the alignment module and the parameters of the recognition model are adjusted until the training stop condition is met.

7. The method according to claim 6, characterized in that The step of adjusting the parameters of the alignment module and the parameters of the recognition model according to the label of the sample video at time t and the predicted label of the sample video at time t until a training stop condition is met includes: Constructing a loss function according to the label of the sample video at time t and the predicted label of the sample video at time t; According to the loss function, back propagation is used to adjust the parameters of the alignment module and the parameters of the recognition model until a training stop condition is met.

8. The method according to any one of claims 1 to 5, characterized in that: The video tag recognition model also includes a low-order adaptive LoRA module of the model, and the video tag recognition model is trained according to the visual features of the sample video, the text description information of the sample video, the first predicted tag of the sample video, and the first preset paradigm instruction, including: Inputting the visual features of the sample video into the alignment module, and outputting the aligned visual features; Inputting the aligned visual features, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction into the recognition model, and outputting the predicted label of the sample video at time t; The parameters of the recognition model are fixed, and the parameters of the alignment module and the parameters of the LoRA module are adjusted according to the label of the sample video at time t and the predicted label of the sample video at time t until the training stop condition is met.

9. The method according to claim 8, characterized in that The step of adjusting the parameters of the alignment module and the parameters of the LoRA module according to the label of the sample video at time t and the predicted label of the sample video at time t until the training stop condition is met includes: Constructing a loss function according to the label of the sample video at time t and the predicted label of the sample video at time t; According to the loss function, back propagation adjusts the parameters of the alignment module and the parameters of the LoRA module until the training stop condition is met.

10. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: Acquire a second training sample set, wherein each training sample in the second training sample set includes a sample video and text description information of the sample video; The parameters of the visual feature encoder and the parameters of the recognition model are fixed, and an alignment module is trained according to the second training sample set to obtain the pre-trained alignment module.

11. The method according to claim 10, characterized in that The training of the alignment module according to the second training sample set to obtain the pre-trained alignment module includes: For each training sample, inputting a sample video of the training sample into the visual feature encoder, and outputting visual features of the sample video; Clustering the visual features of the sample video using a clustering algorithm to obtain K cluster center features, where K is a positive integer; Input the K class center features into an alignment module, and output aligned visual features; Inputting the aligned visual features and the second preset paradigm instruction into the recognition model, and outputting the predicted description information of the sample video; According to the text description information of the sample video and the predicted description information of the sample video, the parameters of the alignment module are adjusted until a training stop condition is met to obtain the alignment module.

12. A video tag recognition method, characterized in that: include: Acquire a video to be identified and text description information of the video to be identified, wherein the video to be identified is input in the form of a video frame sequence; Determine whether the label recognition condition is met, obtain the visual features of the video to be recognized, the visual features of the video to be recognized are obtained by clustering and updating the K class center features stored in the storage queue according to the visual features of the video to be recognized from time t to time t1, the K class center features are obtained by clustering the visual features of the video to be recognized before the time t, K is a positive integer, the time t1 is the duration of the video to be recognized when recognizing the video to be recognized, and t1 is less than or equal to the duration of the video to be recognized; Determine a predicted label of the video to be identified according to the visual features of the video to be identified, the text description information of the video to be identified, a first preset paradigm instruction and a trained video label recognition model, wherein the video label recognition model includes a visual feature encoder, the storage queue, an alignment module and a recognition model; Among them, the visual feature encoder is used to convert the input video frame into visual features, the alignment module is used to align and dimensionally transform the visual features, map the visual features to the input dimension of the recognition model, and obtain aligned visual features, and the recognition model is used to use the aligned visual features, the text description information of the video to be recognized, the result of the previous label recognition of the video to be recognized, and the first preset paradigm instruction as input, and output the predicted label of the video to be recognized.

13. The method according to claim 12, characterized in that The determining that the tag identification condition is satisfied includes: If a tag recognition instruction is received which is triggered after the video frame sequence input of the video to be recognized is completed, it is determined that the tag recognition condition is met; or, Starting from the time t, it is determined that the periodic tag identification time has been reached, and then it is determined that the tag identification condition is met.

14. The method according to claim 12, characterized in that The method further comprises: Inputting the video to be identified before time t into a visual feature encoder, and outputting a first visual feature; The first visual features are clustered by a clustering algorithm to obtain K class center features, and the K class center features are written into the storage queue.

15. The method according to claim 12, characterized in that The obtaining of visual features of the video to be identified includes: Input the video of the video to be identified from time t to time t1 into the visual feature encoder, and output the visual features of the video to be identified from time t to time t1; According to the visual features of the video to be identified from time t to time t1, the K class center features are clustered and updated, and the K class center features obtained after the update are determined as the visual features of the video to be identified.

16. The method according to claim 12, characterized in that The step of determining the predicted label of the video to be identified according to the visual features of the video to be identified, the text description information of the video to be identified, the first preset paradigm instruction and the trained video label identification model comprises: Inputting the visual features of the video to be identified into the alignment module, and outputting the aligned visual features; The aligned visual features, the text description information of the video to be identified, the result of the previous label identification of the video to be identified, and the first preset paradigm instruction are input into the recognition model, and the predicted label of the video to be identified is output.

17. A video tag recognition model training device, characterized in that: include: An acquisition module, configured to acquire a first training sample set, wherein each training sample in the first training sample set includes a sample video, a label of the sample video at time t, and text description information of the sample video, wherein the time t is less than or equal to the duration of the sample video; A determination module, for determining, for each training sample, a visual feature of a sample video of the training sample, wherein the visual feature of the sample video is obtained by clustering and updating K class center features stored in a storage queue according to the visual features of the sample video from time t0 to the time t, wherein the K class center features are obtained by clustering the visual features of the sample video before the time t0, wherein K is a positive integer, the time t0 is a preset value, and t0 is less than t; A training module is used to train a video label recognition model according to the visual features of the sample video, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction until a stop training condition is met, thereby obtaining a trained video label recognition model, wherein the video label recognition model includes a visual feature encoder, the storage queue, a pre-trained alignment module and a recognition model, and the first predicted label of the sample video is the result of the last label recognition performed by the video label recognition model on the sample video at time t; Among them, the visual feature encoder is used to convert the input video frames into visual features, the alignment module is used to align and dimensionally transform the visual features, map the visual features to the input dimension of the recognition model, and obtain aligned visual features, and the recognition model is used to take the aligned visual features, the text description information of the sample video, the first predicted label of the sample video and the first preset paradigm instruction as input, and output the predicted label of the sample video at the time t.

18. A video tag recognition device, characterized in that: include: An acquisition module, used for acquiring a video to be identified and text description information of the video to be identified, wherein the video to be identified is input in the form of a video frame sequence; A processing module, used to determine whether a label recognition condition is satisfied, and obtain visual features of the video to be recognized, wherein the visual features of the video to be recognized are obtained by clustering and updating K class center features stored in a storage queue according to the visual features of the video to be recognized from time t to time t1, and the K class center features are obtained by clustering the visual features of the video to be recognized before the time t, wherein K is a positive integer, and the time t1 is the duration of the video to be recognized when the video to be recognized is recognized, and t1 is less than or equal to the duration of the video to be recognized; An identification module, used to determine whether a label identification condition is met, and determine a predicted label of the video to be identified according to the visual features of the video to be identified, the text description information of the video to be identified, the first preset paradigm instruction and the trained video label identification model, wherein the video label identification model includes a visual feature encoder, the storage queue, an alignment module and a recognition model; Among them, the visual feature encoder is used to convert the input video frame into visual features, the alignment module is used to align and dimensionally transform the visual features, map the visual features to the input dimension of the recognition model, and obtain aligned visual features, and the recognition model is used to use the aligned visual features, the text description information of the video to be recognized, the result of the previous label recognition of the video to be recognized, and the first preset paradigm instruction as input, and output the predicted label of the video to be recognized.

19. A computer device, characterized in that: include: A processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the video tag recognition model training method described in any one of claims 1 to 11 or the video tag recognition method described in any one of claims 12 to 16.

20. A computer-readable storage medium, characterized in that: The method comprises instructions which, when executed on a computer, enable the computer to execute the video tag recognition model training method according to any one of claims 1 to 11 or the video tag recognition method according to any one of claims 12 to 16.

Images