Video classification model training methods, devices, electronic equipment and storage media

Abstract

This invention discloses a video classification model training method, apparatus, electronic device, and storage medium. The method includes: acquiring a currently labeled video sample dataset; training a classification task based on the currently labeled video sample dataset to obtain a current video classification model; acquiring target classification processing information based on the current video classification model; selecting target remaining unlabeled video sample data as new unlabeled video sample data from the remaining unlabeled video sample data based on the target classification processing information; ensuring that the difference between the target remaining unlabeled video sample data and the currently labeled video sample dataset is greater than that of the unselected remaining unlabeled video sample data; updating the currently labeled video sample dataset based on the new unlabeled video sample data, until a preset training termination condition is met to obtain the target video classification model. This invention not only saves labeling costs but also provides strong stability in the training method and improves the iterative efficiency of the model.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a video classification model training method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, classifying and labeling videos using video classification models has become an important way to improve the efficiency of downstream video distribution. Video classification models are typically trained on neural networks using deep learning algorithms.

[0003] In related technologies, in the process of implementing video classification models based on deep learning, the fully supervised training method is adopted. This method involves labeling large-scale video data to construct a large-scale labeled video dataset for training the model for video classification tasks. This not only consumes a lot of time and manpower for labeling, but also results in poor stability of the training method and low model iteration efficiency. Summary of the Invention

[0004] To address the problems of existing technologies, embodiments of the present invention provide a video classification model training method, apparatus, electronic device, and storage medium. The technical solution is as follows:

[0005] On the one hand, a method for training a video classification model is provided, the method comprising:

[0006] Obtain the current labeled video sample dataset, and train the current video classification model based on the current labeled video sample dataset; the labeled video sample data in the current labeled video sample dataset is obtained by labeling the video sample data to be labeled, and the video sample data to be labeled is the unlabeled video sample data selected from the unlabeled video sample dataset;

[0007] The labeled video sample data and the remaining unlabeled video sample data are respectively input into the current video classification model for classification processing to obtain the target classification processing information corresponding to the labeled video sample data and the remaining unlabeled video sample data respectively; the remaining unlabeled video sample data refers to the unlabeled video sample data in the unlabeled video sample dataset that was not selected as the video sample data to be labeled.

[0008] Based on the target classification processing information corresponding to the labeled video sample data and the remaining unlabeled video sample data, a first preset number of target remaining unlabeled video sample data are selected from the remaining unlabeled video sample data as newly added video sample data to be labeled; the degree of difference between the target remaining unlabeled video sample data and the current labeled video sample dataset is greater than the degree of difference between the unselected remaining unlabeled video sample data and the current labeled dataset;

[0009] The current labeled video sample dataset is updated based on the newly added unlabeled video sample data until the preset training termination condition is met and the training ends; wherein, the current video classification model at the end of training is used as the target video classification model.

[0010] On the other hand, a video classification model training device is provided, the device comprising:

[0011] The current video classification model training module is used to obtain the current labeled video sample dataset, and to train the current video classification model based on the current labeled video sample dataset. The labeled video sample data in the current labeled video sample dataset is obtained by labeling the unlabeled video sample data, and the unlabeled video sample data is unlabeled video sample data selected from the unlabeled video sample dataset.

[0012] The classification processing information acquisition module is used to input the labeled video sample data and the remaining unlabeled video sample data into the current video classification model for classification processing, and to obtain the target classification processing information corresponding to the labeled video sample data and the remaining unlabeled video sample data respectively; the remaining unlabeled video sample data refers to the unlabeled video sample data in the unlabeled video sample dataset that was not selected as the video sample data to be labeled.

[0013] The target unlabeled video sample data selection module is used to select a first preset number of target unlabeled video sample data as new video sample data to be labeled based on the target classification processing information corresponding to the labeled video sample data and the remaining unlabeled video sample data, respectively; the degree of difference between the target unlabeled video sample data and the current labeled video sample dataset is greater than the degree of difference between the unselected remaining unlabeled video sample data and the current labeled dataset;

[0014] The labeled video sample dataset update module is used to update the current labeled video sample dataset based on the newly added unlabeled video sample data until the preset training termination condition is met to end the training; wherein, the current video classification model at the end of the training is used as the target video classification model.

[0015] In one exemplary embodiment, the target classification processing information is the feature obtained by feature extraction during the classification processing; the target unlabeled video sample data selection module includes:

[0016] The first determining module is used to determine the current first feature set and the current second feature set; wherein, the first feature in the first feature set includes the extracted features corresponding to the labeled video sample data, and the second feature in the second feature set is the extracted features corresponding to the remaining unlabeled video sample data;

[0017] The first similarity determination module is used to determine the first similarity between each second feature in the current second feature set and the current first feature set;

[0018] The selection module is used to determine the target second feature corresponding to the minimum first similarity, and select the remaining unlabeled video sample data corresponding to the target second feature as the target remaining unlabeled video sample data.

[0019] The update module is used to update the current first feature set and the current second feature set based on the target second feature, until a first preset number of remaining unlabeled video sample data of the target are obtained; wherein, the updated current first feature set includes the target second feature and uses the target second feature as the first feature, and the updated current second feature set does not include the target second feature;

[0020] The second determining module is used to take the first preset number of the remaining unlabeled video sample data of the target as the newly added video sample data to be labeled.

[0021] In one exemplary implementation, the first similarity determination module includes:

[0022] The second similarity determination module is used to determine the second similarity between each second feature in the current second feature set and each first feature in the current first feature set for each second feature in the current second feature set;

[0023] The third determining module is used to take the maximum second similarity as the first similarity between the second feature and the first feature set.

[0024] In one exemplary implementation, the current video classification model training module includes:

[0025] An unlabeled video sample dataset acquisition module is used to acquire the unlabeled video sample dataset; the unlabeled video sample dataset includes unlabeled video sample data corresponding to at least one preset video category;

[0026] The initial unlabeled video sample data selection module is used to select unlabeled video sample data from the unlabeled video sample data corresponding to each preset video category as unlabeled video sample data to obtain an initial unlabeled video sample dataset; the total number of unlabeled video sample data in the initial unlabeled video sample dataset is less than the total number of unlabeled video sample data in the unlabeled video sample dataset;

[0027] The annotation processing module is used to annotate the unannotated video sample data in the initial unannotated video sample dataset to obtain the current annotated video sample dataset.

[0028] In one exemplary implementation, the unlabeled video sample dataset acquisition module includes:

[0029] A sample video set acquisition module is used to acquire a sample video set; the sample video set includes sample videos corresponding to the at least one preset video category;

[0030] A multimodal data determination module is used to determine the multimodal data corresponding to each sample video; the multimodal data includes video frames, audio data, and text data of the corresponding sample video;

[0031] The fourth determining module is used to take the multimodal data corresponding to each of the sample videos as unlabeled video sample data to obtain the unlabeled video sample dataset.

[0032] In one exemplary implementation, the classification task includes any one of binary classification, multi-class classification, and multi-label classification.

[0033] On the other hand, an electronic device is provided, including a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the video classification model training method of any of the above aspects.

[0034] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement the video classification model training method as described above.

[0035] On the other hand, a computer program product or computer program is provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the video classification model training method of any of the above aspects.

[0036] This invention provides an embodiment of the invention that obtains a current labeled video sample dataset, trains a classification task based on the current labeled video sample dataset to obtain a current classification model, and inputs labeled video sample data and remaining unlabeled video sample data into the current classification model for classification processing and obtains their respective target classification processing information. Then, based on the target classification processing information corresponding to the labeled video sample data and the remaining unlabeled video sample data, a first preset number of target remaining unlabeled video sample data are selected from the remaining unlabeled video sample data as newly added unlabeled video sample data, and the current labeled video sample dataset is updated using the newly added unlabeled video sample data until the preset training termination condition is met to end the training and obtain the target classification model. In the above technical solution, each iteration of training selects a batch of unlabeled video samples that are more helpful to improve model training from the unlabeled video samples based on the target classification information corresponding to the labeled video sample data and the remaining unlabeled video sample data obtained by the current video classification model. After labeling, the data is updated to the current labeled video sample dataset, and training continues iteratively to improve the model's performance. Compared with the fully supervised training method in related technologies, it can save at least 50% of the labeling cost without affecting the model's classification effect. Moreover, this training method has strong stability and greatly improves the overall iteration efficiency of the classification model. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating a video classification model training method provided in an embodiment of the present invention;

[0039] Figure 2 This is a flowchart illustrating the process of selecting a first preset number of target remaining unlabeled video sample data in each iteration, as provided in an embodiment of the present invention.

[0040] Figure 3 Based on Figure 2The diagram shows the data distribution selected by the method shown.

[0041] Figure 4 This is a schematic diagram illustrating the automatic updating of an online target video classification model provided in an embodiment of the present invention;

[0042] Figure 5 This is a schematic diagram of the process of training a video classification model provided in an embodiment of the present invention;

[0043] Figure 6 This is an example of training a video classification model provided in an embodiment of the present invention;

[0044] Figure 7(a) is a schematic diagram of the training effect of the classification model for video classification provided in the embodiment of the present invention.

[0045] Figure 7(b) is a schematic diagram of the training effect of the classification model for video multi-label classification provided in the embodiment of the present invention;

[0046] Figure 8 This is a structural block diagram of a video classification model training device provided in an embodiment of the present invention;

[0047] Figure 9 This is a hardware structure block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0050] It is understood that in the specific embodiments of this application, data such as user information are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0051] The embodiments of the present invention can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0052] Artificial Intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to have perception, reasoning, and decision-making capabilities. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0053] Computer vision (CV) is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0054] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0055] Key technologies in speech technology include Automatic Speech Recognition (ASR), Text-to-Speech (TTS), and Voiceprint Recognition. Enabling computers to hear, see, speak, and feel is the future direction of human-computer interaction, with speech emerging as one of the most promising methods.

[0056] It should be noted that the classification model training method of the present invention can be applied to a classification model training device, which can be configured in an electronic device, which can be a terminal or a server.

[0057] Terminals include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, and aircraft. Servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0058] In one exemplary embodiment, all electronic devices can be node devices in a blockchain system, capable of sharing acquired and generated information with other node devices in the blockchain system, thereby achieving information sharing among multiple node devices. Multiple node devices in the blockchain system can be configured with the same blockchain, which consists of multiple blocks, and adjacent blocks are related, ensuring that any data tampering in any block can be detected by the next block, thus preventing data tampering in the blockchain and guaranteeing the security and reliability of the data in the blockchain.

[0059] The classification model training method of this invention will be described in detail below.

[0060] Please see Figure 1The diagram shown is a flowchart of a video classification model training method provided in an embodiment of the present invention. It should be noted that the video classification model training method in this embodiment includes multiple iterative training steps. Figure 1 This will be illustrated using only one iteration of the training process.

[0061] Additionally, it should be noted that this specification provides the operational steps of the methods described in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operational steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual system or product execution, the methods shown in the embodiments or drawings can be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 1 As shown, the method may include:

[0062] S101, Obtain the current labeled video sample dataset, and train the current video classification model based on the current labeled video sample dataset.

[0063] The labeled video sample data in the current labeled video sample dataset is obtained by labeling the video sample data to be labeled, and the video sample data to be labeled is the unlabeled video sample data selected from the unlabeled video sample dataset.

[0064] The unlabeled video sample dataset may include unlabeled video sample data of at least one preset video category. The at least one preset video category is the video category that the target video classification model is expected to be able to recognize after training. It can be set according to the actual needs of the business to which the samples belong. Each preset video category may correspond to multiple unlabeled video sample data.

[0065] The classification task can be any of the following: binary classification, multi-class classification, or multi-label classification. Binary classification is used to predict whether labeled video sample data belongs to a certain preset video category; multi-class classification is used to predict which of the multiple preset video categories the labeled video sample data belongs to; and multi-label classification is used to predict one or more possible preset video category labels for the labeled video sample data.

[0066] The annotation information corresponding to the labeled video sample data is matched with the classification task corresponding to the training task. When the classification task is a binary classification task, the annotation information corresponding to each labeled video sample data indicates whether the labeled video sample data belongs to a certain preset video category. For example, 1 indicates that it belongs to a certain preset video category, and 0 indicates that it does not belong to a certain preset video category. When the classification task is a multi-class classification task, the annotation information corresponding to each labeled video sample data indicates which preset video category the labeled video sample data belongs to. That is, a data point has only one category label, but there are multiple category labels. When the classification task is a multi-label classification task, the annotation information corresponding to each labeled video sample data indicates which one or more preset video categories the labeled video sample data belongs to. That is, a data point may have one or more category labels.

[0067] The current video classification model can be a deep learning model. In specific implementations, the deep learning model can be selected based on the modalities of the information contained in the unlabeled video sample data. Since videos involve three modalities of information—image, audio, and text—a deep learning model capable of handling multimodal information can be selected for training the classification task to obtain the current video classification model. This deep learning model can be a single-stream Transformer or a multi-stream Transformer. A single-stream Transformer inputs features / embeddings from different modalities into the same Transformer to capture intermodal information, while a multi-stream Transformer inputs each modality into an independent Transformer to capture intramodal information and constructs cross-modal relationships through other means (e.g., another Transformer).

[0068] Of course, this deep learning model can also be other neural network models capable of performing classification tasks based on multimodal information, such as neural network models with a late-fusion structure. Here, late-fusion is relative to early-fusion. Early-fusion refers to fusion at the feature level, where different features are concatenated and input into a single model for training. Late-fusion, on the other hand, refers to fusion at the score level. This involves training multiple models, each with a prediction score, and then fusion the results of all models to obtain the final prediction result. Common late-fusion methods include taking the average score, the maximum score, and the weighted average.

[0069] In one exemplary implementation, the current labeled video sample dataset can be the initial labeled video sample dataset, that is, the labeled video sample dataset used in the first iteration of training. Accordingly, the current video classification model is the initial classification model that has not yet been trained for the classification task. Therefore, the above step S101 may include the following steps when obtaining the current labeled video sample dataset:

[0070] Obtain an unlabeled video sample dataset; the unlabeled video sample dataset includes unlabeled video sample data corresponding to at least one preset video category;

[0071] Unlabeled video sample data is selected from the unlabeled video sample data corresponding to each preset video category as video sample data to be labeled, to obtain an initial unlabeled video sample dataset. The total number of unlabeled video sample data in the initial unlabeled video sample dataset is less than the total number of unlabeled video sample data in the unlabeled video sample dataset.

[0072] The unannotated video sample data in the initial unannotated video sample dataset is annotated to obtain the current annotated video sample dataset.

[0073] Specifically, for each preset video category, videos of that preset video category can be obtained from a massive amount of videos to obtain unlabeled video sample data.

[0074] When selecting the initial unlabeled video sample dataset from the unlabeled video sample dataset, random selection is possible. However, considering that random selection may not cover all preset video categories, this would be detrimental to improving the model's scalability. To accelerate model iteration and improve training speed, this embodiment of the invention selects a certain number of unlabeled video samples from the unlabeled video sample data corresponding to each preset video category when selecting the initial unlabeled video sample dataset. Then, the unlabeled video sample data selected from each preset video category are merged into the initial unlabeled video sample dataset. The total number of unlabeled video samples in the initial unlabeled video sample dataset is controlled to be less than the total number of unlabeled video samples in the unlabeled video sample dataset. Then, the unlabeled video sample data in the initial unlabeled video sample dataset is labeled, for example, manually. The labeled initial unlabeled video sample data can then be used as the current labeled video sample dataset for training the initial classification model for classification tasks in the first iteration of training.

[0075] Understandably, to reduce the cost of manual annotation, the current annotated video sample dataset used for the first iteration of training can be designed to minimize the total number of corresponding unannotated video sample data without affecting training speed. For example, the current annotated video sample dataset for the first iteration of training can be approximately 10% of the unannotated video sample dataset; that is, the total number of unannotated video sample data in the initial unannotated video sample dataset can be controlled to be 10% of the total number of unannotated video sample data in the unannotated video sample dataset.

[0076] In the above implementation, when obtaining the current labeled video sample dataset for the first iteration training, unlabeled video sample data is selected from the unlabeled video sample data corresponding to each preset video category to form the initial unlabeled video sample dataset. This makes the current labeled video sample dataset obtained based on the initial unlabeled video sample dataset have a higher coverage of the target video category (i.e., multiple preset video categories), which is beneficial to improve the model's ascent speed and thus improve the training speed.

[0077] In one specific implementation, to improve the training effect of the video classification model and thus increase the classification accuracy of the trained target video classification model, the acquisition of the unlabeled video sample dataset may include:

[0078] Obtain a sample video set; the sample video set includes sample videos corresponding to at least one preset video category;

[0079] Determine the multimodal data corresponding to each sample video; the multimodal data includes the video frames, audio data, and text data of the corresponding sample video;

[0080] The multimodal data corresponding to each sample video is used as unlabeled video sample data to obtain the unlabeled video sample dataset.

[0081] Specifically, multiple preset video categories can be set according to the categories required in actual applications, such as lifestyle, beauty, etc.

[0082] Text data in multimodal data can include video titles, text content obtained by OCR processing of video frames, and text content obtained by ASR processing of corresponding audio data. OCR (Optical Character Recognition) is used to detect and recognize text content from images; ASR (Automatic Speech Recognition) is used to convert speech into text.

[0083] S103, the labeled video sample data and the remaining unlabeled video sample data are respectively input into the current video classification model for classification processing to obtain the target classification processing information corresponding to the labeled video sample data and the remaining unlabeled video sample data respectively.

[0084] The remaining unlabeled video sample data refers to the unlabeled video sample data in the unlabeled video sample dataset that was not selected as the video sample data to be labeled. Assuming the current labeled video sample dataset is L, and the remaining unlabeled video sample dataset is U, the total number of unlabeled video sample data in the unlabeled video sample dataset is the sum of the number of remaining unlabeled video sample data in U and the number of labeled video sample data in L.

[0085] The target classification information can be determined based on the data selection strategy used in the actual application. For example, when using the k-center greedy algorithm as the data selection strategy, the target classification information can be the features extracted during the classification process. Specifically, it can be the output of the fully connected layer preceding the classifier in the current video classification model, which is used to predict the classification result based on the output of its preceding fully connected layer. In other data selection strategies, such as Deep Bayesian Active Learning (DBAL), the classification result can also be used as the target classification information.

[0086] Taking the target classification information as the feature extracted during the classification process as an example, the labeled video sample data in the current labeled video sample dataset are input into the current video classification model and the output of the fully connected layer before the classifier is extracted to obtain the first feature corresponding to each labeled video sample data, and thus the first feature set is obtained; similarly, the remaining unlabeled video sample data are input into the current video classification model and the output of the fully connected layer before the classifier is extracted to obtain the second feature corresponding to each remaining unlabeled video sample data, and thus the second feature set is obtained.

[0087] S105, based on the target classification processing information corresponding to the labeled video sample data and the remaining unlabeled video sample data respectively, select a first preset number of target remaining unlabeled video sample data from the remaining unlabeled video sample data as newly added video sample data to be labeled.

[0088] Specifically, the difference between the remaining unlabeled video sample data and the currently labeled video sample dataset is greater than the difference between the remaining unlabeled video sample data that was not selected and the currently labeled video sample dataset.

[0089] Specifically, for each remaining unlabeled video sample data, based on the target classification processing information corresponding to the remaining unlabeled video sample data and the target classification processing information corresponding to each labeled video sample data in the current labeled video sample dataset, the sub-difference degree between the remaining unlabeled video sample data and each labeled video sample data is determined. Then, the minimum sub-difference degree is taken as the difference degree between the remaining unlabeled video sample data and the current labeled video sample dataset. Furthermore, target remaining unlabeled video sample data is selected based on the difference degree between each remaining unlabeled video sample data and the current labeled video sample dataset, so that the difference degree between the target remaining unlabeled video sample data and the current labeled video sample dataset is greater than the difference degree between the unselected remaining unlabeled video sample data and the current labeled video sample dataset.

[0090] For example, the degree of difference can be represented by the degree of similarity. Generally, the greater the degree of similarity, the smaller the degree of difference; conversely, the smaller the degree of similarity, the greater the degree of difference.

[0091] The first preset number can be set according to the actual unlabeled video sample dataset, for example, it can be 10% of the total number of samples in the unlabeled video sample dataset.

[0092] In an exemplary implementation, the target classification processing information can be the features extracted during the classification process of the current video classification model, which can then achieve a more accurate expression of the data distribution rules from the perspective of feature distribution. That is, the first feature set and the second feature set can be obtained through the above step S103. The first feature in the first feature set is the extracted feature corresponding to the labeled video sample data, and the second feature in the second feature set is the extracted feature corresponding to the remaining unlabeled video sample data.

[0093] Based on this, such as Figure 2 As shown, step S105 above, when selecting a first preset number of target remaining unlabeled video sample data as new video sample data to be labeled based on the target classification processing information corresponding to the labeled video sample data and the remaining unlabeled video sample data respectively, may include:

[0094] S1051, determine the current first feature set and the current second feature set.

[0095] It should be noted that in this embodiment of the invention, the first preset number of target remaining unlabeled video sample data is selected in a loop. In each loop, one is selected from the remaining unlabeled video sample data as the target remaining unlabeled video sample data. Therefore, the current first feature set and the current second feature set are the first feature set and the second feature set involved in the current loop.

[0096] It is understandable that the current first feature set in the first loop is the first feature set obtained through step S103, and the current second feature set in the first loop is the second feature set obtained through step S103.

[0097] S1053, determine the first similarity between each second feature in the current second feature set and the current first feature set.

[0098] The first similarity level represents the degree of similarity between the corresponding second feature and the current first feature set as a whole.

[0099] Specifically, the distance between the second feature and the current first feature set can be used to characterize the first similarity. The larger the distance, the smaller the first similarity, and vice versa. In practice, distances such as Euclidean distance (also known as L2 distance) and Manhattan distance (also known as L1 distance) can be used.

[0100] In order to measure the first similarity between the second feature and the current first feature set, in an exemplary embodiment, for each second feature in the current second feature set, the second similarity between the second feature and each first feature in the current first feature set can be determined; the maximum second similarity is taken as the first similarity between the second feature and the first feature set.

[0101] Specifically, the degree of similarity between the second feature and the first feature can be represented by distance. The larger the distance, the smaller the degree of similarity, and vice versa. In practice, distances such as Euclidean distance (also known as L2 distance) and Manhattan distance (also known as L1 distance) can be used.

[0102] S1055, determine the target second feature corresponding to the minimum first similarity, and select the remaining unlabeled video sample data corresponding to the target second feature as the target remaining unlabeled video sample data.

[0103] Specifically, the smallest first similarity can be found from the multiple first similarities determined in step S1053, and then the second feature corresponding to the smallest first similarity can be determined as the target second feature. Thus, the remaining unlabeled video sample data corresponding to the target second feature can be selected as the target remaining unlabeled video sample data.

[0104] Understandably, selecting the remaining unlabeled video sample data of the target based on the second feature of the target corresponding to the minimum first similarity can maximize the difference between the remaining unlabeled video sample data of the target and the current labeled video sample dataset, which is more conducive to improving the training efficiency of the model.

[0105] S1057, Update the current first feature set and the current second feature set based on the target second feature, until a first preset number of target remaining unlabeled video sample data are obtained.

[0106] The updated current first feature set includes the target second feature and uses the target second feature as the first feature, while the updated current second feature set does not include the target second feature.

[0107] In specific implementation, the second feature of the target is added to the current first feature set as the first feature to update the current first feature set, and the second feature of the target is deleted from the current second feature set to update the current second feature set.

[0108] After updating the current first feature set and the current second feature set, steps S1051 to S1057 can be continued until the first preset number of target remaining unlabeled video sample data are selected.

[0109] For example, the remaining unlabeled video sample data for each target can be represented by the following formula: u = arg max i∈U min j∈L Δ(x i x j )

[0110] Wherein, the currently labeled video sample dataset is L, and the remaining unlabeled video sample dataset is U, x i Representing the second feature i, x j Let j represent the first feature; u represent the remaining unlabeled video sample data of the target; Δ() represents the distance calculation.

[0111] S1059, the first preset number of the remaining unlabeled video sample data of the target are used as the newly added video sample data to be labeled.

[0112] Specifically, since the remaining unlabeled video sample data refers to the unlabeled video sample data that has not been selected as the video sample data to be labeled, when the first preset number of target remaining unlabeled video sample data is used as the newly added video sample data to be labeled, the remaining unlabeled video sample data is updated, that is, the first preset number of target remaining unlabeled video sample data is removed from the remaining unlabeled video sample data.

[0113] The above implementation method, from an optimization perspective, provides a subset that can well represent the overall data, allowing the data to better reflect the feature space coverage pattern, which is beneficial to improving the iterative efficiency of the model. For example... Figure 3 As shown in the figure, taking the first preset quantity k as 6 as an example, based on steps S1051 to S1059, c1, c2, c3, c4, c5, and c6 can be selected from the input data, and these 6 data can well characterize the feature space coverage pattern, where Δ(C) represents the radius of the circle in the figure.

[0114] S107, Update the current labeled video sample dataset based on the newly added unlabeled video sample data until the preset training termination condition is met to end the training.

[0115] The current video classification model that meets the preset training termination conditions is used as the target video classification model.

[0116] The preset training termination condition can be set according to actual needs. For example, it could be that the loss value of a single training iteration reaches a preset minimum loss threshold, or that the difference between the loss values ​​of two adjacent training iterations reaches a preset loss difference threshold. Of course, other methods for evaluating the model training effect can also be used to determine when to end the training.

[0117] Specifically, the newly added unannotated video sample data is labeled to obtain newly labeled video sample data. This newly labeled video sample data is then added to the current labeled video sample dataset, thus updating the current labeled video sample dataset. In the next training iteration, training will be conducted based on the updated current labeled video sample dataset and the updated remaining unlabeled video sample data.

[0118] Suppose the current labeled video sample dataset is L, the remaining unlabeled video sample dataset is U, and the newly added unlabeled video sample data (i.e., the first preset number of target remaining unlabeled video sample data) is {u}. Then, the update process of the current labeled video sample dataset can be represented as L=L∪{u}, and the update of the remaining unlabeled video sample data can be represented as U=U / {u}, where U represents the remaining unlabeled video sample dataset, and " / " represents removal.

[0119] Understandably, after the current labeled video sample dataset is updated, steps S101 to S107 can be executed to perform the next iteration of training until the preset termination condition is met to obtain the target video classification model.

[0120] As can be seen from the above technical solutions of the embodiments of the present invention, the embodiments of the present invention start with a small-scale dataset, and in each iteration of training, select a batch of remaining unlabeled video sample data that is more helpful to improve model training based on the target classification processing information corresponding to the labeled video sample data and the remaining unlabeled video sample data obtained by the current classification model, label them, update them to the current labeled video sample dataset, and continue iterative training to improve the performance of the model. Compared with the fully supervised training method in related technologies, it can save at least 50% of the labeling cost without affecting the model classification effect. Moreover, the training method has strong stability, greatly improves the iterative efficiency of the overall video classification model, and improves the training speed.

[0121] Furthermore, active learning methods in related technologies lack robustness, exhibiting significant performance variations under different experimental conditions. Traditional strategic algorithms, such as uncertainty sampling (including least confidence sampling, margin sampling, and entropy sampling), cannot be directly applied effectively to multi-label classification scenarios. However, the data selection method in the active learning process of this invention makes the active learning approach highly versatile and stable, making it well-suited for multi-label video classification.

[0122] In one exemplary embodiment, the target video classification model obtained in this invention can be deployed online. The video classification model training method provided in this invention can be used to automatically update the online target video classification model. During automatic updates, for newly added preset video categories, unlabeled video data corresponding to the newly added preset video category can be extracted to obtain a newly added unlabeled video sample dataset. Then, based on this newly added unlabeled video sample dataset, the following method can be used: Figures 1 to 2 The method embodiment shown trains and updates the target video classification model.

[0123] like Figure 4The diagram illustrates the automatic updating of an online target video classification model. When a new video label is required, the video data corresponding to the new label is collected to obtain unlabeled video sample data. A small number of unlabeled video sample data are then selected and submitted to the labeling node for labeling. This returned labeled video sample data is added to the training dataset to update it. The updated training dataset is then used to train and update the target video classification model. This process is repeated until the model performance meets the requirements, at which point the model is deployed online, thus achieving automatic updating of the online video classification model. It should be noted that the difference between the selected unlabeled video sample data and the training dataset in each iteration is greater than the difference between the unselected unlabeled video sample data and the training dataset. For details on the selection process of unlabeled video sample data in each iteration, please refer to the embodiments of this invention. Figure 1 The relevant description of step 105 of the method embodiment shown will not be repeated here.

[0124] To gain a clearer understanding of the technical solutions of the embodiments of the present invention, the following is in conjunction with... Figure 5 To clarify, the application scenarios of the trained target video classification model can include video classification (which can be binary or multi-class video classification) and multi-label video classification.

[0125] Sample videos can be collected from PGC (Professionally-generated Content) and UGC (User-generated Content) published on video platforms. Specifically, they can be collected for each preset video category, which is the category that the video classification model is expected to recognize. For example, when you want to train a video classification model that recognizes game IP tags, you can collect video content of that game category from relevant video platforms.

[0126] Specifically, for video classification and multi-label video recognition, the corresponding category and label can be predicted by using multimodal information in the sample video. Specifically, three main modalities of data can be used as input to the model: video frames, audio, and text (including video title, video OCR, and video ASR). Therefore, after collecting a large number of sample videos, each sample video is preprocessed to extract the video frames, audio, and text content contained within, which are then used as the corresponding unlabeled video sample data, resulting in an unlabeled video sample dataset denoted as S.

[0127] Annotated video sample dataset L: A certain proportion is extracted from the obtained S and sent to the annotation nodes for annotation processing. For video classification tasks, the corresponding video categories can be labeled in the preset video categories (or target video categories), as shown in Table 1 below.

[0128] Table 1

[0129]

[0130] For multi-tag annotation of videos, a list of corresponding tags can be annotated, as shown in Table 2 below:

[0131] Table 2

[0132] Video ID Tag list 71mGaTngA1MUxsjUi Mahjong; casual games …… …… 7cRUjnh981N2UMqkA Beautiful women; handsome men; classical beauties

[0133] Remaining unlabeled video sample dataset U: After extracting a certain proportion from the unlabeled video sample dataset S to construct L, the remaining unlabeled video sample data is used as the large-scale remaining labeled video sample dataset U, which is used for subsequent candidate labeled video sample data.

[0134] like Figure 6 The example shown illustrates the training of a video classification model. The video classification model G employs a single-stream transformer structure, tokenizing the input of multimodal video information. It utilizes the transformer's self-attention mechanism to align and fuse multimodal features, predicting the video's category / label. For the text portion, the token extraction embedding method follows the BERT model (Bidirectional Encoder Representations from Transformers), adding position and part-of-speech encoding to the word's embedding. For video frames, similar to word embeddings, features extracted using a fast-trained ResNet101 residual network are used as visual token embeddings. Similarly, audio features are extracted using the VGGish model (which supports extracting semantically meaningful 128-dimensional embedding feature vectors from audio waveforms). This allows visual and audio features, like text words, to be input into the transformer as tokens, enabling feature fusion for prediction.

[0135] Since the embedding output by the [CLS] token is generally considered to be a feature fused from the entire video, it can be used for classification and multi-label prediction. A fully connected layer is then added after the [CLS] embedding to predict the probability distribution of the target class. For classification problems, the cross-entropy loss function can be used; for multi-label classification, it can be viewed as N binary classification problems, i.e., for N target labels, predicting whether each label exists. Training can be performed using the binary classification focal loss (dynamically scaled cross-entropy loss function) or CE loss (cross-entropy loss function).

[0136] After training a video classification model G on a labeled video sample dataset L, features are extracted from each sample in L and U using G. Here, the features refer to the embedding at the [CLS] position (i.e., the output of the fully connected layer before the classifier). Then, based on the extracted features, the best k (e.g., 10% of S) unlabeled video samples are selected from U and labeled to obtain L'. L' is then added to L. The process for selecting the best k unlabeled video samples can be found in the previous section. Figure 1 The relevant steps of the method embodiment shown are described, and the k unlabeled video sample data are removed from U (i.e., L' is removed from U as shown in the figure). The model G is retrained based on the updated L. After updating G, k unlabeled video sample data are selected from U (which is actually the updated one). This process is repeated until the model reaches the preset training termination condition.

[0137] To clearly demonstrate the effectiveness of the video classification model training method in this embodiment of the invention, the performance improvement curves of the video classification model were compared under different proportions of the labeled video sample dataset L in the overall unlabeled video sample dataset S. For video classification (which can be binary or multi-class classification), classification accuracy (ACC) was used as the metric. For multi-label classification, the F1 score was used to comprehensively measure the changes in accuracy and recall.

[0138] For video classification, as shown in Figure 7(a), the top dashed line (base_100% data) represents the accuracy obtained by the model when all data in S is labeled and trained; this can be understood as the performance limit of the model under the current settings. The curve of random represents the performance obtained by randomly selecting data in each round, and the middle curve represents the performance obtained using the method of this embodiment. It can be seen that this embodiment achieves near-perfect results using approximately 60% of the unlabeled video sample data, similar to training with all unlabeled video sample data.

[0139] For multi-label classification, as shown in Figure 7(b), it can be seen that the embodiments of the present invention can achieve results close to those trained using all unlabeled video sample data by sampling approximately 50% of the data. Specifically, Embodiment 1 of the present invention constructs an initial labeled video sample dataset L based on randomly selecting 10% of the unlabeled video sample data to obtain the target video multi-label classification model. Embodiment 2 of the present invention obtains the target video multi-label classification model based on an initial labeled dataset L covering all preset video categories (i.e., the target video category). Therefore, Embodiment 2 of the present invention demonstrates greater model performance and faster improvement than Embodiment 1.

[0140] In one exemplary embodiment, this invention also provides an application of the trained target video classification model. Specific application scenarios may include video classification (which could be binary or multi-class video classification) and multi-label video classification. Based on this, after training the target video classification model, the method may further include:

[0141] Get the video to be processed;

[0142] Determine the multimodal data corresponding to the video to be processed; the multimodal data includes video frames, audio data, and text data;

[0143] The multimodal data corresponding to the video to be processed is input into the target video classification model for video classification processing to obtain the video classification result; wherein, the target video classification model is trained based on the video classification model training method of the present invention.

[0144] In specific implementation, when the target video classification model is trained based on a binary classification task, the video classification result indicates whether the video to be processed belongs to a preset video category; when the target video classification model is trained based on a multi-classification task, the video classification result indicates which preset video category the video to be processed belongs to; when the target video classification model is trained based on a multi-label classification task, the video classification result indicates one or more preset video categories to which the video to be processed belongs.

[0145] When determining the multimodal data corresponding to the video to be processed, the video frames and audio data of the video can be extracted separately. OCR recognition is then performed on the video frames to extract text content from the images, and automatic speech recognition is performed on the audio data to convert speech into text. Finally, based on the text content extracted from the images and the speech-converted text, the text data in the multimodal data is obtained. Understandably, the text data may also include descriptive text corresponding to the video to be processed (such as the video title).

[0146] When performing video classification, the target video classification model determines the feature vector for each modality in the input multimodal data. The feature vector for each modality is composed of its token embedding. Specifically, the token embedding for text data consists of the word's embedding, word position encoding, and part-of-speech encoding. The token embedding for video frames can use features extracted by a fast-trained ResNet101 residual network, while the token embedding for audio data can use features extracted by the VGGish model.

[0147] After obtaining the feature vectors of each modality, the target video classification model can process the feature vectors of each modality based on a self-attention mechanism to obtain self-attention vectors for each modality. These self-attention vectors are then fused to obtain a fused multimodal feature vector, which is used to perform video classification to obtain the final classification result. Because multiple modal information from the video to be processed is combined during video classification, the accuracy of video classification can be improved.

[0148] Corresponding to the video classification model training methods provided in the above embodiments, this embodiment of the invention also provides a video classification model training device. Since the video classification model training device provided in this embodiment of the invention corresponds to the video classification model training methods provided in the above embodiments, the implementation methods of the aforementioned video classification model training methods are also applicable to the video classification model training device provided in this embodiment, and will not be described in detail in this embodiment.

[0149] Please see Figure 8 The diagram shows a structural schematic of a video classification model training device provided in an embodiment of the present invention. This device has the function of implementing the video classification model training method described in the above-described method embodiments. This function can be implemented by hardware or by hardware executing corresponding software. Figure 8 As shown, the video classification model training device 800 may include:

[0150] The current video classification model training module 810 is used to obtain the current labeled video sample dataset, and to train the current video classification model based on the current labeled video sample dataset. The labeled video sample data in the current labeled video sample dataset is obtained by labeling the unlabeled video sample data, and the unlabeled video sample data is unlabeled video sample data selected from the unlabeled video sample dataset.

[0151] The classification processing information acquisition module 820 is used to input the labeled video sample data and the remaining unlabeled video sample data into the current video classification model for classification processing, and to obtain the target classification processing information corresponding to the labeled video sample data and the remaining unlabeled video sample data respectively; the remaining unlabeled video sample data refers to the unlabeled video sample data in the unlabeled video sample dataset that was not selected as the video sample data to be labeled.

[0152] The target unlabeled video sample data selection module 830 is used to select a first preset number of target remaining unlabeled video sample data as new video sample data to be labeled, based on the target classification processing information corresponding to the labeled video sample data and the remaining unlabeled video sample data respectively; the degree of difference between the target remaining unlabeled video sample data and the current labeled video sample dataset is greater than the degree of difference between the unselected remaining unlabeled video sample data and the current labeled dataset;

[0153] The labeled video sample dataset update module 840 is used to update the current labeled video sample dataset based on the newly added unlabeled video sample data until the preset training end condition is met to end the training; wherein, the current classification model at the end of the training is used as the target video classification model.

[0154] In one exemplary embodiment, the target classification processing information is the feature obtained by feature extraction during the classification processing; the target unlabeled video sample data selection module 830 includes:

[0155] The first determining module is used to determine the current first feature set and the current second feature set; wherein, the first feature in the first feature set includes the extracted features corresponding to the labeled video sample data, and the second feature in the second feature set is the extracted features corresponding to the remaining unlabeled video sample data;

[0156] The first similarity determination module is used to determine the first similarity between each second feature in the current second feature set and the current first feature set;

[0157] The selection module is used to determine the target second feature corresponding to the minimum first similarity, and select the remaining unlabeled video sample data corresponding to the target second feature as the target remaining unlabeled video sample data.

[0158] The update module is used to update the current first feature set and the current second feature set based on the target second feature, until a first preset number of remaining unlabeled video sample data of the target are obtained; wherein, the updated current first feature set includes the target second feature and uses the target second feature as the first feature, and the updated current second feature set does not include the target second feature;

[0159] The second determining module is used to take the first preset number of the remaining unlabeled video sample data of the target as the newly added video sample data to be labeled.

[0160] In one exemplary implementation, the first similarity determination module includes:

[0161] The second similarity determination module is used to determine the second similarity between each second feature in the current second feature set and each first feature in the current first feature set for each second feature in the current second feature set;

[0162] The third determining module is used to take the maximum second similarity as the first similarity between the second feature and the first feature set.

[0163] In one exemplary embodiment, the current video classification model training module 810 includes:

[0164] An unlabeled video sample dataset acquisition module is used to acquire the unlabeled video sample dataset; the unlabeled video sample dataset includes unlabeled video sample data corresponding to at least one preset video category;

[0165] The initial unlabeled video sample data selection module is used to select unlabeled video sample data from the unlabeled video sample data corresponding to each preset video category as unlabeled video sample data to obtain an initial unlabeled video sample dataset; the total number of unlabeled video sample data in the initial unlabeled video sample dataset is less than the total number of unlabeled video sample data in the unlabeled video sample dataset;

[0166] The annotation processing module is used to annotate the unannotated video sample data in the initial unannotated video sample dataset to obtain the current annotated video sample dataset.

[0167] In one exemplary implementation, the unlabeled video sample dataset acquisition module includes:

[0168] A sample video set acquisition module is used to acquire a sample video set; the sample video set includes sample videos corresponding to the at least one preset video category;

[0169] A multimodal data determination module is used to determine the multimodal data corresponding to each sample video; the multimodal data includes video frames, audio data, and text data of the corresponding sample video;

[0170] The fourth determining module is used to take the multimodal data corresponding to each of the sample videos as unlabeled video sample data to obtain the unlabeled video sample dataset.

[0171] In one exemplary implementation, the classification task includes any one of binary classification, multi-class classification, and multi-label classification.

[0172] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0173] This invention provides an electronic device including a processor and a memory. The memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement any of the video classification model training methods provided in the above method embodiments.

[0174] Memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. Memory can primarily include a program storage area and a data storage area. The program storage area can store the operating system, application programs required for the functions, etc.; the data storage area can store data created based on the use of the device, etc. Furthermore, memory can include high-speed random access memory, and can also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory.

[0175] The method embodiments provided in this invention can be executed in a computer terminal, server or similar computing device, that is, the above-mentioned electronic device may include a computer terminal, server or similar computing device. Figure 9 This is a hardware structure block diagram of an electronic device for running a video classification model training method provided in an embodiment of the present invention, such as... Figure 9As shown, the server 900 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 910 (CPUs 910 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 930 for storing data, and one or more storage media 920 (e.g., one or more mass storage devices) for storing application programs 923 or data 922. The memory 930 and storage media 920 may be temporary or persistent storage. The program stored in the storage media 920 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 910 may be configured to communicate with the storage media 920 and execute the series of instruction operations stored in the storage media 920 on the server 900. Server 900 may also include one or more power supplies 960, one or more wired or wireless network interfaces 950, one or more input / output interfaces 940, and / or one or more operating systems 921, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0176] The input / output interface 940 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 900. In one example, the input / output interface 940 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 940 may be a radio frequency (RF) module for wireless communication with the Internet.

[0177] Those skilled in the art will understand that Figure 9 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 900 may also include... Figure 9 The more or fewer components shown, or having the same Figure 9 The different configurations shown.

[0178] Embodiments of the present invention also provide a computer-readable storage medium, which can be disposed in an electronic device to store at least one instruction or at least one program related to implementing a classification model training method. The at least one instruction or the at least one program is loaded and executed by the processor to implement any of the video classification model training methods provided in the above-described method embodiments.

[0179] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0180] It should be noted that the order of the above embodiments of the present invention is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, specific embodiments have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired result. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0181] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the apparatus embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0182] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0183] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A video classification model training method, characterized in that, The method includes: Obtain the current labeled video sample dataset, and train the current video classification model based on the current labeled video sample dataset; the labeled video sample data in the current labeled video sample dataset is obtained by labeling the video sample data to be labeled, and the video sample data to be labeled is the unlabeled video sample data selected from the unlabeled video sample dataset; The labeled video sample data and the remaining unlabeled video sample data are respectively input into the current video classification model for classification processing to obtain the target classification processing information corresponding to the labeled video sample data and the remaining unlabeled video sample data respectively; the remaining unlabeled video sample data refers to the unlabeled video sample data in the unlabeled video sample dataset that was not selected as the video sample data to be labeled. Based on the target classification processing information corresponding to the labeled video sample data and the remaining unlabeled video sample data, a first preset number of target remaining unlabeled video sample data are selected from the remaining unlabeled video sample data as newly added video sample data to be labeled; the degree of difference between the target remaining unlabeled video sample data and the current labeled video sample dataset is greater than the degree of difference between the unselected remaining unlabeled video sample data and the current labeled video sample dataset; The current labeled video sample dataset is updated based on the newly added unlabeled video sample data until the preset training termination condition is met and the training ends; wherein, the current video classification model at the end of training is used as the target video classification model.

2. The method according to claim 1, characterized in that, The target classification processing information refers to the features extracted during the classification processing; the step of selecting a first preset number of target remaining unlabeled video sample data as new video sample data to be labeled from the remaining unlabeled video sample data based on the target classification processing information corresponding to the labeled video sample data and the remaining unlabeled video sample data includes: Determine the current first feature set and the current second feature set; wherein, the first feature in the first feature set includes the extracted features corresponding to the labeled video sample data, and the second feature in the second feature set is the extracted features corresponding to the remaining unlabeled video sample data; Determine the first similarity between each second feature in the current second feature set and the current first feature set; Determine the target second feature corresponding to the minimum first similarity, and select the remaining unlabeled video sample data corresponding to the target second feature as the target remaining unlabeled video sample data; The current first feature set and the current second feature set are updated based on the target second feature until a first preset number of remaining unlabeled video sample data of the target are obtained; wherein, the updated current first feature set includes the target second feature and uses the target second feature as the first feature, and the updated current second feature set does not include the target second feature; The first preset number of remaining unlabeled video sample data of the target are used as newly added video sample data to be labeled.

3. The method according to claim 2, characterized in that, Determining the first similarity between each second feature in the current second feature set and the current first feature set includes: For each second feature in the current second feature set, determine a second similarity between the second feature and each first feature in the current first feature set; The maximum second similarity is taken as the first similarity between the second feature and the first feature set.

4. The method according to claim 1, characterized in that, The process of obtaining the current labeled video sample dataset includes: Obtain the unlabeled video sample dataset; the unlabeled video sample dataset includes unlabeled video sample data corresponding to at least one preset video category; Unlabeled video sample data is selected from the unlabeled video sample data corresponding to each preset video category as video sample data to be labeled, to obtain an initial unlabeled video sample dataset; the total number of unlabeled video sample data in the initial unlabeled video sample dataset is less than the total number of unlabeled video sample data in the unlabeled video sample dataset; The unannotated video sample data in the initial unannotated video sample dataset is annotated to obtain the current annotated video sample dataset.

5. The method according to claim 4, characterized in that, The process of obtaining the unlabeled video sample dataset includes: Obtain a sample video set; the sample video set includes sample videos corresponding to at least one preset video category; Determine the multimodal data corresponding to each of the sample videos; the multimodal data includes the video frames, audio data, and text data of the corresponding sample video; The multimodal data corresponding to each of the sample videos is used as unlabeled video sample data to obtain the unlabeled video sample dataset.

6. The method according to any one of claims 1 to 5, characterized in that, The classification task includes any one of the following: binary classification task, multi-class classification task, and multi-label classification task.

7. A video classification model training device, characterized in that, The device includes: The current video classification model training module is used to obtain the current labeled video sample dataset, and to train the current video classification model based on the current labeled video sample dataset. The labeled video sample data in the current labeled video sample dataset is obtained by labeling the unlabeled video sample data, and the unlabeled video sample data is unlabeled video sample data selected from the unlabeled video sample dataset. The classification processing information acquisition module is used to input the labeled video sample data and the remaining unlabeled video sample data into the current video classification model for classification processing, and to obtain the target classification processing information corresponding to the labeled video sample data and the remaining unlabeled video sample data respectively; the remaining unlabeled video sample data refers to the unlabeled video sample data in the unlabeled video sample dataset that was not selected as the video sample data to be labeled. The target unlabeled video sample data selection module is used to select a first preset number of target unlabeled video sample data as new video sample data to be labeled based on the target classification processing information corresponding to the labeled video sample data and the remaining unlabeled video sample data, respectively; the degree of difference between the target unlabeled video sample data and the current labeled video sample dataset is greater than the degree of difference between the unselected remaining unlabeled video sample data and the current labeled video sample dataset; The labeled video sample dataset update module is used to update the current labeled video sample dataset based on the newly added unlabeled video sample data until the preset training termination condition is met to end the training; wherein, the current video classification model at the end of the training is used as the target video classification model.

8. An electronic device, characterized in that, The system includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the video classification model training method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction or at least one program, which is loaded and executed by a processor to implement the video classification model training method as described in any one of claims 1 to 6.

10. A computer program, characterized in that, The system includes a computer program, characterized in that, when the computer program is executed by a processor, it implements the video classification model training method according to any one of claims 1 to 6.

Images