Video category determination method and device

By constructing a video keyframe search library and small sample classification method, the problem of inaccurate video classification results in the existing technology is solved, and the accurate detection of advanced semantic features of video is achieved, which improves the effectiveness of video content security detection.

CN119068375APending Publication Date: 2024-12-03NAT COMP NETWORK & INFORMATION SECURITY MANAGEMENT CENT
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310617738.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-30
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

Existing video classification methods cannot accurately detect factors that have high-level semantic characteristics in videos, such as 'whether there is a reversal of right and wrong', 'whether there is a value-oriented problem', and there are data set annotation scale restrictions and models Overfitting problem.

Method used

By obtaining multiple video data sets as training samples, keyframes are extracted using pre-trained networks, and keyframe search library is built based on the Faiss method. Combining the small sample classification method and the K nearest neighbor algorithm, a classification model is built to improve the accuracy of video classification.

Benefits of technology

It realizes a more accurate classification of video dimension information, solves the problem of inaccurate video classification results, and improves the accuracy of video content security detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068375A_ABST
    Figure CN119068375A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a video category determination method and device, and the method comprises the steps: obtaining a plurality of video data sets as training samples, and enabling the training samples to comprise a support set and a test set; respectively extracting key frames of the support set and the test set through the pre-training network, and constructing a key frame retrieval library through a Faiiss method based on the support set; performing classification training based on the key frames of the test set and the key frame retrieval library through a preset small sample classification method to obtain a classification model; and obtaining a to-be-classified video, and classifying the to-be-classified video through the classification model to obtain a classification result. Through the video dimension information classification method and device, the problem that the classification result of certain dimension information of the video is not accurate enough is solved, and the effect of more accurate video dimension information classification is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of video classification, and in particular, to a method and device for determining a video category. Background Art

[0002] For video platforms, video content security is a basic requirement. Existing video content security detection technology is based on traditional video classification methods. The main technical route is to classify key information in a certain mode of the video, such as the background or character behavior in the video content. Limited by its data scale and modeling method, this type of technology cannot detect factors directly related to a certain field (such as economy), or more fine-grained and higher-level semantic features, such as "whether the video is a reversal of right and wrong", "whether it conforms to the background of the times" and "whether there are value orientation problems". In addition, there are problems such as the difficulty of collecting political data sets and the easy overfitting of models. Such videos have important functions and guiding significance in actual media propaganda security detection.

[0003] The most commonly used video classification method currently is to design a deep learning model (such as [1] SlowFast, [2] VideoSwin, [3] VideoMAE, etc.) and perform single-label or multi-label training on a large-scale manually annotated video dataset. For the video to be classified, a specific number of key frames are first extracted as the input of the trained classification model, and the classification results output by the model are used as its labels.

[0004] Currently, the method of training a video classification deep learning model with a large amount of manually annotated video data can complete basic video classification tasks, but the obtained category labels must be directly related to a specific information in the video, and it is impossible to infer labels with more characteristics of certain factors through the high-level semantics of the video content. In addition, the classification categories of these works are completely independent and lack internal connections, and the total number of categories is limited by the annotation scale of the data set.

[0005] It can be seen from this that there is a problem in the related technology that the classification results of a certain dimension of video information are not accurate enough.

[0006] Currently, no effective solution has been proposed for the above-mentioned problems existing in the related technologies. Summary of the invention

[0007] The embodiments of the present invention provide a method and device for determining a video category, so as to at least solve the problem in the related art that the classification result of a certain dimension of video information is not accurate enough.

[0008] According to one embodiment of the present invention, a method for determining a video category is provided, comprising: obtaining a plurality of video data sets as training samples, wherein the training samples include a support set and a test set; extracting key frames of the support set and the test set respectively through a pre-trained network, and constructing a key frame retrieval library based on the support set through the Faiss method; performing classification training based on the key frames of the test set and the key frame retrieval library through a pre-set small sample classification method to obtain a classification model; obtaining a video to be classified, and classifying the video to be classified through the classification model to obtain a classification result.

[0009] Furthermore, the key frames of the support set and the test set are extracted respectively through a pre-trained network, and a key frame retrieval library is constructed based on the support set through the Faiss method, including: extracting the key frames of the support set and the test set; using a large-scale pre-trained network CLIP to extract features of the key frames in the support set and the test set respectively to obtain key frame features, and normalizing the key frame features to obtain normalized features; and constructing a retrieval library based on the support set through the open source library Faiss, wherein the retrieval library contains the number of key frames and feature vectors of specified dimensions.

[0010] Furthermore, classification training is performed based on the key frames of the test set and the key frame retrieval library through a preset small sample classification method to obtain a classification model including: determining the key frame with the most political categories in the K retrieval results through the K nearest neighbor algorithm as the first prediction result of the category of the target sample; calculating the inter-frame distance between a single key frame in the test video and the supporting sample in the key frame retrieval library of the support set, completing retrieval iterations for the key frames of each video in the test set, and summing the inter-frame distance parameters as the average inter-frame distance between the test video and the supporting video, and taking the supporting video with the smallest average inter-frame distance of the test video as the second prediction result; calculating the average of the average inter-frame distances between the test video and all supporting videos in each small sample category, and taking the category with the smallest average inter-frame distance as the third prediction result of the test video; and correcting the classification model according to the first prediction result, the second prediction result and the third prediction result to obtain the target classification model.

[0011] Furthermore, after obtaining the classification model, the method also includes: preprocessing and labeling the training samples; constructing a classification network, using the preprocessed pretrained network as the backbone, using the linear layer as the classification head, randomly selecting N consecutive frames from the video key frames as network input, and obtaining the video prediction category; and revising the target classification model according to the video prediction category.

[0012] Furthermore, after constructing a retrieval library based on the support set through the open source library Faiss, the method further includes: calculating the similarity index between samples through the following formula: Among them, di is the distance between frames, x and xi are the samples to be retrieved and any key frames in the support set, respectively.

[0013] Furthermore, the designated dimension is a political dimension.

[0014] According to another embodiment of the present invention, a video category determination device is provided, comprising: a first acquisition unit, used to acquire multiple video data sets as training samples, wherein the training samples include a support set and a test set; an extraction unit, used to extract key frames of the support set and the test set respectively through a pre-trained network, and construct a key frame retrieval library based on the support set through the Faiss method; a training unit, used to perform classification training based on the key frames of the test set and the key frame retrieval library through a preset small sample classification method to obtain a classification model; a second acquisition unit, used to acquire a video to be classified, classify the video to be classified through the classification model, and obtain a classification result.

[0015] Furthermore, the extraction unit includes: an extraction module, used to extract key frames of the support set and the test set; a processing module, used to perform feature extraction on the key frames in the support set and the test set using a large-scale pre-trained network CLIP to obtain key frame features, and normalize the key frame features to obtain normalized features; a construction module, used to construct a retrieval library based on the support set through the open source library Faiss, wherein the retrieval library contains the number of key frames and feature vectors of specified dimensions.

[0016] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0017] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0018] Through the present invention, multiple video data sets are obtained as training samples, wherein the training samples include a support set and a test set; the key frames of the support set and the test set are respectively extracted through a pre-trained network, and a key frame retrieval library is constructed based on the support set through the Faiss method; classification training is performed based on the key frames of the test set and the key frame retrieval library through a pre-set small sample classification method to obtain a classification model; a video to be classified is obtained, and the video to be classified is classified through the classification model to obtain a classification result. Therefore, the problem that the classification result of a certain dimensional information of the video is not accurate enough in the related art can be solved, and a more accurate effect of classifying the video dimensional information can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a hardware structure block diagram of a mobile terminal of a method for determining a video category according to an embodiment of the present invention;

[0020] Figure 2 is a flowchart of a method for determining a video category according to an embodiment of the present invention;

[0021] Figure 3 is a schematic diagram of a political video classification method based on large-scale pre-training in this embodiment;

[0022] Figure 4 is a schematic diagram of a political video classification method based on large-scale pre-training in this embodiment;

[0023] Figure 5 is a schematic diagram of the small sample classification result of this embodiment;

[0024] Figure 6 is a schematic diagram of the supervised fine-tuning result of this embodiment;

[0025] Figure 7 is a structural block diagram of a device for determining a video category according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0028] The meanings of the terms in this embodiment are as follows:

[0029] 1. dim: key frame feature dimension.

[0030] 2. Topk samples: the k samples with the most similar retrieval results.

[0031] 3. Retrieval samples: pre-trained features of key frames in the test set.

[0032] 4. Retrieval results: pre-trained features of key frames in the support set.

[0033] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a mobile terminal of a method for determining a video category according to an embodiment of the present invention. Figure 1 As shown, the mobile terminal may include one or more ( Figure 1 Only one is shown in the figure) a processor 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data, wherein the mobile terminal may also include a transmission device 106 and an input / output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0034] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the video category determination method in the embodiment of the present invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0035] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0036] In this embodiment, a method for determining a video category is provided. Figure 2 is a flow chart of a method for determining a video category according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0037] Step S101, obtaining multiple video data sets as training samples, wherein the training samples include a support set and a test set;

[0038] Step S102, extracting key frames of the support set and the test set respectively through the pre-trained network, and constructing a key frame retrieval library based on the support set through the Faiss method;

[0039] Step S103, performing classification training based on the key frames of the test set and the key frame retrieval library by a preset small sample classification method to obtain a classification model;

[0040] Step S104, obtaining a video to be classified, and classifying the video to be classified using a classification model to obtain a classification result.

[0041] In the above embodiment, the training samples composed of multiple video data sets can be video data sets that have been manually labeled, wherein the training samples are divided into two parts, one part is a support set and a test set, and the key frame features in the two sets are extracted through the pre-trained network CLIP, and a key frame retrieval library is constructed based on the key frame features in the support set through the Faiss method, and the key frames of the test set and the key frame retrieval library are classified and trained through one or more pre-set small sample classification methods to obtain a classification model. The classification model can learn the features and labels of each video to learn the classification rules. When there are videos to be classified later, the classification model is used to classify the videos to obtain results. Such a video classification method can solve the problem that the classification results of a certain dimension of video information in the related technology are not accurate enough, and achieve a more accurate effect of classifying video dimension information.

[0042] In this embodiment, after the video data is classified, corresponding operations can be performed as needed, such as labeling the video, or pushing or limiting the flow of the video, etc. The specific application can be adjusted according to the needs of the application scenario. The technical solution of this embodiment can be applied to the classification of multiple categories.

[0043] Optionally, key frames of the support set and the test set are extracted respectively through a pre-trained network, and a key frame retrieval library is constructed based on the support set through the Faiss method, including: extracting key frames of the support set and the test set; using a large-scale pre-trained network CLIP to extract features of the key frames in the support set and the test set respectively to obtain key frame features, and normalizing the key frame features to obtain normalized features; and constructing a retrieval library based on the support set through the open source library Faiss, wherein the retrieval library contains the number of key frames and feature vectors of specified dimensions.

[0044] In this embodiment, the specified dimension can be a political dimension or a dimension of other categories, such as economy, beauty, etc. The key frames in the support set and the test set are respectively extracted using a large-scale pre-trained network CLIP to obtain key frame features, and then the key frame features are normalized to obtain normalized features; a retrieval library is constructed based on the support set through the open source library Faiss, wherein the retrieval library contains the number of key frames and feature vectors of the specified dimension. By extracting features, normalizing, and constructing a retrieval library, the data during model training can be made more accurate.

[0045] Optionally, classification training is performed based on the key frames and key frame retrieval library of the test set through a pre-set small sample classification method to obtain a classification model including: determining the key frame with the most specific categories in the K retrieval results through the K nearest neighbor algorithm as the first prediction result of the category of the target sample; calculating the inter-frame distance between a single key frame in the test video and the supporting sample in the key frame retrieval library of the support set, completing the retrieval iteration for the key frames of each video in the test set, and summing the inter-frame distance parameters as the average inter-frame distance between the test video and the supporting video, and taking the supporting video with the smallest average inter-frame distance with the test video as the second prediction result; calculating the average of the average inter-frame distances between the test video and all supporting videos in each small sample category, and taking the category with the smallest average inter-frame distance as the third prediction result of the test video; and correcting the classification model according to the first prediction result, the second prediction result and the third prediction result to obtain the target classification model.

[0046] Classification prediction is performed through three pre-set small sample classification methods to obtain three classification results. The three classification results are used as labels of the classification results. The initially trained model is corrected according to these three results to make the model more accurate.

[0047] The average inter-frame distance means: assuming that the first video has 20 frames in total, and the second video has 50 frames in total, of which there are 5 key frames, the similarity between each of the 20 frames of the first video and the 5 key frames of the second video is calculated one by one, and the two frames with the highest similarity are taken, and the frame spacing is calculated. The average of these 100 frame spacings is the average inter-frame distance. The average inter-frame distance is used to describe the distance between two videos.

[0048] Optionally, after obtaining the classification model, the training samples are preprocessed and labeled; a classification network is constructed, the pre-trained network after preprocessing is used as the backbone, a linear layer is used as the classification head, and N consecutive frames are randomly selected from the video key frames as the network input to obtain the video prediction category; the target classification model is revised again according to the video prediction category.

[0049] Building a classification network can be based on the optimization of some existing networks, or directly building a new network, and revising the target classification model according to the predicted category and the label of the video itself to make the model more accurate.

[0050] Optionally, after building a retrieval library based on the support set using the open source library Faiss, the similarity index between samples is calculated using the following formula: Among them, di is the distance between frames, x and xi are the samples to be retrieved and any key frames in the support set, respectively.

[0051] When calculating the similarity index between samples, using the above formula can make the distance calculation more accurate and further improve the accuracy of the model.

[0052] The present invention also provides a specific implementation manner.

[0053] This embodiment introduces a large-scale pre-trained feature network and combines it with a small sample learning paradigm to achieve better security detection of political video content. In this process, the K-nearest neighbor algorithm, video retrieval, cluster center, and supervised fine-tuning are used to more comprehensively measure the effect of video content security detection.

[0054] Three methods based on large-scale pre-trained network feature extraction and supervised fine-tuning were used as controls: (1) K nearest neighbor algorithm, that is, frame-level features were extracted through large-scale pre-trained models, and a video key frame retrieval library was established using the Faiss open source method. Topk similarity retrieval was performed on the video key frames in the test set, and video classification results were obtained based on the key frame similarity retrieval information. (2) Video retrieval, based on the video key frame retrieval library established in (1), the video-level similarity distance was quantified, and the test video was classified according to the high-level semantic information at the video level. (3) Clustering center, the small sample political video data in the support set was clustered into multiple "meta" types, and the test video was clustered and analyzed using the "meta" type network clustering method. (4) Supervised fine-tuning, using supervised label information to fine-tune on the basis of the large-scale pre-trained model, and the fine-tuned model was used to predict the category to which the small sample data belongs based on the high-level semantic information of the video.

[0055] Figure 3 and Figure 4 It is a schematic diagram of the political video classification method based on large-scale pre-training of this embodiment. This embodiment mainly includes three key steps: the first step is to divide the collected video data set into a support set and a test set according to the category, and then extract the key frame of the original video, and use the open source library Faiss to establish a support set retrieval library for all the key frames in the support set. For each key frame of the video in the test set, the most similar topk samples are calculated in the support set retrieval library to obtain the category information of the most similar sample of the test topk and the distance with the key frame. The second step is to use the K nearest neighbor algorithm to retrieve all the key frames of each video in the test set, and determine the number of subdivided categories with the largest number of occurrences in the topk retrieval results corresponding to all the key frames of the entire video as the test video classification result. On this basis, the method based on video retrieval makes full use of the distance information between the key frame of the test video and the topk retrieval results. The distance between the test key frame and the topk retrieval results is smaller, and it is believed that the distance between the test video and the video to which the retrieval result belongs should also be smaller. Therefore, by calculating all the key frames in the video, the support set video with the smallest distance from the test video can be retrieved, and the category to which the most similar support set video belongs is used as the test video classification result. The cluster center method cleverly borrows the "meta" method to calculate the "meta" center of all support set videos of each small sample category. Based on video retrieval, the distance between the test video and the entire small sample "meta" center is calculated as the classification standard. Similarly, the category with the closest distance is used as the classification result. In the third step, in order to compare with the traditional supervised video classification method, this embodiment also completes the supervised experiment, that is, using the category of the support set as the supervision signal, fine-tuning the CLIP model on the small sample video dataset related to politics to obtain the final classification result.

[0056] Feature extraction and retrieval library construction

[0057] Step 1: Feature extraction and retrieval library construction. The detailed process is as follows:

[0058] Step 1.1: The dataset proposed in the present invention contains 910 uncut videos, with video durations ranging from 5 seconds to 10 minutes. The dataset contains 9 major categories, each of which contains several subcategories, totaling 42 subcategories. According to the dataset division principle, 70% of the videos are randomly selected from each subcategories as the support set each time, and the remaining 30% of the videos are used as the test set, where both the support set and the test set contain all 42 subcategories.

[0059] Step 1.2: Since there is a lot of redundant information between video frames, in order to improve the retrieval speed and representativeness, one frame per second is taken as the key frame of the video for all videos, and the large-scale pre-trained network CLIP is used to extract features for the video key frames in the support set and the test set, and then the features are normalized to meet the needs of later calculation of similarity distance. The CLIP pre-trained network is a classic large-scale pre-trained network proposed by openAI in 2021. The model is pre-trained using the contrastive learning method on a 400 million-scale dataset, and has achieved the best results in zero-shot on a variety of downstream tasks, meeting the needs of the large-scale pre-trained network of the present invention.

[0060] Step 1.3: For the support set, a retrieval library is constructed based on the open source library Faiss. The specific form of the retrieval library is a binary file of the form (nframes, dim), where nframes and dim represent the total number of key frames and the feature vector of the specified dimension respectively. The similarity index between samples calculated by the retrieval library is the classic L2 norm, and the calculation formula is:

[0061]

[0062] where x, x i are the samples to be retrieved and any key frames in the support set, respectively, i Represents the distance between the two.

[0063] We use all key frame features in the test set as samples to search in the support set retrieval library, and return the index of the K most similar results in the support set to the test sample and the distance to the retrieval sample, where K is an optional hyperparameter. Based on the obtained retrieval results, the classification of the test set is further analyzed.

[0064] Step 2: Small sample classification method, the detailed process is as follows:

[0065] Step 2.1: K-nearest neighbor algorithm

[0066] K nearest neighbor algorithm is a commonly used non-parametric statistical method for classification and regression in the field of pattern recognition. In the present invention, the topk search results of the search sample are the K nearest neighbors. Without considering the order of the K search results, they are given the same weight. The original specific category of each search result is used as the predicted category of the search sample. The category with the most identical specific categories in the K search results is considered to be the final predicted category of the search sample. Then, the categories of all key frames in each test video are counted, and the key frames with the most identical categories are also used as the category prediction results of the entire test video. Finally, the test video category prediction results are compared with the true category labels, and the category prediction accuracy of the video level is used as the measurement index of the K nearest neighbor algorithm.

[0067] Step 2.2: Video Retrieval

[0068] Based on the key frames obtained in step 2.1, the inter-frame distance information is introduced. According to the distance information returned from the Faiss retrieval, it is considered that the inter-frame distance between the retrieval sample and the topK retrieval result is less than 1. The higher the similarity between the two, the smaller the distance. On the contrary, the retrieval sample is completely different from other samples in the support set retrieval library, and the inter-frame distance is 1.

[0069] In this way, in one retrieval iteration, the distance between a single key frame in the test video and all supporting samples in the support set retrieval library is calculated, and the minimum distance between frames is taken as the distance between the test video and the supporting video. Based on this method, all key frames of each video in the test set are retrieved iteratively, and the distance information is summed as the distance between the test video and the supporting video. The supporting video with the smallest distance to the test video is considered to be the video retrieval result, and its category is determined to be the category of the test video.

[0070] In this setting, the similarity information of all key frames in the video is fused through the inter-frame distance information, and reasonable weights are given to the key frames, which can better mine the internal category information of the video.

[0071] Step 2.3: Cluster Centers

[0072] Based on step 2.2, the average distance between the test video and all supporting videos in each small sample category is calculated as the distance to the category, and the category with the smallest distance among all categories is taken as the predicted category of the test video.

[0073] In this setting, the center of each small sample category is a dynamic center, which effectively solves the problem that the category center in the traditional clustering method is fixed and cannot be effectively changed with the change of the test sample. In the political video data set collected by the present invention, all videos belong to the small sample category, and there is no irrelevant forced clustering, so the test video can be predicted into the corresponding category.

[0074] In the actual system, since the distance information is used as the measurement indicator in this setting, as long as all the supporting videos in the small sample category are selected reasonably, the distance from all small sample categories can be accurately calculated for any test video. If the test video is irrelevant to all small sample categories, the distance from all categories is large, and it can be considered that the test video does not belong to any small sample category, and there will be no irrelevant forced clustering.

[0075] Step 3: Supervised fine-tuning. The detailed process is as follows:

[0076] Step 3.1:

[0077] Label the small sample category videos and map the actual video categories to the classification model. Select the support set and test set divided in step 1.2 and preprocess them to form a single video key frame raw data as model training data to ensure fair comparison with the small sample classification method.

[0078] Step 3.2:

[0079] Based on the political video classification dataset in step 3.1, design the model structure and parameters based on the traditional video classification method.

[0080] At the data input and output level, during model training, N consecutive key frames are randomly selected for each support set video as model input, the N feature vectors output by the model are averaged as the global vector representation of the video, and the predicted category is obtained. The model parameters are updated by calculating the loss value with the real category label. N is a manually set hyperparameter. If the length of the video key frame is less than N, the key frame of the original video is copied at the end to complete it. During model inference, all key frames of the test set video are used as model input, and the average of all key frame feature vectors is taken as the global variable of the test video, which is used to predict the real category of the video.

[0081] In terms of training methods, the present invention chooses to fine-tune the entire network using labeled data instead of only training the classification head. This is because the end-to-end fine-tuning method can make the model better fit the political video classification data set and achieve higher classification accuracy.

[0082] Step 3.3:

[0083] In terms of model architecture and training parameter selection, the present invention keeps the selection of CLIP as the backbone network unchanged, and adds a linear layer as the classification head at the end of the model. The dimension of the linear layer is (dim, 42) corresponding to the 42 small sample categories in the political video data. The learning rate is 5*10 -5 , the number of iterations is set to 30, and the cross entropy function is selected as the loss function:

[0084]

[0085] Since the video length N determines the amount of training data for the model, the value of N has an important impact on whether the model can fit and the final classification accuracy. In the experiment, the changes in experimental results will be compared by adjusting the value of N.

[0086] Beneficial effects of the technical solution

[0087] 4.1 Small Sample Classification

[0088] By using the large-scale pre-training + small sample paradigm political video classification method proposed in the present invention, a high video classification accuracy can be obtained without almost any computational effort. The experiment was conducted on the political video classification dataset proposed in the present invention. Specifically, 650 videos were selected as the support set and 260 videos were selected as the test set. In order to eliminate contingency, N-fold cross validation was used on the dataset to ensure that the videos contained in the support set and the test set used in each experiment were not exactly the same. Finally, the average of the video classification accuracy of N verifications was taken as the return result.

[0089] On this basis, we first verified the classification accuracy of different measurement indicators in the small sample paradigm. Here, the hyperparameter K was selected as 5, 10, 15, and 20 respectively. The experimental results are as follows: Figure 5 shown.

[0090] from Figure 5 It can be seen that the method based on large-scale pre-training + small sample paradigm, using video retrieval, cluster center and K nearest neighbor algorithm on the political video dataset without any training, has achieved a recognition accuracy rate of more than 75%, which can preliminarily meet the needs of the political video classification system.

[0091] The experimental results also show that the hyperparameter K, that is, the number of most similar samples, has little effect on the classification results, and the overall range fluctuates within 2%. This is because for any subdivided category, there are often a large number of samples of this category in the support set retrieval library (there are few support set videos, but many key frames of the support set videos). When K increases within a reasonable range, the returned retrieval results are basically the key frames of the videos in this category, so the model can still predict the correct test video category. On the contrary, if the K value is set too small or too large, it will lead to misjudgment of categories due to accidental errors such as inherent noise and bias in the video, and forced to find the most similar results in other categories because the category has been fully retrieved.

[0092] 4.2 Supervised Fine-tuning

[0093] The second step of the present invention is to conduct experiments under the supervised fine-tuning setting. According to the relevant parameters given in step 3, the political video classification network, in which the total model parameter quantity is 28M, and the training video frame length N is selected as 32, 64, 96, 128, 160, and 192 respectively, observe the values ​​of classification accuracy acc1 and acc5 under different length settings. The experimental results are as follows Figure 6 shown.

[0094] from Figure 6 It can be seen that when the video frame length N is small, the classification accuracy acc1 and acc5 of the model are very low. As N increases, the video classification accuracy improves significantly, and when N=128, it is basically the same as the small sample classification method.

[0095] It was found during the training process that when the N value is small, the loss function value and the video classification accuracy fluctuate greatly, and the model has serious overfitting. As the N value increases, the overfitting phenomenon gradually weakens and the model can achieve better fitting effects. When the training video length is 160, the supervised fine-tuning method can significantly exceed the small sample classification method and achieve higher classification accuracy. However, there is a serious problem at this time, that is, about 60% of the videos in the support set do not meet the key frame length greater than or equal to 160. Although this setting can improve the classification accuracy of political videos, it is obviously unreasonable in practical applications. In a complex video environment, there is no guarantee that the length of any political video is greater than a specific length. Often, very short videos contain rich semantic information.

[0096] 4.3 Discussion of Experimental Results

[0097] By comparing the experimental results of small sample classification and supervised fine-tuning, it can be concluded that although the application of traditional video classification methods to political video classification can achieve higher classification accuracy than small sample classification, there is a serious overfitting phenomenon, and it is very sensitive to the length of the data set video, and the model scalability is poor. If the data volume is "forcibly" increased, the length of many video key frames will not meet the requirements, and in the actual system, it is impossible to effectively process very short political videos with rich semantic information. In addition, the classification backbone network CLIP used in the supervised fine-tuning of the present invention is based on the VIT Small model, and the total parameter volume is only 28M. If we continue to explore networks with smaller parameter volumes instead of using networks that have been proven to be effective in academia and industry, it means more manpower and material resources will be consumed.

[0098] On the contrary, the classification method based on large-scale pre-training and small samples proposed in the present invention can achieve an accuracy of 75%+ in the classification of political video data sets without any model training and computing resources, and preliminarily meets the actual system requirements. And based on the cluster center method proposed in step 2.3, it has strong scalability and dynamic clustering characteristics. If you want to introduce new categories, you only need to collect relevant support set and test set video samples without considering the design of related modules such as network and hyperparameters. It effectively solves the problems of model overfitting and complex engineering, and has good application prospects.

[0099] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0100] In this embodiment, a video category determination device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0101] Figure 7 is a structural block diagram of a video category determination device according to an embodiment of the present invention. Figure 7 As shown, the device comprises:

[0102] A first acquisition unit 10 is used to acquire multiple video data sets as training samples, wherein the training samples include a support set and a test set;

[0103] An extraction unit 20 is used to extract key frames of a support set and a test set respectively through a pre-trained network, and to construct a key frame retrieval library based on the support set through a Faiss method;

[0104] A training unit 30 is used to perform classification training based on the key frames of the test set and the key frame retrieval library by a preset small sample classification method to obtain a classification model;

[0105] The second acquisition unit 40 is used to acquire the video to be classified, and classify the video to be classified by using the classification model to obtain a classification result.

[0106] The technical solution of this embodiment is through a first acquisition unit 10, which is used to acquire multiple video data sets as training samples, wherein the training samples include a support set and a test set; an extraction unit 20, which is used to extract key frames of the support set and the test set respectively through a pre-trained network, and build a key frame retrieval library based on the support set through the Faiss method; a training unit 30, which is used to perform classification training based on the key frames of the test set and the key frame retrieval library through a preset small sample classification method to obtain a classification model; a second acquisition unit 40, which is used to acquire a video to be classified, classify the video to be classified through the classification model, and obtain a classification result, which can solve the problem of inaccurate classification results of a certain dimension information of the video existing in the related art, and achieve a more accurate effect of classifying the video dimension information.

[0107] In an exemplary embodiment, the extraction unit 20 includes: an extraction module for extracting key frames from a support set and a test set; a processing module for performing feature extraction on the key frames in the support set and the test set using a large-scale pre-trained network CLIP to obtain key frame features, and normalizing the key frame features to obtain normalized features; and a construction module for constructing a retrieval library based on the support set through the open source library Faiss, wherein the retrieval library contains the number of key frames and feature vectors of specified dimensions.

[0108] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0109] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0110] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0111] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0112] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0113] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0114] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0115] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for determining a video category, characterized in that: include: Acquire multiple video data sets as training samples, wherein the training samples include a support set and a test set; Extracting key frames of the support set and the test set respectively through a pre-trained network, and constructing a key frame retrieval library based on the support set through the Faiss method; Performing classification training based on the key frames of the test set and the key frame retrieval library by a preset small sample classification method to obtain a classification model; Obtain a video to be classified, and classify the video to be classified using the classification model to obtain a classification result.

2. The method according to claim 1, characterized in that Extracting key frames of the support set and the test set respectively through a pre-trained network, and constructing a key frame retrieval library based on the support set through the Faiss method includes: Extracting key frames of the support set and the test set; Using a large-scale pre-trained network CLIP to extract features from the key frames in the support set and the test set to obtain key frame features, and normalizing the key frame features to obtain normalized features; A retrieval library is constructed based on the support set through the open source library Faiss, wherein the retrieval library contains the number of key frames and feature vectors of specified dimensions.

3. The method according to claim 1, characterized in that Classification training is performed based on the key frames of the test set and the key frame retrieval library by a preset small sample classification method, and the classification model obtained includes: The K nearest neighbor algorithm is used to determine the key frames with the most political categories in the K search results as the first prediction result of the category of the target sample; Calculate the inter-frame distance between a single key frame in the test video and the supporting samples in the key frame retrieval library of the support set, complete the retrieval iteration for the key frame of each video in the test set, and sum the inter-frame distance parameters as the average inter-frame distance between the test video and the supporting video, and use the supporting video with the smallest average inter-frame distance with the test video as the second prediction result; Calculating the average of the average inter-frame distances between the test video and all supporting videos in each small sample category, and taking the category with the smallest average inter-frame distance as the third prediction result of the test video; The classification model is modified according to the first prediction result, the second prediction result and the third prediction result to obtain a target classification model.

4. The method according to claim 3, characterized in that After obtaining the classification model, the method further includes: Preprocessing and labeling the training samples; Build a classification network, use the pre-processed pre-trained network as the backbone, use the linear layer as the classification head, randomly select N consecutive frames from the video key frames as the network input, and obtain the video prediction category; The target classification model is revised again according to the video prediction category.

5. The method according to claim 2, characterized in that: After constructing a search library based on the support set through the open source library Faiss, the method further includes: The similarity index between samples is calculated by the following formula: Among them, di is the distance between frames, x and xi are the samples to be retrieved and any key frames in the support set, respectively.

6. The method according to claim 2, characterized in that The designated dimension is the political dimension.

7. A video category determination device, characterized in that: include: A first acquisition unit is used to acquire multiple video data sets as training samples, wherein the training samples include a support set and a test set; An extraction unit, used to extract key frames of the support set and the test set respectively through a pre-trained network, and construct a key frame retrieval library based on the support set through the Faiss method; A training unit, used for performing classification training based on the key frames of the test set and the key frame retrieval library by a preset small sample classification method to obtain a classification model; The second acquisition unit is used to acquire the video to be classified, and classify the video to be classified by using the classification model to obtain a classification result.

8. The device according to claim 7, characterized in that The extraction unit comprises: An extraction module, used for extracting key frames of the support set and the test set; A processing module, used to extract features from the key frames in the support set and the test set using a large-scale pre-trained network CLIP to obtain key frame features, and normalize the key frame features to obtain normalized features; A construction module is used to construct a retrieval library based on the support set through the open source library Faiss, wherein the retrieval library contains the number of key frames and feature vectors of specified dimensions.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 6 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 6.

Citation Information

Cited By

  • Video classification method and device, computer equipment, storage medium and program product

    CN121582849A