A method for adaptive frame-skipping single monkey video data classification for long-tailed distribution

Abstract

The application is a kind of adaptive frame extraction single monkey video data classification method for long tail distribution, comprising the following steps: step 1: collecting single monkey video, establishing labeled single monkey action video dataset; step 2: using action dataset to perform adaptive frame extraction resampling training network model; step 3: using action dataset to evaluate network model, and returning to step 2 for further training for network model not meeting the expectation. The method is simple and effective for solving the problem of long tail distribution imbalance of video data. Through the knowledge learned by the neural network in the training stage, the training data is adaptively resampled by frame extraction, and the essence of resampling is to sample more tail class frames with poor performance and less head class frames with good performance to improve the classification performance of the model.

Description

Technical Field

[0001] This invention relates to the field of video image recognition technology, and in particular to an adaptive frame-sampling single monkey video data classification method for long-tailed distribution. Background Technology

[0002] In nature, real-world data often exhibits an imbalanced, long-tailed distribution. For example, in this dataset, the frequency of monkeys sitting is very high, while the frequency of jumping upwards and downwards is very low. In natural scenes, only a small number of categories have a large number of samples, while the remaining categories have only a small number of samples. This extreme class imbalance poses a significant challenge to deep learning and visual recognition, causing data-based model learning to be dominated by the "head" category (those with a large number of samples), leading to overfitting, while the "tail" category (those with a small number of samples) underfits, resulting in poor model performance and poor generalization ability. Therefore, addressing the imbalance problem of long-tailed data distribution has significant application potential. Currently, most algorithms for addressing data imbalance are designed for image data, with limited research on video data. Therefore, there is a more urgent need to address the problem of long-tailed video data distribution.

[0003] Currently, the main methods for solving the long-tail imbalance problem mainly fall into three categories: (1) Class rebalancing. This includes: data resampling, artificially balancing the data by oversampling the "tail" class or undersampling the "head" class before training; class cost-sensitive learning, reweighting or adjusting the classification boundary; and Logit adjustment. This method is simpler and easier to operate than other methods, and has better results and theoretical support. However, the improvement of minority class performance comes at the expense of majority class performance, and it still does not fundamentally solve the problem of data imbalance. (2) Information augmentation. This includes: transfer learning and data augmentation. Transfer learning includes: transfer learning from head class to tail class, model pre-training, knowledge distillation, and self-training. This method enhances model training by introducing additional information, which can improve minority class performance without losing majority class performance. It is practical, but the augmentation of head class is greater than that of tail class, which may further aggravate the data imbalance problem. (3) Network module enhancement. This includes enhancing the feature learning network; designing special classifiers; decoupling the feature extractor and classifier; and ensemble learning. Feature learning networks can be improved through methods such as metric learning, sequential training, prototype learning, and transfer learning. Ensemble learning can be divided into two-branch networks and multi-expert networks. While using multiple experts can achieve better head-to-tail performance, it increases the computational cost of the model. These three types of methods are effective in addressing the imbalance problem in image data, but their direct application to video data still presents certain challenges. Summary of the Invention

[0004] The purpose of this invention is to propose a simple algorithm that does not require additional training and is applicable to video data, so that it can achieve good classification performance for both the first and last classes of long-tailed distributed video data.

[0005] This invention provides an adaptive frame-stripping single-monkey video data classification method for long-tailed distributions, the adaptive frame-stripping single-monkey video data classification method comprising the following steps:

[0006] Step 1: Collect videos of individual monkeys and create a dataset of labeled videos of individual monkey actions;

[0007] Step 2: Use the action dataset to perform adaptive frame resampling and train the network model;

[0008] Step 3: Evaluate the network model using the action dataset. For network models that do not meet expectations, return to Step 2 for further training.

[0009] Furthermore, in step 1, the start and end times of the specified actions performed by the monkey in the single monkey video are recorded; monkey video action segments are extracted based on the start and end times, and labeled single monkey action video datasets are obtained by standardizing the monkey actions in the video segments;

[0010] The labeled single monkey action video dataset includes a training set and a test set.

[0011] Furthermore, step 2 includes the following steps:

[0012] Step 21: Resample according to the sampling ratio parameter, and extract features for each type of video data respectively;

[0013] Step 22: Calculate the average precision for each video category;

[0014] Step 23: Update the sampling ratio parameter based on the average accuracy of each video category;

[0015] Step 24: Repeat steps 21 to 24 until training is complete.

[0016] Furthermore, in step 21, the initial sampling ratio parameter β = 0.5.

[0017] Furthermore, in step 21, the resampling process is as follows:

[0018] Randomly select a pair of videos, perform feature extraction, and extract L based on the sampling ratio parameter β. i Frame and L j The feature vector of a frame.

[0019] Number of frames extracted L i and L j They are respectively:

[0020] L i = (1-β)×L

[0021] L j =β×L

[0022] Where L is the number of frames in the monkey video action segment;

[0023] Extracting frame number L i eigenvectors In the middle, extract L sequentially from the beginning. i Frame feature vector fragment Extracting frame number L j eigenvectors In the middle, extract the remaining L in order. j Frame feature vector fragment

[0024] connect Obtain the feature vector corresponding to frame number L

[0025] Furthermore, in step 21, in each batch of samples used to train the neural network, half of the total number of original samples are randomly selected, and at the same time, the same number of resampled samples as the randomly selected original samples are generated as new samples, and the neural network is trained using the new samples.

[0026] Furthermore, in step 22, the predicted and true values ​​for each batch in the training set are recorded, and after one generation of training is completed, the average precision rAP for each category of videos in the training set is calculated.

[0027] Furthermore, in step 23, the sampling ratio parameter is updated:

[0028]

[0029] Among them, rAP i and rAP j The average precision of corresponding categories in a randomly selected pair of videos from the same batch.

[0030] The beneficial effects achieved by this invention are:

[0031] This invention addresses the problem of imbalanced long-tail distribution in video data by proposing a simple and effective method. The method adaptively resamples training data frames using knowledge learned by the neural network during the training phase. The essence of resampling is to sample more poorly performing tail-type frames and fewer well-performing head-type frames to improve the model's classification performance. Attached Figure Description

[0032] Figure 1 This is a flowchart illustrating an adaptive frame-slicing single-monkey video data classification method for long-tailed distributions. Detailed Implementation

[0033] The technical solution of the present invention will be described in more detail below with reference to the accompanying drawings. The present invention includes, but is not limited to, the following embodiments.

[0034] As attached Figure 1 As shown, this invention provides an adaptive frame-sampling single-monkey video data classification method for long-tailed distribution, which mainly includes the following steps:

[0035] Step 1: Collect videos of individual monkeys and create a dataset of labeled videos of individual monkey actions;

[0036] Step 2: Use the action dataset to perform adaptive frame resampling and train the network model;

[0037] Step 3: Evaluate the network model using the action dataset. For network models that do not meet expectations, return to Step 2 for further training.

[0038] In step 1, video is first captured of the crab-eating macaques in the cage using a camera to obtain video data of individual monkeys and establish a dataset of individual monkey actions. Next, the entire video is reviewed, and if a monkey performs a specified action, the start and end times of the action are recorded to categorize the monkey's actions. Then, video clips of monkey actions are extracted based on the recorded times, and labeled individual monkey action video datasets are obtained by standardizing the monkey actions in the video clips. These datasets are then divided into training and testing sets. Finally, statistics are performed on the various action videos to determine if the dataset exhibits a long-tail distribution.

[0039] In this dataset, for monkey video action segments uniformly represented as L frames, the video data containing L frames in the single monkey action annotation dataset is denoted as V = {f1, f2, ..., f...} L Let its label be y. Let n be... k Let k be the number of videos in the k-th category. This is the total number of videos across all categories.

[0040] In step 2, let the classification model parameters be f. θ Training a single-monkey video action classification model on a deep neural network using a training set specifically includes:

[0041] Step 21: Resample according to the sampling ratio parameter, and extract features for each type of video data.

[0042] The initial sampling ratio parameter β = 0.5. Let V be the input L frames of video data. The feature vector extracted from video data V is denoted as... This method does not resample the original data, but resamples in the feature space to preserve the temporal information in the video data.

[0043] During the resampling process, a pair of videos is randomly selected for feature extraction, and L is extracted according to the sampling ratio parameter. i Frame and L j The feature vector of a frame.

[0044] Number of frames extracted L i and L j They are respectively:

[0045] L i = (1-β)×L

[0046] L j =β×L

[0047] Extracting frame number L i eigenvectors In the middle, extract L sequentially from the beginning. i Frame feature vector fragment Extracting frame number L j eigenvectors In the middle, extract the remaining L in order. j Frame feature vector fragment

[0048] Through connection This will give us a feature vector corresponding to frame number L. Feature vector It is a fragment containing multiple labels, denoted as a vector.

[0049] In each batch of samples used to train the neural network, half of the original samples are randomly selected, and at the same time, the same number of resampled samples as the randomly selected original samples are generated as new samples. The neural network is then trained using these new samples.

[0050] Step 22: Calculate the average accuracy for each video category.

[0051] Record the predicted and true values ​​for each batch in the training set. After completing one epoch of training, calculate the average precision (ap) for each category of videos in the training set and record it as rAP. Update rAP as training progresses.

[0052] Step 23 updates the sampling ratio parameter based on the average accuracy of each video category.

[0053] Let a randomly selected pair of feature vectors extracted from the video be denoted as... And according to step 22, the corresponding category y can be calculated. i rAP i and category y j rAP j Define a sampling ratio parameter as This indicates that under the current parameters, the network's response to category y... i and category y j The relative performance.

[0054] Step 24: Repeat steps 21 to 24 until training is complete.

[0055] In step 3, accuracy and recall are selected as the criteria for evaluating model performance. The backbone parameters trained using the original long-tailed distribution data and the normalized classifier parameters are used to predict and evaluate the training set. Accuracy Recall rate Here, TP represents the number of positive classes predicted as positive, FP represents the number of negative classes predicted as positive, and FN represents the number of negative classes predicted as negative. In addition to calculating the accuracy for all classes, the accuracy for multi-class (large sample size), medium-class (moderate sample size), and minority-class (small sample size) classes can also be calculated to compare the model's predictive performance on "head" and "tail" data.

[0056] If the results are not satisfactory, you can go back to step 2 to further train the neural network to obtain a better performing model.

[0057] This invention addresses the problem of imbalanced long-tail distribution in video data by proposing a simple and effective method. The method adaptively resamples training data frames using knowledge learned by the neural network during the training phase. The essence of resampling is to sample more poorly performing tail-type frames and fewer well-performing head-type frames to improve the model's classification performance.

[0058] This invention is not limited to the specific embodiments described above. Those skilled in the art can implement this invention using various other specific embodiments based on the disclosed content of the embodiments and accompanying drawings. Therefore, any design that adopts the design structure and concept of this invention and makes some simple changes or modifications falls within the protection scope of this invention.

Claims

1. An adaptive frame-stripping single-monkey video data classification method for long-tailed distribution, characterized in that, The adaptive frame-stripping single-monkey video data classification method includes the following steps: Step 1: Collect videos of individual monkeys and create a dataset of labeled videos of individual monkey actions; Step 2: Use the action dataset to perform adaptive frame resampling and train the network model; Step 3: Evaluate the network model using the action dataset. For network models that do not meet expectations, return to Step 2 for further training. Step 2 includes the following steps: Step 21: Resample according to the sampling ratio parameter, and extract features for each type of video data respectively; Step 22: Calculate the average precision for each video category; Step 23: Update the sampling ratio parameter based on the average accuracy of each video category; Step 24: Repeat steps 21 to 24 until training is complete; In step 21, the resampling process is as follows: Randomly select a pair of videos, extract features, and then apply the sampling ratio parameter to each pair. extract Frames and The feature vector of the frame; Number of frames extracted and They are respectively: ； ； Where L is the number of frames in the monkey video action segment; Extracting frame numbers eigenvectors Extract from the beginning in sequence. Frame feature vector fragment ; in extracting frame number eigenvectors Extract the remaining ones in order. Frame feature vector fragment ; connect , Obtain the feature vector corresponding to frame number L ; In each batch of samples used to train the neural network, half of the original samples are randomly selected, and at the same time, the same number of resampled samples as the randomly selected original samples are generated as new samples. The neural network is then trained using the new samples. In step 23, the sampling ratio parameter is updated: ； in, and The average precision of corresponding categories in a randomly selected pair of videos from the same batch.

2. The adaptive frame-sampling single-monkey video data classification method according to claim 1, characterized in that, In step 1, the start and end times of the specified actions performed by the monkeys in the single monkey video are recorded; monkey video action segments are extracted based on the start and end times, and labeled single monkey action video datasets are obtained by standardizing the monkey actions in the video segments; The labeled single monkey action video dataset includes a training set and a test set.

3. The adaptive frame-sampling single-monkey video data classification method according to claim 1, characterized in that, In step 21, the initial sampling ratio parameter .

4. The adaptive frame-sampling single-monkey video data classification method according to claim 1, characterized in that, In step 22, the predicted and true values ​​for each batch in the training set are recorded. After completing one generation of training, the average precision rAP for each category of videos in the training set is calculated.

Images