Video classification model course learning training method fusing annealing mode

Abstract

The invention discloses a video classification model course learning training method fusing an annealing mode, and the method comprises the steps: 1) extracting a frame from a Kinetics400 data set, and obtaining a frame image for describing a video; 2) performing evaluation by using a SimpleVQA video quality evaluation algorithm; 3) performing evaluation by using a Fast VQA video quality evaluation algorithm; 4) carrying out weighted integration on the obtained video quality evaluation scores, and carrying out data subset division according to an annealing organization method; and 5) modifying a training scheduling method of the Video Swin Transformer model, and after the model converges, starting the scheduling of the training subset in the next difficulty stage by the training method until all the Kinetics400 data sets are completed. The method belongs to the technical field of computer video classification, and is high in recognition precision and easy to reuse when high-difficulty action videos are processed.

Description

Technical Field

[0001] The invention belongs to the technical field of computer video classification, and relates to a video classification model course learning and training method in a fusion annealing manner. Background Art

[0002] With the rapid development of the Internet and multimedia technology, a huge amount of videos are recorded and shared every day. As an important feature of video content, human motion analysis and recognition have become the key and difficult problem of video understanding. The current datasets try to cover various actions, but there is still a huge gap with the behaviors in the real world. Most of the pedestrian behaviors collected in the existing datasets are performed and simulated by staff. Even the sample sets extracted from the real world cannot completely solve the loss of accuracy caused by the ambiguity and uncertainty of the actions.

[0003] In traditional machine learning algorithms, all training samples are randomly presented to the model, ignoring the various complexities of the data samples and the current learning state of the model. Existing video action recognition methods mainly focus on model modification, improving the recognition accuracy of human behavior by designing clever models, ignoring the importance of data set quality and training method modification in tapping the model's potential.

[0004] Therefore, there is an urgent need to develop a new training method that can narrow the gap between simulated data and real events in the context of existing data sets and better adapt to action recognition in actual extreme scenarios. Summary of the invention

[0005] The purpose of the present invention is to provide a video classification model course learning and training method in a fusion annealing manner, which solves the problem that the existing technology has low recognition accuracy of human behavior and is not easy to reuse (reuse) when processing high-difficulty action videos.

[0006] The technical solution adopted by the present invention is to integrate the video classification model course learning and training method in an annealing manner, and implement it according to the following steps:

[0007] Step 1: Extract frames from the Kinetics400 dataset to obtain frame images describing the video;

[0008] Step 2: Use the SimpleVQA video quality assessment algorithm to evaluate all videos in the processed dataset;

[0009] Step 3: Use the Fast VQA video quality assessment algorithm to evaluate all videos in the processed dataset;

[0010] Step 4: Perform weighted synthesis on the obtained video quality assessment scores and divide the data into subsets according to the "annealing" organization method;

[0011] Step 5: Modify the training scheduling method of the Video Swin Transformer model. When the model converges, the training method starts scheduling the training subset for the next difficulty stage until the entire Kinetics400 dataset is completed.

[0012] The beneficial effects of the present invention are: first, the video quality assessment method is integrated with the annealing method, and a new deep learning model training method is proposed, which improves the recognition accuracy of the Video Swin Transformer video action recognition model in the test set while retaining the original processing speed. Second, the course learning idea is integrated into the deep learning training process. This course organization mechanism training can better eliminate the catastrophic forgetting problem of simple knowledge during the training process, and tap the recognition potential of the VideoSwin Transformer model in the face of difficult action recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 It is the overall framework structure diagram of the course learning of the method of the present invention;

[0014] Figure 2 It is a schematic diagram of organizing a data set by the annealing method of the method of the present invention;

[0015] Figure 3 It is a model training flow chart of the method of the present invention. DETAILED DESCRIPTION

[0016] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] The present invention proposes a training method that combines the basic idea of ​​course learning with video quality evaluation and annealing. The principle is: first, a video training data set is subjected to video quality evaluation and data subset division. According to the "annealing" method, the data set videos are divided into M groups of data subsets according to the evaluation score S. Different data subsets represent different difficulty levels and the number of these M groups of data subsets is different and there is no overlap between them. Starting from the simple, the Video Swin Transformer model is trained with the easy data subsets; then, the difficulty level of the data subset is gradually increased, and the next difficulty level course training is started whenever the model loss value reaches stable convergence until the training of the entire data set is completed, so as to improve the recognition accuracy of the video action recognition model.

[0018] Reference Figure 1 The video classification model course learning and training method of the present invention is implemented according to the following steps:

[0019] Step 1: Extract frames from the Kinetics400 dataset to obtain frame images describing the video;

[0020] For the large-scale Kinetics400 dataset (all video datasets), using the original data in video format can save a lot of space, but video decoding must be performed when running the model, which consumes a lot of computing power. However, if the original video frames are directly stored on the hard disk, it will take up a lot of disk space and will cause problems such as blurring noise and compression noise. Therefore, the original frames need to be stored on the solid-state drive through soft links to increase the speed of processing frame format data.

[0021] The specific process is:

[0022] 1.1) Arrange the downloaded dataset into a two-level structure directory, set the final output folder of the dataset to the folder on the SSD, and soft-link the output folder to the folder on the SSD using the ln -s command;

[0023] 1.2) Use the ffmpeg tool through the bash script file to extract the video frames, generate the output images required by the model, and obtain the processed data set.

[0024] Step 2: Use the SimpleVQA video quality assessment algorithm to evaluate all videos in the data set processed in step 1, and record the corresponding score of each video;

[0025] The Simple VQA video quality assessment algorithm mainly relies on two quality perception features, spatial distortion and motion distortion, as indicators when evaluating the quality of user-generated videos. Spatial distortion refers to artifacts introduced into video frames, such as noise, blur, compression, and low visibility. Motion distortion refers to jitter, lag, and other phenomena, which are mainly caused by unstable shooting equipment, fast-moving objects, and low network bandwidth.

[0026] The specific process is:

[0027] 2.1) Extract spatial distortion quality perception features. Input the video frame into the CNN model. Since the features extracted by the shallow layer of the CNN model contain low-level information of the image, which are mainly low-level features such as edges, corners, and textures, the low-level information is easily affected by distortion and is therefore distortion-perceptual.

[0028] Calculate the feature maps extracted at all stages of the CNN model, denoted as x k , x represents the video frame currently input into the CNN model, k represents the feature map extracted from the kth layer in the CNN model; the feature maps extracted from all stages of the CNN model are calculated to obtain the global mean and standard deviation Next, we extract the quality perception features, expressed as follows:

[0029]

[0030] Then, the global mean and standard deviation Perform splicing operation to obtain the distortion features corresponding to frame x The expression is as follows:

[0031]

[0032] Finally, the distortion features of N frames contained in a video are spliced ​​to obtain the spatial distortion quality perception feature F s , the expression is as follows:

[0033]

[0034] 2.2) Extracting motion distortion quality perception features. Motion distortion quality perception features are features that exist between frames and are difficult to extract. However, the feature representation of the action recognition network can reflect the motion information of the video to a certain extent. Therefore, the pre-trained 3D CNN action recognition model is used as a motion feature extractor to extract motion features as complementary quality perception features to obtain the motion distortion quality perception features F of each video. m , the expression is as follows:

[0035] F m =Motion(V) (5)

[0036] Among them, V represents the input video clip, and Motion represents the 3D CNN action recognition model;

[0037] 2.3) The obtained spatial distortion quality perception feature and the obtained motion distortion quality perception feature are concatenated to obtain the quality perception feature F for subsequent evaluation needs. The expression is as follows:

[0038] F = cat(F s ,F m ) (6)

[0039] 2.4) Using regression models to map quality perception features into quality scores,

[0040] Due to the simplicity and effectiveness of multi-layer perception, a multi-layer perception model (MLP) is used as a regression model to obtain the quality of the video; the multi-layer perception model consists of two fully connected layers, and the two fully connected layers have 128 neurons and 1 neuron respectively. The expression is as follows:

[0041] q=MLP(F) (7)

[0042] Among them, MLP represents the multi-layer perception model function, q represents the quality score of the video, and F represents the quality perception feature.

[0043] Step 3: Use the Fast VQA video quality assessment algorithm to evaluate all videos in the data set processed in step 1, and record the corresponding score of each video;

[0044] The Fast-VQA video quality assessment algorithm uses Grid Mini-patch Sampling (GMS) to consider local quality at the original resolution while covering global quality and contextual relationships through sampling in a uniform grid. It also aligns video clips in time to retain the inter-frame changes of small tiles at the original resolution, thereby maintaining temporal quality sensitivity, being able to detect temporal distortion in the video, and distinguishing between videos with severe lens shaking and videos with relatively stable lenses.

[0045] The Fast VQA video quality assessment algorithm can effectively implement end-to-end video quality assessment and is a video quality assessment method that is more suitable for long videos. The specific process is:

[0046] 3.1) Input the video into the GMS sampling module for segmented sampling. The specific steps are as follows:

[0047] 3.1.1) Complete the uniform division of video frames,

[0048] In order to include each part of the video frame in the quality assessment range and uniformly evaluate the quality of different regions, the video frame needs to be divided into multiple uniform G f ×G f Each local area of ​​the video frame is represented by a grid of size Indicates that i and j represent the grid of the i-th row and j-th column respectively. The expression is as follows:

[0049]

[0050] Where V represents the input video clip, t represents the tth frame of the video, H represents the height of the video frame, and W represents the width of the video frame;

[0051] 3.1.2) Extract local texture features,

[0052] In order to preserve the local texture which is crucial in video quality assessment, it is necessary to extract the local texture quality in the divided grid; random block sampling is used to extract the local texture quality from each grid. Sampling a size S f ×S f The feature block is represented as represents the local texture sampling block of the grid at row i and column j of frame t, then The expression is as follows:

[0053]

[0054] in, represents the local area of ​​the i-th row and j-th column in the video frame t, A t Represents a local texture sampling operation;

[0055] 3.1.3) Time alignment,

[0056] In order to preserve the original time changes and global scene information in the video, strict constraints are required for time alignment when sampling grids on different frames to maintain the contextual relationship between different frames. The expression of the constraint process is as follows:

[0057]

[0058] in, Represents a mathematical symbol, with any meaning; Indicates the time independent variable in the formula; in addition, 0≤i<G f And 0≤j<G f .

[0059] 3.2) Input the sampled block into the FANet network to obtain the video quality score.

[0060] The obtained sampling blocks are input into the FANet network. Due to the discontinuity between different sampling blocks, each sampling block has different texture properties.

[0061] If the features of different sampling blocks are pooled before evaluation, the quality of each sampling block will be confused with each other. To avoid this problem, the sampling block needs to be regressed through a nonlinear layer (RNL) and then pooled. The expression of the obtained quality evaluation score is as follows:

[0062] q=Pool(RNL(f)) (11)

[0063] Among them, f represents the intermediate extracted features of the FANet network, q represents the quality assessment score, Pool represents the pooling operation, and RNL represents the nonlinear layer.

[0064] Step 4: Weighted synthesis of the video quality assessment scores obtained in step 2 and step 3. Divide the videos in the data set into data subsets according to the recorded scores in accordance with the "annealing" organization method. The higher the score, the higher the video resolution, the less video content information, the smaller the video noise, and the lower the recognition difficulty. This provides a soft transition from easy to difficult training for the model.

[0065] Reference Figure 2 , the specific process is:

[0066] 4.1) Score weighted synthesis,

[0067] The scores obtained by the Simple VQA video quality assessment algorithm and the Fast VQA video quality assessment algorithm are weighted. Since there are many indoor scenes in the Kinetics400 dataset, the Simple VQA algorithm, which is more biased towards video quality assessment of indoor scenes, has a higher weight when this step is implemented, and the final quality score of each video in the final dataset is obtained;

[0068] 4.2) The dataset is organized again.

[0069] The datasets are sorted according to their quality scores and divided into M data subsets according to M difficulty intervals, without overlap, to obtain the re-divided data subsets {D i}, different data subsets correspond to different difficulty levels, from the simplest D1 to the most difficult D M ;

[0070] Then, we use the "annealing" method to sample and construct a new training data set, also called a course, from easy to difficult. The expression of the sampling ratio is as follows:

[0071]

[0072] Where i represents courses of different difficulty levels, Num(·) represents the number of videos in the computational data subset,

[0073] Thus, we construct a course A = {A1, A2, ..., A M}, i=1,2,...,M represents the increasing difficulty level of the course.

[0074] Step 5: Modify the training scheduling method of the Video Swin Transformer model.

[0075] Whether the model has reached convergence is determined based on the loss function and loss fluctuation value during the model training process. When the model converges, the training method starts scheduling the training subset for the next difficulty stage until the model training completes the entire Kinetics400 data set. At this time, the model test results significantly improve the accuracy on the high-difficulty test subset while retaining the original processing speed.

[0076] Reference Figure 3 , the specific process is:

[0077] Modify the training scheduling method of the Video Swin Transfommer model. According to the loss function and loss fluctuation value during the model training process, determine whether the model has reached convergence. The expression for determining the loss fluctuation value is as follows:

[0078] σ(Aj)≤σ model (13)

[0079] Among them, σ model Represents the loss fluctuation value of the model, which is set to a constant of 0.1. When the loss function converges and the current course loss is less than the loss fluctuation value of the model, the scheduling of the training subset for the next difficulty stage can be started until the model training completes the entire Kinetics400 data set.

[0080] Example 1

[0081] The first set of Kinetics400 large-scale video action recognition datasets contains 400 human action categories, mainly indoor and outdoor entertainment, daily life and other scenes. The video clips of each category are about 10 seconds long and come from YouTube, covering a variety of backgrounds, lighting, action speeds, complex scenes and background interference. Figure 1 According to the above steps of the present invention, the specific implementation is as follows:

[0082] Step 1: Frame extraction: extract all videos in the Kinetics400 dataset, such as running and jumping in sports games, and walking in public scenes. Use bash script files and ffmpeg tools to extract video frames and generate the input data required by the model.

[0083] Step 2: Use Simple VQA to evaluate the quality of the extracted video frames in the Kinetics400 dataset using the Simple VQA algorithm to evaluate the clarity, color saturation, motion blur, etc. Each video is given a quality score, and the higher the score, the better the video quality (such as high definition and low noise).

[0084] Step 3: Use Fast VQA evaluation, and then use the Fast VQA algorithm to evaluate the quality of the dataset videos again. Fast VQA focuses on faster evaluation of the overall quality of the video, including the ability to distinguish actions in complex backgrounds. This step complements Simple VQA and generates another set of scores.

[0085] Step 4: Weighted fusion and annealing division. According to the quality scores of 232,380 training videos in the Kinetics400 dataset generated by Simple VQA and Fast VQA, the video name and evaluation score are recorded in the corresponding txt file; since the Kinetics400 dataset is biased towards indoor scenes, the weight of Fast VQA is 70% and the weight of Simple VQA is 30%. The weighted average is performed to obtain the comprehensive quality score of each video. Based on the score, the "annealing" strategy is applied, and a total of 5 courses are organized in proportion from easy to difficult, referring to Figure 2 , the final result is that the first course has 49,685 videos, the second course has 63,534 videos, the third course has 61,335 videos, the fourth course has 55,769 videos, and the fifth course has 46,476 videos. The total number of videos in these five courses exceeds 260,000. This is because of the way the dataset is organized later, and there may be duplications between each subset. The videos are divided into subsets from low difficulty to high difficulty. The subset with low video quality score is used as the initial training data, and the subset with high video score is used for later training.

[0086] Step 5: Training and Results, during the model training phase, refer to Figure 3 First, the Video Swin Transformer model is trained with a low-difficulty video subset. As the training progresses, the model gradually introduces higher-quality and more complex video subsets. When the model loss fluctuation is less than 0.1, the next course is started until the five courses are trained. When the model is fully trained, the test results on the Kinetics400 dataset test set show that the model not only maintains the ability to process quickly, but also improves the top-5 recognition accuracy in complex scenes and action recognition from 95.5% to 95.7%. This method is suitable for security monitoring systems and solves the problem of low recognition rate of difficult actions in complex scenes.

[0087] Example 2

[0088] For the second group of Something-SomethingV2 large video action recognition dataset, this dataset focuses on object interaction, contains 220,847 video clips, covers 174 action categories, and the action types have high time correlation and complexity, which is suitable for studying tasks such as action classification and spatiotemporal modeling. The above steps of the present invention are implemented as follows:

[0089] Step 1: Frame extraction, extract video frames from the Something-Somethingv2 dataset. Since the action categories of this dataset are mostly interactive actions between hands and objects, frame extraction pays special attention to key time points such as the start and end of the action to ensure that the complete action process is captured, providing high-quality data for subsequent quality evaluation and model training.

[0090] Step 2: Use Simple VQA to evaluate. Many video clips in Something-Somethingv2 have complex scenes and fast-changing actions, so Simple VQA specifically evaluates factors such as clarity, separation of objects and backgrounds, and motion blur in the video. Videos with higher scores mean that they have clear actions and less background interference, and are suitable as initial training data for the model.

[0091] Step 3: Use Fast VQA evaluation to evaluate the same set of video frames using the Fast VQA algorithm. FastVQA focuses on noise, brightness changes, etc. in the video, quickly analyzes the overall quality of the video in complex scenes, and generates a second set of scores corresponding to Simple VQA.

[0092] Step 4: Weighted fusion and annealing division. According to the quality scores of 220,847 training videos in the Something-Somethingv2 dataset generated by Simple VQA and Fast VQA, the video names and evaluation scores are recorded in the corresponding txt files. Since the Something-Somethingv2 dataset is biased towards indoor scenes, the weight of Fast VQA is 80% and the weight of Simple VQA is 20%. The weighted average is performed to obtain the comprehensive quality score of each video. Based on the score, the "annealing" strategy is applied to divide the video into subsets from low difficulty to high difficulty. The subset with low video quality score is used as the initial training data, and the subset with high video score is used for later training.

[0093] Step 5: Training and Results, during the model training phase, refer to Figure 3 First, the Video Swin Transformer model is trained with a low-difficulty video subset. As the training progresses, the model gradually introduces higher-quality and more complex video subsets. When the model is fully trained, the test results on the Something-Somethingv2 dataset test set show that the model not only maintains the ability to process quickly, but also improves the top-5 recognition accuracy from 92.7% to 92.9% in complex scene and action recognition.

[0094] Example 3

[0095] For the third set of NWPU Campus video action recognition data set, the data set contains 43 scenes, 28 types of abnormal events, and 16 hours of video. Various types of abnormal actions will appear in the campus scene, and the action category definition of this data set is mostly related to the environment. The above steps of the present invention are implemented as follows:

[0096] Step 1: Frame extraction: extract video frames from the NWPU Campus dataset to provide high-quality data for subsequent quality assessment and model training.

[0097] Step 2: Use Simple VQA to evaluate. Many video clips of NWPU Campus have complex scenes and fast-changing actions, so Simple VQA specifically evaluates factors such as clarity, separation of objects and backgrounds, and motion blur in the video. Videos with higher scores mean that they have clear actions and less background interference, and are suitable as initial training data for the model.

[0098] Step 3: Use Fast VQA evaluation to evaluate the same set of video frames using the Fast VQA algorithm. FastVQA focuses on noise, brightness changes, etc. in the video, quickly analyzes the overall quality of the video in complex scenes, and generates a second set of scores corresponding to Simple VQA.

[0099] Step 4: Weighted fusion and annealing division. According to the NWPU Campus data quality scores generated by Simple VQA and Fast VQA, the video name and evaluation score are recorded in the corresponding txt file; since the NWPU Campus dataset is biased towards outdoor scenes, the weight of Fast VQA is 20% and the weight of Simple VQA is 80%. The weighted average is performed to obtain the comprehensive quality score of each video. Based on the score, the "annealing" strategy is applied to divide the video into subsets from low difficulty to high difficulty. The subset with low video quality score is used as the initial training data, and the subset with high video score is used for later training.

[0100] Step 5: Training and Results, during the model training phase, refer to Figure 3 First, the Video Swin Transformer model was trained with a low-difficulty video subset. As the training progressed, the model gradually introduced higher-quality and more complex video subsets. When the model was fully trained, the test results on the NWPU Campus dataset test set showed that the model not only maintained its fast processing capabilities, but also improved the top-5 recognition accuracy in complex scene and action recognition from 85.1% to 85.6%.

Claims

1. A video classification model course learning and training method using fusion annealing, characterized in that: Follow these steps to implement: Step 1: Extract frames from the Kinetics400 dataset to obtain frame images describing the video; Step 2: Use the SimpleVQA video quality assessment algorithm to evaluate all videos in the processed dataset; Step 3: Use the Fast VQA video quality assessment algorithm to evaluate all videos in the processed dataset; Step 4: Perform weighted synthesis on the obtained video quality assessment scores and divide the data into subsets according to the "annealing" organization method; Step 5: Modify the training scheduling method of the Video Swin Transformer model. When the model converges, the training method starts scheduling the training subset for the next difficulty stage until the entire Kinetics400 dataset is completed.

2. The video classification model course learning and training method of the fusion annealing method according to claim 1 is characterized in that: In step 1, the specific process is: 1.1) Arrange the downloaded dataset into a two-level structure directory, set the final output folder of the dataset to the folder on the SSD, and soft-link the output folder to the folder on the SSD using the ln -s command; 1.2) Use the ffmpeg tool through the bash script file to extract the video frames, generate the output images required by the model, and obtain the processed data set.

3. The video classification model course learning and training method of the fusion annealing method according to claim 1 is characterized in that: In step 2, the specific process is: 2.1) Extract spatial distortion quality perception features, Calculate the feature maps extracted at all stages of the CNN model, denoted as x k , x represents the video frame currently input into the CNN model, k represents the feature map extracted from the kth layer in the CNN model; the feature maps extracted from all stages of the CNN model are calculated to obtain the global mean and standard deviation Next, we extract the quality perception features, expressed as follows: Then, the global mean and standard deviation Perform splicing operation to obtain the distortion features corresponding to frame x The expression is as follows: Finally, the distortion features of N frames contained in a video are spliced ​​to obtain the spatial distortion quality perception feature F s , the expression is as follows: 2.2) Extract motion distortion quality perception features. Use the pre-trained 3D CNN action recognition model as a motion feature extractor to extract motion features as complementary quality perception features to obtain the motion distortion quality perception features F of each video. m , the expression is as follows: F m =Motion(V) (5) Where V represents the input video clip, and Motion represents the 3D CNN action recognition model; 2.3) The obtained spatial distortion quality perception feature and the obtained motion distortion quality perception feature are concatenated to obtain the quality perception feature F for subsequent evaluation needs. The expression is as follows: F=cat(F s ,F m ) (6) 2.4) Using regression models to map quality perception features into quality scores, The multi-layer perception model is used as a regression model to obtain the quality of the video; the multi-layer perception model consists of 2 fully connected layers, and the two fully connected layers have 128 neurons and 1 neuron respectively. The expression is as follows: q=MLP(F) (7) Among them, MLP represents the multi-layer perception model function, q represents the quality score of the video, and F represents the quality perception feature.

4. The video classification model course learning and training method of the fusion annealing method according to claim 1 is characterized in that: In step 3, the specific process is: 3.1) Input the video into the GMS sampling module for segmented sampling. The specific steps are as follows: 3.1.1) Complete the uniform division of video frames, Divide the video frame into multiple uniform G f ×G f Each local area of ​​the video frame is represented by a grid of size Indicates that i and j represent the grid of the i-th row and j-th column respectively. The expression is as follows: Where V represents the input video clip, t represents the tth frame of the video, H represents the height of the video frame, and W represents the width of the video frame; 3.1.2) Extract local texture features, Extract local texture quality from the divided grid; use random block sampling to extract local texture quality from each grid. Sampling a size S f ×S f The feature block is represented as represents the local texture sampling block of the grid at row i and column j of frame t, then The expression is as follows: in, represents the local area of ​​the i-th row and j-th column in the video frame t, A t Represents a local texture sampling operation; 3.1.3) Time alignment, When sampling grids on different frames, strict constraints are required for time alignment to maintain the contextual relationship between different frames. The expression of the constraint process is as follows: in, Represents a mathematical symbol, with any meaning; Indicates the time independent variable in the formula; in addition, 0≤i<G f And 0≤j<G f ; 3.2) Input the sampled block into the FANet network to obtain the video quality score. The obtained sampling blocks are input into the FANet network. Due to the discontinuity between different sampling blocks, each sampling block has different texture properties. The sampling blocks are first regressed through a nonlinear layer and then pooled. The expression of the obtained quality assessment score is as follows: q=Pool(RNL(f)) (11) Among them, f represents the intermediate extracted features of the FANet network, q represents the quality assessment score, Pool represents the pooling operation, and RNL represents the nonlinear layer.

5. The video classification model course learning and training method of the fusion annealing method according to claim 1 is characterized in that: In step 4, the specific process is: 4.1) Score weighted synthesis, The scores obtained by the Simple VQA video quality assessment algorithm and the Fast VQA video quality assessment algorithm are weighted to obtain the final quality score of each video in the final data set; 4.2) The dataset is organized again. The datasets are sorted according to their quality scores and divided into M data subsets according to M difficulty intervals, without overlap, to obtain the re-divided data subsets {D i }, different data subsets correspond to different difficulty levels, from the simplest D1 to the most difficult D M ; Then, we use the "annealing" method to sample and construct a new training data set, also called a course, from easy to difficult. The expression of the sampling ratio is as follows: Where i represents courses of different difficulty levels, and Num(·) represents the number of videos in the calculated data subset, thereby constructing a course A = {A1, A2, ..., A M }, i=1,2,...,M represents the increasing difficulty level of the course.

6. The video classification model course learning and training method of the fusion annealing method according to claim 1 is characterized in that: In step 5, the specific process is: Modify the training scheduling method of the Video Swin Transfommer model. According to the loss function and loss fluctuation value during the model training process, determine whether the model has reached convergence. The expression for the loss fluctuation value is as follows: σ(Aj)≤σ model (13) Among them, σ model Represents the loss fluctuation value of the model, which is set to a constant of 0.

1. When the loss function converges and the current course loss is less than the loss fluctuation value of the model, the scheduling of the training subset for the next difficulty stage can be started until the model training completes the entire Kinetics400 data set.

Images