Video classification and identification method based on Video Swin Transform

Abstract

The invention discloses a video classification and identification method based on Video Swin Transform. The method comprises the following steps: preparing a data set to be identified; the data set is preprocessed, feature map information enters a network block, and a feature map is subjected to a multi-scale fusion module; the model uses an I3D classification head to carry out final classification on eight categories; and drawing a confusion matrix according to a classification result of the model, and reclassifying samples with model classification errors or adjusting classification boundaries. According to the method, the model adopts a sliding window design of Vdio Swin Transform, so that the model can learn feature information in a cross-window manner, and the parameter quantity in calculation is greatly reduced; moreover, the hierarchical model can process super-resolution picture information, and consumption of computing resources is also reduced; a time dimension T is added to the model, the time dimension is used for capturing continuous time frame information of an incoming video, and the action range of windowed attention is expanded to the space-time field from the space field.

Description

Technical Field

[0001] The present invention relates to the field of video classification, and in particular to a video classification and recognition method based on Video Swin Transformer. Background Art

[0002] A video classification model is a machine learning model that is specifically designed to recognize the content of a video. This model recognizes the actions or activities shown in the video by analyzing the video frames and the spatiotemporal relationships between frames. Video classification models are trained on a dataset of videos with a unique set of categories (such as different actions or activities). These categories can be human action recognition, such as running, clapping, waving, etc. The input to the model is the video frames, and the output is the probability of each class being represented in the video. Similar to image classification models, video classification models also use images as input to predict the probability that these images belong to predefined classes, but video classification models also process the spatiotemporal relationships between adjacent frames to recognize actions in the video.

[0003] Existing video classification models, such as the Transformer model, have defects. Although it can accurately extract the features of widely distributed objects in images, its computational resources and runtime memory requirements are relatively large, especially when the image resolution to be processed is larger. In addition, the detection performance of small objects in images is not ideal. Summary of the invention

[0004] The purpose of the present invention is to provide a video classification and recognition method based on Video Swin Transformer, so as to solve the above-mentioned problems existing in the prior art.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A video classification and recognition method based on Video Swin Transformer comprises the following steps:

[0007] S100, preparing a data set to be identified;

[0008] S200, preprocessing the data set and entering feature map information into a network block, and performing a multi-scale fusion module on the feature map;

[0009] The S300,model uses the I3D classification head to perform the final classification,on 8 categories;

[0010] S400: Draw a confusion matrix according to the classification results of the model, and reclassify or adjust the classification boundaries of samples that are misclassified by the model.

[0011] In some specific embodiments, the specific method of step S100 is:

[0012] The dataset uses 8 types of action videos selected from Kinetic400 as the experimental dataset, with 300 30fps videos in each category and 40 30fps videos in the test set. The length of each video is about 10s, and the video is in mp4 format.

[0013] In some specific embodiments, the specific method of step S200 is:

[0014] First, the data set is preprocessed, a linear embedding layer is used using 3D convolution, and the feature map size of the model is adjusted; the feature map information enters four network blocks, each of which includes a self-attention module and a downsampling module.

[0015] In some specific embodiments, the specific method of step S200 further includes:

[0016] The self-attention module includes a sliding window and a non-sliding window;

[0017] The self-attention module can effectively establish information connections between feature maps without increasing too much computing resources; the downsampling module uses large kernel convolution and Inverted Bottleneck architecture to reduce the size of the model feature map while adjusting the number of channels; the feature map multi-scale fusion module selects the feature map information of different sizes of the model for fusion so that the model can recognize objects of different sizes.

[0018] In some specific embodiments, the specific method of step S400 further includes:

[0019] The model accuracy, precision and recall rate indicators are calculated based on the confusion matrix. These indicators can reflect the performance of the model from different angles.

[0020] In some specific embodiments, the downsampling module is used to extract features of global information, reduce the size of the image feature map, filter out redundant feature information, and screen key information; and add multi-scale spatiotemporal feature information fusion to the network to improve the model's ability to detect small objects and enhance the model's feature fusion performance; finally, the model is simplified.

[0021] In some specific embodiments, it also includes: a self-attention mechanism module;

[0022] The self-attention mechanism module is used to encode the pixel-level information of all incoming data, and then focus on the relationship between their feature sequence information, so as to establish a network with associated global information.

[0023] In some specific embodiments, it further includes: a multi-scale feature information fusion module;

[0024] The multi-scale feature information fusion module is used to address the multi-scale problem in object detection and improve the model's ability to detect small objects without substantially increasing computing resources.

[0025] The beneficial effects of the present invention are as follows: the present invention discloses a video classification and recognition method based on Video Swin Transformer, comprising the following steps: preparing a data set to be recognized; preprocessing the data set, and entering feature map information into a network block, and performing a multi-scale fusion module on the feature map; the model uses an I3D classification head to perform a final classification on 8 categories; drawing a confusion matrix according to the classification results of the model, and reclassifying or adjusting the classification boundaries of samples that are misclassified by the model. The model of the present invention adopts the sliding window design of Video Swin Transformer, which enables the model to learn feature information across windows, thereby greatly reducing the amount of parameters in the calculation. In addition, the hierarchical model can process super-resolution image information and also reduces the consumption of computing resources. The model adds a time dimension T, which is used to capture the continuous time frame information of the incoming video, and expands the scope of the windowed attention from the spatial domain to the spatiotemporal domain. The model of the present invention surpasses the previous spatiotemporal decomposition model in terms of operating efficiency, because it utilizes the inherent spatiotemporal locality of the video, and pixels that are closer to each other in spatiotemporal distance are more likely to be associated with each other, and utilizes this assumption bias to achieve higher modeling efficiency. Since a non-overlapping window design is used, the original sliding window design is also redefined to conform to the information of the two dimensions of time and space. The present invention can effectively reduce the loss of information during network training and retain valid information. And without increasing the computational complexity, the model's ability to detect small objects is improved. The model is lightweight, the depth of the model is reduced, and the model performance is increased while the number of model parameters is reduced by half. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a flow chart of a video classification and recognition method based on Video Swin Transformer of the present invention;

[0027] Figure 2 It is a network structure diagram of the MSFF-VST model of the present invention;

[0028] Figure 3 is a network structure diagram of the downsampling module of the present invention;

[0029] Figure 4 It is a schematic diagram of the Patch Merging layer structure of the present invention;

[0030] Figure 5 is a schematic diagram of the structure of the multi-scale fusion module of the present invention;

[0031] Figure 6 It is a comparison chart of the accuracy of the model of the present invention;

[0032] Figure 7 It is a PR curve diagram of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0034] Reference Figure 1 , Figure 2 , Figure 3 , Figure 4 , Figure 5 , Figure 6 and Figure 7 The video classification and recognition method based on Video SwinTransformer shown in the figure comprises the following steps:

[0035] S100: Prepare a data set to be identified.

[0036] S200, preprocessing the data set, and entering feature map information into a network block, and performing a multi-scale fusion module on the feature map.

[0037] The S300,model uses the I3D classification head to perform the final classification,on 8 categories.

[0038] In this embodiment, model classification: The model uses the I3D classification head to perform final classification on 8 categories. The classification accuracy of the network model of the present invention reaches 87.19% in the prepared data set, which is 2.81% higher than the original model VST. Accuracy, precision, loss, F1-Score, mean average precision (mAP), Macro-F1 and Micro-F1 are all improved.

[0039] S400: Draw a confusion matrix according to the classification results of the model, and reclassify or adjust the classification boundaries of samples that are misclassified by the model.

[0040] In some specific embodiments, the specific method of step S100 is:

[0041] The dataset uses 8 types of action videos selected from Kinetic400 as the experimental dataset, with 300 30fps videos in each category and 40 30fps videos in the test set. The length of each video is about 10s, and the video is in mp4 format.

[0042] In some specific embodiments, the specific method of step S200 is:

[0043] First, the data set is preprocessed, a linear embedding layer is used using 3D convolution, and the feature map size of the model is adjusted; the feature map information enters four network blocks, each of which includes a self-attention module and a downsampling module.

[0044] The present invention is based on the improvement of Video Swin Transformer (VST) network. The overall network structure of Multi-scale Feature Fusion Video Swin Transformer (MSFF-VST) is as follows: Figure 2 As shown in the figure, the model is hierarchically designed, and BackBone is divided into four stages. The feature map size and the number of channels are continuously adjusted to extract information from multi-scale feature maps.

[0045] In some specific embodiments, the specific method of step S200 further includes:

[0046] The self-attention module includes a sliding window and a non-sliding window;

[0047] The self-attention module can effectively establish information connections between feature maps without increasing too much computing resources; the downsampling module uses large kernel convolution and Inverted Bottleneck architecture to reduce the size of the model feature map while adjusting the number of channels; the feature map multi-scale fusion module selects the feature map information of different sizes of the model for fusion so that the model can recognize objects of different sizes.

[0048] In some specific embodiments, the specific method of step S400 further includes:

[0049] The model accuracy, precision and recall rate indicators are calculated based on the confusion matrix. These indicators can reflect the performance of the model from different angles.

[0050] In some specific embodiments, the downsampling module is used to extract features of global information, reduce the size of the image feature map, filter out redundant feature information, and screen key information; and add multi-scale spatiotemporal feature information fusion to the network to improve the model's ability to detect small objects and enhance the model's feature fusion performance; finally, the model is simplified.

[0051] In some specific embodiments, a self-attention mechanism module is also included;

[0052] The self-attention mechanism module is used to encode the pixel-level information of all incoming data, and then pay attention to the relationship between their feature sequence information, so as to establish a network with associated global information. This mechanism model can learn the information between each element without the need for position information in the sequence. The input matrix is ​​passed through three different learnable weight matrices Calculate the vector

[0053]

[0054]

[0055]

[0056] in Index of attention block 2,...,L, L is the total number of attention blocks, LN is the layer normalization function Layernorm, z is the output feature map of the previous model. D is the number of hidden dimensions of the input sequence. a=1,2,...,A is the number of multi-head attention heads, and the dimension of the feature map processed by each attention head is D h =D / A.

[0057]

[0058] Self-attention is obtained by calculating the linear change of different attention weights They are the query matrix, key matrix and value matrix, which determine the weight distribution during parameter training. d is the dimension of query and key features. Each element in the formula is divided by This keeps the parameter gradient stable during self-attention calculation. B is the relative position bias.

[0059] The network structure of the down sampling module is as follows: Figure 3 , reducing the H and W of the feature map while increasing the number of feature maps. This structure conforms to the Inverted Bottleneck architecture and the information channel design that is wide in the middle and narrow at both ends. This can effectively reduce the loss of information during network training and retain more effective information. The Patch Merging layer in the model is as follows Figure 4 ,The function of this module is to perform downsampling, which is used to reduce the size of the feature map, adjust the number of channels, conform to the hierarchical design of the model, and reduce the amount of computation.

[0060] This model uses less regularization, which can avoid introducing prior knowledge and bias into the model as much as possible, while reducing the training time and computing resources of the network. Regularization uses layer normalization (LN) commonly used in Transformer. LN can normalize the hidden layer dimensions of all samples, reduce the differences between samples, and make the distribution of samples more stable. It also helps to transfer information between layers during model learning and reduce the differences in sample information transmission between layers.

[0061] The convolution layer design of this module is consistent with the design of large-core convolution. The size of the convolution kernel directly limits the size of the receptive field. Large-core convolution can cover more feature map information in one convolution operation because of its larger convolution kernel size. In this way, more upper and lower layer information can be obtained and global features can be extracted.

[0062] In some specific embodiments, a multi-scale feature information fusion module is also included;

[0063] The multi-scale feature information fusion module is used to address the multi-scale problem in object detection and improve the model's ability to detect small objects without substantially increasing computing resources.

[0064] The video classification network of the present invention covers two algorithms, namely, the optimized down-sampling module (Down Sample) and the multi-scale information fusion module (Multi Scale Feature Fusion). First, the pre-processed video frame feature map enters the optimized down-sampling module, which performs down-sampling processing on the feature map to reduce the feature map size and adjust the number of channels. The design conforms to the network hierarchical design, effectively reducing the number of parameters of the network operation, and adopts the large kernel convolution idea in the down-sampling module, using the Inverted Bottleneck architecture, increasing the feature information receptive field, and reducing the loss of information during the training process. Subsequently, the feature map enters the multi-scale feature information fusion module, which is mainly aimed at the multi-scale problem of Zhou Ang in object detection. Under the premise of not increasing the network computing resource consumption, the model's detection ability for small objects is improved.

[0065] In the feature integration stage, representative features are extracted from the optimized downsampling module and the multi-scale information fusion module to form a neutralized feature vector. This process aims to integrate different feature map information in the two dimensions of video time and space and capture its multi-scale information features. The final integrated feature vector is input into the classification head for effective classification of video feature map information. This comprehensive feature extraction method can not only reveal the spatiotemporal characteristics of the signal more comprehensively, but also provide more discerning input for the classification model, providing strong support for video classification research.

[0066] The public dataset Kinetic400 was selected, and multiple action categories were selected for experiments. Each classification experiment set had 300 30fps videos, and the test set had 40 30fps videos. Each video was about 10s long, and the video was in mp4 format. The dataset was divided into 8 categories, and a mapping table was established for the 8 categories using numbers 1-8.

[0067] The specific mappings are: {<1:bobsledding>, <2:bowling>, <3:bungee_jumping>, <4:capoeira>, <5:catching_fish>, <6:clean_and_jerk>, <7:eating_ice_cream>, <8:triple_jump>.

[0068] Accuracy, Precision, Loss, F1-Score, mean average precision (mAP), Macro-F1 and Micro-F1 are used as evaluation indicators. The experimental parameter settings are shown in Table 1.

[0069] Table 1 Experimental parameters

[0070]

[0071] The model is trained using the test set, and the precision, recall, and F1-score of each category are calculated as shown in Table 2.

[0072] Table 2 Performance comparison of each category

[0073]

[0074] In order to verify the performance of the MSFF-VST model proposed in this paper in the field of video classification, this paper conducts a comparative experiment with it and a common video classification model. This experiment is conducted in the same environment to reduce the impact of the experimental environment on the experiment. The accuracy of the MSFF-VST model is compared with the VST model and other classic models. Figure 6 , Overall, they all show a trend of first rising rapidly and then rising slowly. The overall increase of VST and MSFF-VST is better than that of other models, and the improved MSFF-VST performs better than VST.

[0075] The PR curve is used to analyze the relationship between precision and recall. Recall is the horizontal axis and precision is the vertical axis. The larger the area under the curve, the better the classification performance. Figure 7 PR curve, the area under the PR curve of the MSCFF-VST model is larger than that of other models, indicating that its image classification performance is stronger than that of the other two models. In addition, according to the intersection of the PR curve and the diagonal line, the balance point of the PR curve can be obtained. The larger the balance point value, the better the model performance. It can be seen that the balance point of the MSCFF-VST model is higher than that of other models, indicating that its performance is better.

[0076] As shown in Table 3, on the dataset used by the present invention, the Accurary index of the MSFF-VST model is improved by 3.13% compared with the VST model, indicating that the improved model proposed by the present invention has better effect in the video classification task. In addition, the model also achieved good results in the mAP, Macro-F1 and Micro-F1 evaluation indicators. This shows that the model proposed by the present invention can capture more effective feature information in the video classification task and has higher performance.

[0077] Table 3 Performance comparison of different models

[0078]

[0079] Comparing the parameter distribution of MSFF-VST and VST as shown in Table 4, it can be seen that the MSCFF-VST model is simplified compared with the original model VST, and the parameters are reduced in Stage 3 and Stage 4 of the model, which significantly reduces the Params and GFLOPs of the model.

[0080] Table 4 Model parameters

[0081]

[0082] By adopting the above technical solution disclosed in the present invention, the following beneficial effects are obtained:

[0083] The invention discloses a video classification and recognition method based on Video Swin Transformer, comprising the following steps: preparing a data set to be recognized; preprocessing the data set, and entering feature map information into a network block, and performing a multi-scale fusion module on the feature map; using an I3D classification head in the model to perform final classification on 8 categories; drawing a confusion matrix according to the classification result of the model, and reclassifying or adjusting the classification boundary for samples classified incorrectly by the model. In the downsampling module of the invention, the number of channels in the model increases first and then decreases, which conforms to the Inverted Bottleneck architecture and conforms to the information channel design with a wide middle and narrow ends of the channel number, which can effectively reduce the loss of information during network training and retain more effective information. While reducing the size of the feature map, the number of feature maps is increased, and the number of channels is adjusted, which conforms to the hierarchical design of the model and reduces the amount of calculation. The multi-scale convolution module extracts features in the hierarchical network to obtain multi-scale information. Feature maps of different levels are merged to identify targets of different sizes. By extracting information at different levels, a wider contextual relationship is obtained, global semantic information is obtained from the feature map of a deeper level, and more local information is obtained from the feature map of a shallower level. Global semantic and local feature information are obtained at the same time, and the image feature content is better understood. In addition, the multi-scale convolution module can effectively reduce the loss of feature information. As the number of network layers increases, the size of the feature map will gradually decrease, which may lead to the loss of feature information, especially for smaller targets and details. By fusing multi-scale feature information, the feature information in each layer of the hierarchical network can be preserved.

[0084] The model of the present invention adopts the sliding window design of Vdieo Swin Transformer, which enables the model to learn feature information across windows and greatly reduces the amount of parameters in the calculation. In addition, the hierarchical model can process super-resolution image information and also reduce the consumption of computing resources. The model adds a time dimension T, which is used to capture the continuous time frame information of the incoming video, and expands the scope of windowed attention from the spatial domain to the spatiotemporal domain. The model of the present invention surpasses the previous spatiotemporal decomposition model in terms of operating efficiency, because it utilizes the inherent spatiotemporal locality of the video, and pixels that are closer to each other in spatiotemporal distance are more likely to be associated with each other, and this assumption is used to achieve higher modeling efficiency. Due to the use of a non-overlapping windowed design, the original sliding window design is also redefined to conform to the information of the two dimensions of time and space.

[0085] The present invention can effectively reduce the loss of information during network training and retain valid information. It can also improve the model's ability to detect small objects without increasing computational complexity. The model is lightweight, the depth of the model is reduced, and the model performance is increased while the number of model parameters is reduced by half.

[0086] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A video classification and recognition method based on Video Swin Transformer, characterized in that: The following steps are involved: S100, preparing a data set to be identified; S200, preprocessing the data set and entering feature map information into a network block, and performing a multi-scale fusion module on the feature map; The S300,model uses the I 3D classification head to perform the final classification,on 8 categories; S400: Draw a confusion matrix according to the classification results of the model, and reclassify or adjust the classification boundaries of samples that are misclassified by the model.

2. The video classification and recognition method based on Video Swin Transformer according to claim 1, characterized in that: The specific method of step S100 is: The dataset uses 8 types of action videos selected from Kinetic400 as the experimental dataset, with 300 30fps videos in each category and 40 30fps videos in the test set. The length of each video is about 10s, and the video is in mp4 format.

3. The video classification and recognition method based on Video Swin Transformer according to claim 2, characterized in that: The specific method of step S200 is: First, the data set is preprocessed, a linear embedding layer is used using 3D convolution, and the feature map size of the model is adjusted; the feature map information enters four network blocks, each of which includes a self-attention module and a downsampling module.

4. The video classification and recognition method based on Video Swin Transformer according to claim 3 is characterized in that: The specific method of step S200 also includes: The self-attention module includes a sliding window and a non-sliding window; The self-attention module can effectively establish information connections between feature maps without increasing too much computing resources; the downsampling module uses large kernel convolution and Inverted Bottleneck architecture to reduce the size of the model feature map while adjusting the number of channels; the feature map multi-scale fusion module selects the feature map information of different sizes of the model for fusion so that the model can recognize objects of different sizes.

5. The video classification and recognition method based on Video Swin Transformer according to claim 4, characterized in that: The specific method of step S400 also includes: The model accuracy, precision and recall rate indicators are calculated based on the confusion matrix. These indicators can reflect the performance of the model from different angles.

6. The video classification and recognition method based on Video Swin Transformer according to claim 5, characterized in that: The downsampling module is used for feature extraction of global information, which reduces the size of the image feature map, filters out redundant feature information, and screens key information. Multi-scale spatiotemporal feature information fusion is added to the network to improve the model's ability to detect small objects and enhance the model's feature fusion performance. Finally, the model is simplified.

7. The video classification and recognition method based on Video Swin Transformer according to claim 6, characterized in that: Also includes: self-attention mechanism module; The self-attention mechanism module is used to encode the pixel-level information of all incoming data, and then focus on the relationship between their feature sequence information, so as to establish a network with associated global information.

8. The video classification and recognition method based on Video Swin Transformer according to claim 7, characterized in that: It also includes: a multi-scale feature information fusion module; The multi-scale feature information fusion module is used to address the multi-scale problem in object detection and improve the model's ability to detect small objects without substantially increasing computing resources.

Images