Video action classification method based on segmented contrastive mask autoencoder

Abstract

The present invention discloses a video action classification method based on a segmented contrastive masked autoencoder, which divides the video content into a global view and multiple local views, adopts a masked data enhancement technology, and utilizes a joint processing mechanism of a teacher model and a student model to capture the global discriminative features and local fine-grained features of the video; the segmented contrastive masked autoencoder reduces the dependence on labeled data and improves the training efficiency through self-supervised learning; the output features of the student model and the teacher model are aligned, which enhances the model's adaptability to input diversity and effectively integrates temporal information and local information into contrastive learning; in addition, the introduced masking strategy improves the model's fine-grained understanding ability of the video content; the present invention performs well in terms of training efficiency, only requires fewer training rounds to achieve excellent performance, and significantly improves the accuracy of video action classification.

Description

Technical Field

[0001] The invention belongs to the field of video understanding, and relates to a video action classification technology, and in particular to a video action classification method based on a segmented contrast mask autoencoder. Background Art

[0002] In the field of self-supervised learning (SSL), especially after combining with the Transformer model, video understanding tasks such as action recognition, action localization, and video retrieval have achieved significant performance improvements. Self-supervised video representation learning provides effective feature representations for these tasks by pre-training unlabeled video data. In order to further improve the model's understanding of the spatiotemporal information of video data, researchers are exploring more efficient self-supervised learning task designs.

[0003] Vision Transformer (ViT) has become a competitive model architecture since its success in image classification tasks. In order to find a balance between computational efficiency and accuracy, researchers began to explore ways to improve the computational efficiency of ViT. ViT has also been applied to video understanding tasks, where the self-attention mechanism in the spatial and temporal dimensions has been improved, such as TimeSformer, ViViT, Video Swin, and UniFormer. These models enhance ViT's ability to understand the spatiotemporal information of video data by introducing the processing of the temporal dimension.

[0004] Contrastive Learning (CL), as a method of SSL, learns data representation by distinguishing positive and negative sample pairs. In the field of images, contrastive learning has achieved remarkable results, such as MoCo and SimCLR. DINO is an innovative contrastive learning method that does not require negative samples. When used in combination with Vision Transformer (ViT), it can learn feature representations that contain object contour information, which is not available in traditional supervised learning methods. However, the application of contrastive learning in the field of video understanding faces specific challenges: video data has rich spatiotemporal information, which requires the model to capture more complex patterns. In addition, contrastive learning usually requires a large batch size, which is a challenge in terms of computing resource consumption. In addition, when contrastive learning is combined with ViT, the self-attention mechanism in ViT will lead to homogenization problems, that is, different query tokens and attention heads show the same self-attention distribution, which limits the diversity of network learning representations.

[0005] To address these challenges, researchers have proposed the SVT model, a contrastive learning method that does not require large batch sizes and negative samples. It creates local and global spatiotemporal views of different spatial sizes and frame rates to perform self-supervised pre-training to help ViT understand the contextual information in the video. However, contrastive learning combined with ViT's self-attention mechanism will lead to homogenization problems, that is, different query tokens and attention heads show the same self-attention distribution, which limits the diversity of network learning representations. This problem also occurs in the SVT model. Generative learning, as another method of SSL, is represented by masked visual modeling. It can effectively learn the local correlation information of video data by masking a part of the visual block and reconstructing its corresponding pixel values ​​or high-level features for self-supervised pre-training. VideoMAE is a typical self-supervised learning method based on masked visual modeling. By constraining pixel reconstruction at the patch level, it encourages each query token to pay attention to tokens in the surrounding related areas, effectively identifying the local related features of different regions, but it is insufficient in capturing the overall motion information of the video.

[0006] Nevertheless, existing methods still face challenges in the task of video action classification. Different attention heads in ViT often have the same attention pattern, which makes it difficult for the model to learn the differences between different objects in the video frame, limiting the video representation ability. Due to this homogeneity problem, the model cannot effectively capture the key information in the video, so the generated video features lack diversity, which ultimately affects the accuracy of video action classification. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a video action classification method based on a segmented contrast mask autoencoder, which effectively enhances the model's ability to learn the differences between different objects in video frames by introducing a segmented contrast mask mechanism, thereby improving the video representation capability and ultimately improving the accuracy of video action classification.

[0008] The technical solution adopted by the present invention to solve the above technical problems is: a video action classification method based on segmented contrast mask autoencoder, characterized by comprising the following steps:

[0009] Step 1: Get several original videos;

[0010] Step 2: Obtain a global view and n local views of each original video;

[0011] Step 3: Mask the global view of each original video to obtain the corresponding global mask view, the masked blocks are invisible blocks, and the unmasked blocks are visible blocks; similarly, mask the n local views of each original video to obtain the corresponding local mask views, the masked blocks are invisible blocks, and the unmasked blocks are visible blocks;

[0012] Step 4: construct a segmented contrastive mask autoencoder capable of simultaneously performing contrastive learning and mask reconstruction tasks, which includes a teacher model consisting of a teacher encoder and a first multi-layer perceptron, and two branches of a student model consisting of a student encoder, a second multi-layer perceptron and a decoder;

[0013] The global view of the original video is input into the teacher model, and each video frame in the global view is divided into several blocks; then all the blocks corresponding to the global view are added with a class token feature and input into the teacher encoder to obtain a class token feature corresponding to the global view and a token feature corresponding to each block in the global view; then the class token feature corresponding to the global view is input into the first multi-layer perceptron to obtain a global representation vector; and all the token features are used as high-level features output by the teacher model;

[0014] The global mask view corresponding to the global view of the original video and the local mask views corresponding to the n local views are input into the student model, and all visible blocks are extracted from each video frame in the global mask view and each local mask view; then all visible blocks are added with a class token feature and input into the student encoder to obtain a token feature corresponding to each visible block in the global mask view, a class token feature corresponding to each local mask view, and a token feature corresponding to each visible block in each local mask view; then the class token features corresponding to the n local mask views are input into the second multi-layer perceptron to obtain n local representation vectors; then, based on the token features corresponding to all visible blocks in the global mask view, randomly initialized token features are added and input into the decoder to reconstruct the contents of all invisible blocks in the global mask view;

[0015] Step 5: Based on the global view of the original video, the global mask view corresponding to the global view of the original video, and the local mask views corresponding to all local views of the original video, the segmented contrast mask autoencoder is self-supervised pre-trained. After each iteration, the global feature consistency contrast loss and the feature mask reconstruction loss are calculated for each related original video. The sum of the global feature consistency contrast loss and the feature mask reconstruction loss is then taken as the total loss, and the network parameters are updated according to the total loss.

[0016] Step 6: After the self-supervised pre-training is completed, the optimal weight of the teacher encoder is obtained; then combined with the linear classification layer, the teacher encoder under the optimal weight is fine-tuned or linearly detected; after the fine-tuning or linear detection is completed, the teacher encoder can output a token feature corresponding to the video, and use the token feature to classify the video action.

[0017] Further defined, in step 2, the process of obtaining a global view of an original video is:

[0018] Step 2.1a: Along the video timeline, extract the middle 90% of the video frames from the original video to form a new video;

[0019] Step 2.2a: extracting T1 frames of video frames from the new video in an equal step-size manner, and forming a video segment from the extracted T1 frames of video frames in index order, wherein T1 and the step-size are both preset values;

[0020] Step 2.3a: Unify the spatial resolution of the T1 frames in the video segment to 224×224, and then construct the global view of the original video with the T1 frames with a spatial resolution of 224×224.

[0021] Further defined, in step 2, the process of obtaining n partial views of an original video is:

[0022] Step 2.1b: Along the video timeline, extract 90% of the video frames in the middle from the original video to form a new video;

[0023] Step 2.2b: along the video time axis, divide the new video into n non-overlapping first video segments, where n=2, 4, 8, etc.;

[0024] Step 2.3b: extracting T2 frames of video frames from each first video segment in an equal step length manner, and forming a second video segment from the T2 frames of video frames extracted from each first video segment in index order, wherein T2 and the step length are both preset values;

[0025] Step 2.4b: Unify the spatial resolution of the T2 frame video frames in each second video segment to 224×224, and then construct a local view of the original video with the T2 frame video frames with a spatial resolution of 224×224.

[0026] It is further defined that in step 3, the process of masking the global view is: dividing each video frame in the global view into 14×14 non-overlapping blocks of size 16×16; and then performing tube masking on each video frame in the global view according to a preset mask rate to obtain a global mask view corresponding to the global view.

[0027] It is further defined that in step 3, the process of masking the i-th local view is: dividing each video frame in the i-th local view into 14×14 non-overlapping blocks of size 16×16, where i=1,2,…,n; and then performing block-level masking on each video frame in the i-th local view according to a preset mask rate to obtain a local mask view corresponding to the i-th local view.

[0028] It is further defined that in step 4, in the teacher model, the size of the blocks into which each video frame in the global view is divided is 16×16, the number of blocks is 14×14, and they do not overlap with each other. The global view corresponds to 14×14×T1 blocks in total, T1 represents the total number of video frames contained in the global view, and the dimension of the class token features on all blocks corresponding to the global view is 768; in the student model, the dimension of the class token features on all visible blocks is 768, the dimension of the randomly initialized token features is 768 and the number of added blocks is 14×14×T1-M, and M represents the number of all visible blocks in the global mask view.

[0029] Further, in step 5, the global feature consistency contrast loss is calculated using the cross entropy loss function, and the global feature consistency contrast loss is recorded as , ,in, represents the global representation vector, represents the i-th local representation vector, i=1,2,…,n; the feature mask reconstruction loss is calculated using the mean square error loss function, and the feature mask reconstruction loss is recorded as , ,in, represents the set of indices of all invisible blocks in the global mask view, represents the index of the invisible block in the global mask view, The index reconstructed by the decoder is The contents of the invisible block, The index in the high-level features representing the output of the teacher model is The token feature of the symbol " ” is the symbol for taking absolute value.

[0030] It is further defined that in step 5, for self-supervised pre-training, 80 epochs are set, the batch size is set to 128, the Adam optimizer is used, the weight decay in the Adam optimizer is set to 0.04 to 0.1, the initial learning rate is set to 5.0e-4, and the learning rate is adjusted according to the law of the cosine function during the training process, and the first 10 rounds are linearly warmed up. During the training process, the weights of the student encoder are updated by back propagation, and the weights of the teacher encoder are updated by the exponential moving average of the weights of the student encoder.

[0031] Further defining, in step 6, the process of fine-tuning the teacher encoder under the optimal weights in combination with the linear classification layer is:

[0032] Step 6.1a: Select multiple original videos and their action labels, each of which has a duration of 5 to 20 seconds;

[0033] Step 6.2a: Obtain a global view of each original video, wherein the spatial resolution of the video frame in the global view is 224×224;

[0034] Step 6.3a: Build a fine-tuner consisting of a teacher encoder and a linear classification layer with optimal weights.

[0035] Step 6.4a: A global view of an original video is input into the fine-tuner, and each video frame in the global view is divided into 14×14 non-overlapping blocks of size 16×16; then the 14×14×T3 blocks corresponding to the global view are added with a class token feature of dimension 768 and input into the teacher encoder, and the teacher encoder outputs a class token feature corresponding to the global view and a token feature corresponding to each block in the global view, where T3 represents the total number of video frames contained in the global view; then the token features corresponding to all blocks in the global view are input into the linear classification layer, and the linear classification layer outputs a feature vector;

[0036] Step 6.5a: Based on the global view of all original videos, the fine-tuner is trained. After each iteration, the cross entropy loss between the feature vector output by the linear classification layer and the action label is calculated for each relevant original video. The network parameters of the teacher encoder and the linear classification layer are updated according to the cross entropy loss.

[0037] Further defining, in step 6, the process of linear detection of the teacher encoder under the optimal weight in combination with the linear classification layer is:

[0038] Step 6.1b: Select multiple original videos and their action labels, each of which has a duration of 5 to 20 seconds;

[0039] Step 6.2b: Obtain a global view of each original video, wherein the spatial resolution of the video frame in the global view is 224×224;

[0040] Step 6.3b: Construct a linear detector consisting of a teacher encoder and a linear classification layer with optimal weights.

[0041] Step 6.4b: A global view of an original video is input into the linear detector, and each video frame in the global view is divided into 14×14 non-overlapping blocks of size 16×16; then the 14×14×T3 blocks corresponding to the global view are added with a class token feature of dimension 768 and input into the teacher encoder, and the teacher encoder outputs a class token feature corresponding to the global view and a token feature corresponding to each block in the global view, where T3 represents the total number of video frames contained in the global view; then the token features corresponding to all blocks in the global view are input into the linear classification layer, and the linear classification layer outputs a feature vector;

[0042] Step 6.5b: Based on the global view of all original videos, the linear detector is trained. After each iteration, the cross entropy loss between the feature vector output by the linear classification layer and the action label is calculated for each relevant original video. The network parameters of the linear classification layer are updated according to the cross entropy loss, while the network parameters of the teacher encoder are frozen.

[0043] Compared with the prior art, the advantages of the present invention are:

[0044] 1) By dividing the video content into a global view and multiple local views, a multi-scale representation of the video data is achieved, and the mask data enhancement technology is used to generate the corresponding global mask view and local mask view. The method of the present invention uses the joint processing mechanism of the teacher model and the student model: the teacher model focuses on extracting features from the global view, while the student model processes the local mask view and the global mask view. This process enables the method of the present invention to simultaneously capture the global discriminant features and local fine-grained feature information of the video content, increase the representation diversity of network learning, and effectively overcome the limitations of the existing technology that only relies on local features or global features; the fusion of global discriminant features and local fine-grained features helps to understand the video content more accurately, thereby improving the accuracy of video action classification.

[0045] 2) The method of the present invention introduces a mask data enhancement strategy to mask a part of the visual blocks in the global view and the local view. Under this strategy, the global view and the local view generate corresponding mask views after mask processing, in which a part of the blocks (patch) are set to be invisible. This process not only provides key data enhancement for the mask reconstruction task, but also, as an efficient contrast learning strategy, forces the model to learn more detailed and discriminative feature representations in order to accurately predict the content of the masked part. In this way, the model is forced to focus on those unmasked blocks, thereby improving the fine-grained understanding of the video content and further improving the accuracy of video action classification.

[0046] 3) The method of the present invention significantly improves the ability of video action recognition by constructing a segmented contrastive masked autoencoder (SCMA). SCMA adopts the architecture of a student model and a teacher model. The output features of the student model are aligned with the global features of the teacher model. This feature alignment strategy not only improves the training efficiency, but also enhances the model's adaptability to input diversity. Through this alignment, SCMA can effectively integrate temporal information and local information into contrastive learning, forcing the model to learn features that can characterize the global structure and local details of the video. At the same time, it solves the homogenization problem caused by the combination of contrastive learning and ViT and the self-attention mechanism in ViT, further improving the accuracy of video action classification.

[0047] 4) In the method of the present invention, SCMA reduces the dependence on labeled data and improves training efficiency through self-supervised learning. Since the model can learn richer and more robust feature representations, it shows excellent performance in tasks such as video action recognition. Through the masking strategy, the model is forced to focus on the unmasked parts, thereby improving the fine-grained understanding of the video content. Ultimately, these improvements together contribute to the rapid convergence and performance improvement of the model, achieving a significant enhancement in the ability to recognize video action.

[0048] 5) The proposed method has excellent training efficiency, requiring only one-tenth of the training rounds of the masked autoencoder method to achieve or even exceed its performance in video action classification tasks. This significant improvement is due to multiple improvements in contrastive learning, including the optimization of the self-attention mechanism when contrastive learning is combined with ViT, and the introduction of new data augmentation strategies such as masking partial visual blocks and video average segmentation. These improvements together contribute to the rapid convergence and performance improvement of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 The overall implementation process of the method of the present invention is as follows;

[0050] Figure 2For simple and complex scenes, three models pre-trained on the K400 dataset: SVT, VideoMAE, SCMA, self-attention map visualization of the class token in the last self-attention layer and the query token corresponding to part of the original video frame content. DETAILED DESCRIPTION

[0051] The present invention is further described in detail below with reference to the accompanying drawings.

[0052] The present invention proposes a video action classification method based on segmented contrast mask autoencoder, and its overall implementation process is as follows: Figure 1 As shown, it includes the following steps:

[0053] Step 1: Obtain several original videos, each of which is 5 to 20 seconds long.

[0054] Step 2: Obtain a global view g and n local views of each original video, where n is a multiple of 2, and the spatial resolution of the video frames in the global view g and the local views is 224×224.

[0055] In this embodiment, in step 2, the process of obtaining a global view of an original video is:

[0056] Step 2.1a: Along the video timeline, extract the middle 90% of the video frames from the original video to form a new video. Here, generally, 5% of the video frames at the head and tail of the original video are discarded because the actions in these video frames are not important.

[0057] Step 2.2a: Extract T1 video frames from the new video in an equal step manner, and form a video segment with the extracted T1 video frames in index order, wherein T1 and the step size are both preset values, usually a multiple of 2. In the specific implementation, T1=16 and the step size is set to 32 frames.

[0058] Here, the video segment acquisition can adopt the following specific process: randomly extract a video frame as the starting frame within the first 1 second of the new video; then start from the starting frame, extract video frames in an equal step manner, and determine whether a total of T1 video frames including the starting frame have been extracted from the new video. If so, the extracted T1 video frames constitute a video segment; otherwise, recycle and extract video frames from the new video until a total of T1 video frames are extracted, and the extracted T1 video frames constitute a video segment. Usually, when the new video is relatively long, the T1 video frame can be directly extracted from the new video. If the new video is relatively short, the T1 video frame cannot be extracted from the new video and can only be extracted cyclically.

[0059] Step 2.3a: Unify the spatial resolution of the T1 frames in the video segment to 224×224, and then construct the global view of the original video with the T1 frames with a spatial resolution of 224×224.

[0060] In this embodiment, in step 2, the process of obtaining n partial views of an original video is as follows:

[0061] Step 2.1b: Along the video timeline, extract the middle 90% of the video frames from the original video to form a new video. Here, 5% of the video frames at the head and tail of the original video are generally discarded because the actions in these video frames are not important.

[0062] Step 2.2b: Divide the new video into n non-overlapping first video segments along the video timeline, where n=2, 4, 8, etc. Through a large number of experiments, it is found that the final action classification effect is best when n=8.

[0063] Step 2.3b: Extract T2 frame video frames from each first video segment in an equal step size manner, and form a second video segment in index order using the T2 frame video frames extracted from each first video segment, wherein T2 and the step size are both preset values, usually a multiple of 2. In specific implementation, T2=8 is taken, and the step size is set to 8 frames.

[0064] Here, the second video segment can be obtained by the following specific process: randomly extract a video frame as the starting frame within the first 1 to 2 seconds of the first video segment; then start from the starting frame, extract video frames in an equal step manner, and determine whether a total of T2 video frames including the starting frame have been extracted from the first video segment. If so, the extracted T2 video frames constitute a second video segment; otherwise, recycle and extract video frames from the first video segment until a total of T2 video frames are extracted, and the extracted T2 video frames constitute a second video segment. Usually, when the first video segment is relatively long, the T2 video frames can be directly extracted from the first video segment. If the first video segment is relatively short, the T2 video frames cannot be extracted from the first video segment and can only be extracted cyclically.

[0065] Step 2.4b: Unify the spatial resolution of the T2 frame video frames in each second video segment to 224×224, and then construct a local view of the original video with the T2 frame video frames with a spatial resolution of 224×224.

[0066] Step 3: Mask the global view g of each original video to obtain the global mask view corresponding to the global view g , where the masked blocks in the global mask view are invisible blocks, and the unmasked blocks are visible blocks; similarly, the n local views of each original video are masked respectively to obtain the local mask views corresponding to each local view. The local mask views corresponding to the n local views are L1,…,L n , where the masked blocks in the local view are invisible blocks, and the unmasked blocks are visible blocks.

[0067] In this embodiment, the process of masking the global view is as follows: each video frame in the global view is divided into 14×14 non-overlapping patches of 16×16; and then tube masking is performed on each video frame in the global view according to a preset masking rate to obtain a global mask view corresponding to the global view. Here, the preset masking rate is set to 80%, that is, only 20% of the patches are visible.

[0068] Since there is a large amount of temporal redundant information in video data, it is unnecessary to input the complete video data into the model for training. Therefore, the process of obtaining the local view adopts a segmented mask sampling strategy. On the one hand, it can greatly increase the amount of data without increasing the amount of calculation too much, thereby improving the learning efficiency of the model. On the other hand, it increases the diversity of input. The model needs to find meaningful information in limited spatiotemporal data, so that it can explore key patterns and structures in a large amount of unlabeled data, thereby improving the adaptability of the model to input changes, so that it can extract rich discriminant information from complex visual environments. In this embodiment, the process of masking the i-th local view is: divide each video frame in the i-th local view into 14×14 non-overlapping blocks (patch) of size 16×16, where i=1,2,…,n; then perform block-level masking on each video frame in the i-th local view according to the preset mask rate to obtain the local mask view corresponding to the i-th local view. Here, the preset mask rate is set to 80%, that is, only 20% of the blocks are visible.

[0069] Step 4: Construct a segmented contrastive mask autoencoder (SCMA) that can simultaneously perform contrastive learning and mask reconstruction tasks. It includes two branches: a teacher model and a student model. Both the teacher model and the student model are based on the ViT-Base structure. The teacher model consists of a teacher encoder and the first multi-layer perceptron (MLP Head), and the student model consists of a student encoder, a second multi-layer perceptron (MLP Head) and a decoder.

[0070] The global view of an original video is input into the teacher model, and each frame of the video in the global view is divided into 14×14 non-overlapping patches of size 16×16; then the 14×14×T1 patches corresponding to the global view are added with a class token feature of dimension 768 (16×16×3 channels) and input into the teacher encoder. The teacher encoder outputs a class token feature corresponding to the global view and a token feature corresponding to each block in the global view, where T1 represents the total number of video frames contained in the global view; then the classtoken feature corresponding to the global view is Input into the first multilayer perceptron, the first multilayer perceptron outputs the global representation vector ; and use 14×14×T1 token features as high-level features output by the teacher model Here, in order to prevent the collapse of the teacher model, a multi-layer perceptron is added after the teacher encoder to output a more comprehensive and accurate global representation vector; the teacher encoder processes a maximum of T1=16 time tokens and a maximum of 14×14=196 spatial tokens per frame; the teacher model inputs a more complete global video data, i.e., a global view, and can output a stable and accurate video representation, which is used as the learning goal of the student model.

[0071] The global mask view corresponding to the global view of an original video and the local mask views corresponding to n local views are input into the student model, all visible blocks are extracted from each video frame in the global mask view, and all visible blocks are extracted from each video frame in the i-th local mask view, where i=1,2,…,n; then all visible blocks in the global mask view and all visible blocks in each local mask view are input into the student encoder with a classtoken feature of dimension 768, and the student encoder outputs a token feature corresponding to each visible block in the global mask view, a class token feature corresponding to each local mask view, and a token feature corresponding to each visible block in each local mask view; then the class token features corresponding to the n local mask views, that is, a total of n class token features, are added. Input into the second multilayer perceptron, the second multilayer perceptron outputs n local representation vectors, which are ; Then the token features corresponding to all visible blocks in the global mask view On the basis of , add 14×14×T1-M randomly initialized token features with a dimension of 768 Then it is input into the decoder, which reconstructs the contents of all invisible blocks in the global mask view. , where M represents the number of all visible blocks in the global mask view. Here, the student model inputs the global mask view and the local mask view, and specifically inputs the unmasked blocks in the global mask view and the local mask view, i.e., the visible blocks.

[0072] The class token features are used to impose constraints on the global view and the local mask view. The goal is to enable the student model to infer the global semantic information of the video from shorter spatiotemporal views. At the feature level, this is manifested as constraining the local view and the global view to be projected into the same latent space.

[0073] Constraints are imposed at the token level to construct spatiotemporal local correlation information in the video and enhance the diversity of representations learned by contrastive learning.

[0074] Step 5: Based on the global views of all original videos, the global mask views corresponding to the global views of all original videos, and the local mask views corresponding to all local views of all original videos, the segmented contrast mask autoencoder is self-supervised pre-trained. After each iteration, the global feature consistency contrast loss and the feature mask reconstruction loss are calculated for each relevant original video. The sum of the global feature consistency contrast loss and the feature mask reconstruction loss is taken as the total loss, and the network parameters are updated according to the total loss.

[0075] The training process of SCMA consists of two self-supervised tasks. The first is to compare the global feature consistency of the local representation vector output by the student model with the global representation vector output by the teacher model in the potential feature space. The second is to perform feature mask reconstruction on the global mask view to enhance the model's ability to capture local correlation information. Specifically, the teacher model processes the complete global view, and the student model processes the global mask view and n local mask views. First, for the contrastive learning task, the n local representation vectors output by the student model are respectively aligned with the global representation vector output by the teacher model, so that the model learns the global context information in the video. The mask reconstruction task is to predict the high-level features output by the teacher model through the decoder through the token features output by the student encoder and the masked token features, so as to learn the local correlation information in the video.

[0076] Further limiting step 5, the global feature consistency contrast loss is calculated using the cross entropy loss function, and the global feature consistency contrast loss is recorded as , ,in, represents the global representation vector, represents the i-th local representation vector, i=1,2,…,n; the feature mask reconstruction loss is calculated using the mean square error loss function, and the feature mask reconstruction loss is recorded as , ,in, represents the set of indices of all invisible blocks in the global mask view, represents the index of the invisible block in the global mask view, The index reconstructed by the decoder is The contents of the invisible block, The index in the high-level features representing the output of the teacher model is The token feature of the symbol " ” is the symbol for taking absolute value.

[0077] Further limit step 5, for self-supervised pre-training, set 80 epochs, the batch size is set to 128, use the Adam optimizer, the weight decay in the Adam optimizer is set to 0.04 to 0.1, the initial learning rate is set to 5.0e-4, and the learning rate is adjusted according to the law of the cosine function during training, and linear warm-up is performed in the first 10 rounds. During the training process, the weights of the student encoder are updated through back propagation, and the weights of the teacher encoder are updated through the exponential moving average (EMA) of the weights of the student encoder.

[0078] Step 6: After the self-supervised pre-training of the segmented contrast mask autoencoder is completed, the optimal weight of the teacher encoder is obtained. The teacher encoder under the optimal weight has learned the general feature representation of the video data; then, combined with the linear classification layer, the teacher encoder under the optimal weight is fine-tuned or linearly detected; after the fine-tuning or linear detection is completed, the teacher encoder can output a token feature corresponding to a video, and the token feature is used to classify the video action, that is, in the test phase, for a test video, a global view of the test video is obtained, the spatial resolution of the video frame in the global view is 224×224, each frame of the video frame in the global view is divided into 14×14 non-overlapping blocks (patch) of size 16×16, all blocks corresponding to the global view are added with a class token feature of dimension 768 (16×16×3 channels) and then input into the teacher encoder after fine-tuning or linear detection, the teacher encoder outputs a class token feature corresponding to the global view and a token feature corresponding to each block in the global view, and these token features are used to classify the video action.

[0079] In this embodiment, in step 6, the process of fine-tuning the teacher encoder under the optimal weight in combination with the linear classification layer is:

[0080] Step 6.1a: Select multiple original videos and their action labels, each of which has a duration of 5 to 20 seconds.

[0081] Step 6.2a: Obtain a global view of each original video, wherein the spatial resolution of the video frame in the global view is 224 × 224. Here, the global view is obtained using the same process as steps 2.1a to 2.3a.

[0082] Step 6.3a: Build a fine-tuner consisting of the teacher encoder and linear classification layer with optimal weights.

[0083] Step 6.4a: A global view of the original video is input into the fine-tuner, and each frame of the video in the global view is divided into 14×14 non-overlapping patches of size 16×16; then the 14×14×T3 patches corresponding to the global view are added with a class token feature of dimension 768 (16×16×3 channels) and input into the teacher encoder, which outputs a class token feature corresponding to the global view and a token feature corresponding to each patch in the global view, where T3 represents the total number of video frames contained in the global view; then the token features corresponding to all patches in the global view are input into the linear classification layer, which outputs a feature vector.

[0084] Step 6.5a: Based on the global view of all original videos, the fine-tuner is trained. After each iteration, the cross entropy loss between the feature vector output by the linear classification layer and the action label is calculated for each relevant original video. The network parameters of the teacher encoder and the linear classification layer are updated according to the cross entropy loss. The training adopts the stochastic gradient descent (SGD) optimizer. The weight decay in the stochastic gradient descent optimizer is set to 1e-4, the initial learning rate is set to 5e-3, and the momentum is set to 0.9.

[0085] In this embodiment, in step 6, the process of linear detection of the teacher encoder under the optimal weight is combined with the linear classification layer as follows:

[0086] Step 6.1b: Select multiple original videos and their action labels, each of which has a duration of 5 to 20 seconds.

[0087] Step 6.2b: Obtain a global view of each original video, wherein the spatial resolution of the video frame in the global view is 224 × 224. Here, the global view is obtained using the same process as steps 2.1a to 2.3a.

[0088] Step 6.3b: Construct a linear detector consisting of a teacher encoder and a linear classification layer with optimal weights.

[0089] Step 6.4b: A global view of an original video is input into the linear detector, and each frame of the video in the global view is divided into 14×14 non-overlapping patches of size 16×16; then the 14×14×T3 patches corresponding to the global view are added with a class token feature of dimension 768 (16×16×3 channels) and input into the teacher encoder. The teacher encoder outputs a class token feature corresponding to the global view and a token feature corresponding to each block in the global view, where T3 represents the total number of video frames contained in the global view; then the token features corresponding to all blocks in the global view are input into the linear classification layer, and the linear classification layer outputs a feature vector.

[0090] Step 6.5b: Based on the global view of all original videos, the linear detector is trained. After each iteration, the cross entropy loss between the feature vector output by the linear classification layer and the action label is calculated for each relevant original video. The network parameters of the linear classification layer are updated according to the cross entropy loss, while the network parameters of the teacher encoder are frozen. The training uses a stochastic gradient descent (SGD) optimizer, with the initial learning rate set to 5e-3 and the momentum set to 0.9.

[0091] Compared with fine-tuning, in the linear detection task, the backbone network, i.e., the teacher encoder, is frozen, and only the final linear classification layer participates in the training. This setting can better highlight the generalization ability of the representation learned by the pre-trained model.

[0092] In the present invention, class token can be translated into category token, and token can be translated into token.

[0093] In order to further illustrate the feasibility and effectiveness of the method of the present invention, a simulation experiment was carried out on the method of the present invention.

[0094] In the simulation experiment, the Segmented Contrastive Mask Autoencoder (SCMA) was pre-trained in a self-supervised manner on the K400 dataset. The pre-training process was trained for 80 epochs on 32 Ascend-910 B NPUs with a batch size of 128. The attention weights related to the time dimension in the backbone network were randomly initialized, while the attention weights related to the spatial dimension were derived from the Vision Transformer (ViT) model that completed self-supervised training on the ImageNet-1k dataset. The Adam optimizer was used, and the weight decay in the Adam optimizer was set to 0.04 to 0.1. The initial learning rate was set to 5.0e-4. During the training process, the learning rate was adjusted according to the law of the cosine function and linear warm-up was performed in the first 10 rounds. During the training process, the weights of the student encoder were updated by back-propagation, and the weights of the teacher encoder were updated by the exponential moving average (EMA) of the weights of the student encoder. Note that in all ablation experiments, in order to speed up the experiments, a subset of the K400 (Kinetics-400) dataset (consisting of 60k videos) is used and trained for 20 epochs.

[0095] In the simulation experiment, fine-tuning training was performed on the UCF-101 dataset and the HMDB-51 dataset, with a batch size of 32 and 15 epochs; training was performed on the K400 dataset and the SSv2 dataset, with a batch size of 128 and 30 epochs. The training used the stochastic gradient descent (SGD) optimizer, in which the weight decay was set to 1e-4, the initial learning rate was set to 5e-3, and the momentum was set to 0.9.

[0096] In the simulation experiment, linear detection training was performed on the UCF-101 dataset and the HMDB-51 dataset, with a batch size of 32 and 20 epochs; training was performed on the K400 dataset and the SSv2 dataset, with a batch size of 128 and 30 epochs. The training used a stochastic gradient descent (SGD) optimizer, in which the weight decay was set to 1e-4, the initial learning rate was set to 5e-3, and the momentum was set to 0.9.

[0097] In the simulation experiment, the methods used for comparison include methods based on contrastive learning: MCN method, Vi2CLR method, MoDist method, CORP method, CVRL method, and methods based on masked autoencoders: VIMPAC method, SVT method, VideoMAE method, BEVT method, and OmniMAE method.

[0098] Table 1 shows the Top-1 accuracy (%) of the proposed method and the existing methods when pre-trained on the K400 dataset and fine-tuned or linearly detected on the UCF-101 and HMDB-51 datasets.

[0099] Table 1 Top-1 accuracy (%) of the proposed method and existing methods after pre-training on the K400 dataset and fine-tuning or linear detection on the UCF-101 and HMDB-51 datasets

[0100]

[0101] In Table 1, the MoDist method uses additional optical flow input. It is represented as the reproduction result of the method of the present invention. As can be seen from Table 1, the method of the present invention has reached the most advanced level in linear detection evaluation on both UCF-101 and HMDB-51 datasets, and has shown competitive results in fine-tuning. The result of fine-tuning significantly exceeds the SVT method (baseline method) and is comparable to the performance of the VideoMAE method. It should be emphasized that in the pre-training stage, compared with the VideoMAE method using batch size=1024 for 800 rounds of training, the method of the present invention only used batch size=128 and conducted 80 rounds of training; in the fine-tuning stage, the VideoMAE method trained for 100 rounds, while the method of the present invention only trained for 15 rounds. The fast convergence performance of SCMA benefits from the powerful generalization ability of the method of the present invention. In addition, in terms of the performance of linear detection, the method of the present invention far exceeds the VideoMAE method, which fully proves that the quality of the representation learned by SCMA is better than other methods.

[0102] Table 2 shows the Top-1 accuracy (%) of the method of the present invention and the existing methods when pre-training is performed on the K400 dataset and fine-tuning or linear detection is performed on the validation set of the K400 dataset.

[0103] Table 2 Top-1 accuracy (%) of the proposed method and existing methods after pre-training on the K400 dataset and fine-tuning or linear detection on the validation set of the K400 dataset

[0104]

[0105] As can be seen from Table 2, for fine-tuning, the proposed method outperforms the existing methods, improving by 2.5% over the SVT method and by more than 0.6% over the VideoMAE method trained for 800 rounds, approaching the results of OmniMAE. It is worth noting that compared with the OmniMAE method, the proposed method uses fewer frames and fewer training rounds. For linear detection, the proposed method achieves the state-of-the-art level.

[0106] In order to verify the recognition effect of the method of the present invention on the action classification dataset that is strongly correlated with time, the method of the present invention is compared with the existing methods on another larger-scale dataset SSv2. Table 3 shows the Top-1 accuracy (%) of the method of the present invention and the existing methods after pre-training on the K400 dataset and fine-tuning or linear detection on the validation set of the SSv2 dataset.

[0107] Table 3 Top-1 accuracy of the proposed method and existing methods after pre-training on the K400 dataset and fine-tuning or linear detection on the validation set of the SSv2 dataset (%)

[0108]

[0109] It can be seen from Table 3 that for fine-tuning evaluation, the method of the present invention is improved by 3.2% compared with the SVT method (baseline method); for linear detection evaluation, the method of the present invention is improved by 6.6% compared with the SVT method (baseline method).

[0110] In order to make a more intuitive analysis, the self-attention maps of the class token in the last self-attention layer and the query token corresponding to the original video frame content of the three models pre-trained on the K400 dataset: SVT, VideoMAE, and SCMA are visualized for simple and complex scenes, respectively. Figure 2 Note that since the VideoMAE model does not contain a class token, the self-attention map of the query token corresponding to the original video frame content before and after fine-tuning the VideoMAE model is visualized. Figure 2 In the figure, part (a) shows the visualization effect in a simple scene, and part (b) shows the visualization effect in a complex scene. Figure 2In parts (a) and (b) of the figure, the first column shows two frames of original video frames, in which the red box marks the query token to be visualized; the second column shows the class token corresponding to the two frames of original video frames when the SVT model is used. The class token represents the global feature of the video frame. The results of different heads are omitted here because they show consistent results; the third column shows the self-attention map of the query token corresponding to the red box part in the two frames of original video frames when the SVT model is used; the fourth column shows the self-attention map of the query token corresponding to the red box part in the two frames of original video frames before fine-tuning using the VideoMAE model; the fifth column shows the self-attention map of the query token corresponding to the red box part in the two frames of original video frames after fine-tuning using the VideoMAE model; the sixth column shows the class token corresponding to the two frames of original video frames when the SCMA model in the method of the present invention is used; the seventh column shows the self-attention map of the query token corresponding to the red box part in the two frames of original video frames when the SCMA model in the method of the present invention is used.

[0111] When using the SVT model, in simple scenarios, the class token related to global information can focus on the action area of ​​the video (see the first row, second column and the second row, second column in part (a)), but the query token (sky) related to the background still focuses on dynamic objects (see the second row, third column in part (a)), indicating that it lacks diversity in representation learning; in complex scenarios, the self-attention of the query token (baffle) related to the background mixes the human action and background information (see the first row, third column in part (b)), and the self-attention map of the query token (floor) related to the background is close to that of the class token (see the second row, third column in part (b)). For downstream tasks, this will cause confusion in the model's judgment and affect its performance.

[0112] The VideoMAE model implements pixel reconstruction constraints at the token level, which can guide the query token to focus on the local area related to it in both simple and complex scenes (see the 4th column of parts (a) and (b)). However, its semantic information is weak, and only after end-to-end fine-tuning can it effectively focus on the character movements in the video (see the 5th column of parts (a) and (b)). This also provides a reasonable explanation for the poor performance of the masked autoencoder-based model in linear detection performance.

[0113] In contrast, whether in simple or complex scenes, the SCMA model can capture character action information on the class token only after pre-training (see the sixth column of parts (a) and (b)), and learn local features related to it in the query tokens corresponding to the red box part (see the seventh column of parts (a) and (b)). For example, the query related to the sky in the second row and sixth column of part (a) in simple scenes can effectively focus on the sky information, and the query tokens related to the baffle and the floor in the seventh column of parts (a) and (b) in complex scenes can respectively focus on the information related to them, which verifies that the method of the present invention effectively solves the homogenization problem of contrastive learning and significantly enhances the diversity of representations learned by the model.

Claims

1. A video action classification method based on segmented contrast mask autoencoder, characterized in that The following steps are involved: Step 1: Get several original videos; Step 2: Obtain a global view and n local views of each original video; wherein the global view is obtained as follows: intercept the video frames of the middle part from the original video to form a new video, extract T1 frame video frames from the new video to form a video segment, and unify the spatial resolution of the video frames in the video segment to form a global view; the local view is obtained as follows: intercept the video frames of the middle part from the original video to form a new video, divide the new video into n non-overlapping first video segments, extract T2 frame video frames from each first video segment to form a second video segment, and unify the spatial resolution of the video frames in each second video segment to form a local view; Step 3: Mask the global view of each original video to obtain the corresponding global mask view, the masked blocks are invisible blocks, and the unmasked blocks are visible blocks; similarly, mask the n local views of each original video to obtain the corresponding local mask views, the masked blocks are invisible blocks, and the unmasked blocks are visible blocks; Step 4: construct a segmented contrastive mask autoencoder capable of simultaneously performing contrastive learning and mask reconstruction tasks, which includes a teacher model consisting of a teacher encoder and a first multi-layer perceptron, and two branches of a student model consisting of a student encoder, a second multi-layer perceptron and a decoder; The global view of the original video is input into the teacher model, and each video frame in the global view is divided into several blocks; then all the blocks corresponding to the global view are added with a class token feature and input into the teacher encoder to obtain a class token feature corresponding to the global view and a token feature corresponding to each block in the global view; then the class token feature corresponding to the global view is input into the first multi-layer perceptron to obtain a global representation vector; and all the token features are used as high-level features output by the teacher model; The global mask view corresponding to the global view of the original video and the local mask views corresponding to the n local views are input into the student model, and all visible blocks are extracted from each video frame in the global mask view and each local mask view; then all visible blocks are added with a class token feature and input into the student encoder to obtain a token feature corresponding to each visible block in the global mask view, a class token feature corresponding to each local mask view, and a token feature corresponding to each visible block in each local mask view; then the class token features corresponding to the n local mask views are input into the second multi-layer perceptron to obtain n local representation vectors; then, based on the token features corresponding to all visible blocks in the global mask view, randomly initialized token features are added and input into the decoder to reconstruct the contents of all invisible blocks in the global mask view; Step 5: Based on the global view of the original video, the global mask view corresponding to the global view of the original video, and the local mask views corresponding to all local views of the original video, the segmented contrast mask autoencoder is self-supervised pre-trained. After each iteration, the global feature consistency contrast loss and the feature mask reconstruction loss are calculated for each related original video. The sum of the global feature consistency contrast loss and the feature mask reconstruction loss is then taken as the total loss, and the network parameters are updated according to the total loss. In step 5, the global feature consistency contrast loss is calculated using the cross entropy loss function, and the global feature consistency contrast loss is recorded as , ,in, represents the global representation vector, represents the i-th local representation vector; the feature mask reconstruction loss is calculated using the mean square error loss function, and the feature mask reconstruction loss is recorded as , ,in, represents the set of indices of all invisible blocks in the global mask view, represents the index of the invisible block in the global mask view, The index reconstructed by the decoder is The contents of the invisible block, The index in the high-level features representing the output of the teacher model is Token features, symbol" ” is the symbol for taking absolute value; Step 6: After the self-supervised pre-training is completed, the optimal weight of the teacher encoder is obtained; then combined with the linear classification layer, the teacher encoder under the optimal weight is fine-tuned or linearly detected; after the fine-tuning or linear detection is completed, the teacher encoder can output a token feature corresponding to the video, and use the token feature to classify the video action.

2. The video action classification method based on segmented contrast mask autoencoder according to claim 1 is characterized in that In step 2, the process of obtaining a global view of an original video is as follows: Step 2.1a: Along the video timeline, extract 90% of the video frames in the middle from the original video to form a new video; Step 2.2a: extracting T1 frames of video frames from the new video in an equal step-size manner, and forming a video segment from the extracted T1 frames of video frames in index order, wherein T1 and the step-size are both preset values; Step 2.3a: Unify the spatial resolution of the T1 frames in the video segment to 224×224, and then construct the global view of the original video with the T1 frames with a spatial resolution of 224×224.

3. The video action classification method based on segmented contrast mask autoencoder according to claim 1 is characterized in that In step 2, the process of obtaining n partial views of an original video is as follows: Step 2.1b: Along the video timeline, extract 90% of the video frames in the middle from the original video to form a new video; Step 2.2b: along the video time axis, divide the new video into n non-overlapping first video segments, where n=2, 4, 8, etc.; Step 2.3b: extracting T2 frames of video frames from each first video segment in an equal step length manner, and forming a second video segment from the T2 frames of video frames extracted from each first video segment in index order, wherein T2 and the step length are both preset values; Step 2.4b: Unify the spatial resolution of the T2 frame video frames in each second video segment to 224×224, and then construct a local view of the original video with the T2 frame video frames with a spatial resolution of 224×224.

4. The video action classification method based on segmented contrast mask autoencoder according to claim 1 is characterized in that In step 3, the process of masking the global view is as follows: each video frame in the global view is divided into 14×14 non-overlapping blocks of 16×16 in size; and then tube masking is performed on each video frame in the global view according to a preset mask rate to obtain a global mask view corresponding to the global view.

5. The video action classification method based on segmented contrast mask autoencoder according to claim 1 is characterized in that In step 3, the process of masking the i-th local view is as follows: each video frame in the i-th local view is divided into 14×14 non-overlapping blocks of size 16×16, where i=1,2,…,n; and then block-level masking is performed on each video frame in the i-th local view according to a preset mask rate to obtain a local mask view corresponding to the i-th local view.

6. The video action classification method based on segmented contrast mask autoencoder according to claim 1 is characterized in that In step 4, in the teacher model, each video frame in the global view is divided into blocks of size 16×16, the number of blocks is 14×14, and they do not overlap with each other. The global view corresponds to 14×14×T1 blocks in total, T1 represents the total number of video frames contained in the global view, and the dimension of the class token features on all blocks corresponding to the global view is 768; in the student model, the dimension of the class token features on all visible blocks is 768, the dimension of the randomly initialized token features is 768, and the number of added blocks is 14×14×T1-M, where M represents the number of all visible blocks in the global mask view.

7. The video action classification method based on segmented contrast mask autoencoder according to claim 1 is characterized in that In the step 5, for self-supervised pre-training, 80 epochs are set, the batch size is set to 128, the Adam optimizer is used, the weight decay in the Adam optimizer is set to 0.04 to 0.1, the initial learning rate is set to 5.0e-4, and the learning rate is adjusted according to the law of the cosine function during the training process, and the first 10 rounds are linearly preheated. During the training process, the weights of the student encoder are updated by back propagation, and the weights of the teacher encoder are updated by the exponential moving average of the weights of the student encoder.

8. The video action classification method based on segmented contrast mask autoencoder according to claim 2 is characterized in that In step 6, the process of fine-tuning the teacher encoder under the optimal weight in combination with the linear classification layer is as follows: Step 6.1a: Select multiple original videos and their action labels, each of which has a duration of 5 to 20 seconds; Step 6.2a: Obtain a global view of each original video, wherein the spatial resolution of the video frame in the global view is 224×224; Step 6.3a: Build a fine-tuner consisting of a teacher encoder and a linear classification layer with optimal weights. Step 6.4a: A global view of an original video is input into the fine-tuner, and each video frame in the global view is divided into 14×14 non-overlapping blocks of size 16×16; then the 14×14×T3 blocks corresponding to the global view are added with a class token feature of dimension 768 and input into the teacher encoder, and the teacher encoder outputs a class token feature corresponding to the global view and a token feature corresponding to each block in the global view, where T3 represents the total number of video frames contained in the global view; then the token features corresponding to all blocks in the global view are input into the linear classification layer, and the linear classification layer outputs a feature vector; Step 6.5a: Based on the global view of all original videos, the fine-tuner is trained. After each iteration, the cross entropy loss between the feature vector output by the linear classification layer and the action label is calculated for each relevant original video. The network parameters of the teacher encoder and the linear classification layer are updated according to the cross entropy loss.

9. The video action classification method based on segmented contrast mask autoencoder according to claim 2 is characterized in that In step 6, the process of linear detection of the teacher encoder under the optimal weight is combined with the linear classification layer as follows: Step 6.1b: Select multiple original videos and their action labels, each of which has a duration of 5 to 20 seconds; Step 6.2b: Obtain a global view of each original video, wherein the spatial resolution of the video frame in the global view is 224×224; Step 6.3b: Construct a linear detector consisting of a teacher encoder and a linear classification layer with optimal weights. Step 6.4b: A global view of an original video is input into the linear detector, and each video frame in the global view is divided into 14×14 non-overlapping blocks of size 16×16; then the 14×14×T3 blocks corresponding to the global view are added with a class token feature of dimension 768 and input into the teacher encoder, and the teacher encoder outputs a class token feature corresponding to the global view and a token feature corresponding to each block in the global view, where T3 represents the total number of video frames contained in the global view; then the token features corresponding to all blocks in the global view are input into the linear classification layer, and the linear classification layer outputs a feature vector; Step 6.5b: Based on the global view of all original videos, the linear detector is trained. After each iteration, the cross entropy loss between the feature vector output by the linear classification layer and the action label is calculated for each relevant original video. The network parameters of the linear classification layer are updated according to the cross entropy loss, while the network parameters of the teacher encoder are frozen.

Images