A video masking auto-encoding method and system

Abstract

The application relates to the technical field of video learning, and discloses a video mask self-encoding method and system. e In the pre-training stage, the encoder and the decoder are masked based on a double mask strategy, including: using cubic embedding to process an input video to generate a video block; e determining visible video blocks and invisible video blocks, and extracting features of partially visible video blocks; and generating a second mask map M d by the decoder according to the second mask map M d reconstructing features of partially invisible video blocks. The application can reduce pre-training overhead while maintaining model performance based on the above-mentioned double mask strategy, thereby effectively saving computer operation resources.

Description

Technical Field

[0001] This invention generally relates to the field of video learning technology. Specifically, this invention relates to a video mask self-encoding method and system. Background Technology

[0002] Efficient pre-training of large-scale foundational models on massive datasets is a popular paradigm for learning general representations. This paradigm has been successful across various data modalities, including natural language, audio, images, video, and visual language. These foundational models possess strong generalization capabilities and demonstrate powerful performance in many downstream tasks, driving advancements in numerous areas of artificial intelligence.

[0003] In the field of vision, numerous works have been published on model pre-training, among which Transformer models pre-trained using the MAE (Masked Autoencoder) method are becoming simple and effective visual learners. Given the success of language models, the academic community believes that scaling up the model and data scale is a crucial factor in improving the performance of such visual learners. However, there is very little work on scaling MAE pre-training onto large-scale vision models.

[0004] In fact, in the video domain, due to the additional temporal dimension of video, the computational cost is extremely high, and no such work has yet emerged. Therefore, it is necessary to fill this gap by researching the scaling of VideoMAE (Video Masked Autoencoder) to scales such as those with billions of parameters, extending it, and improving its performance ceiling on various downstream video tasks. Summary of the Invention

[0005] To at least partially solve the above-mentioned problems in the prior art, the present invention proposes a video mask auto-encoding method, which includes the following steps:

[0006] During the pre-training phase, cubic embedding is used to process the input video to generate video blocks;

[0007] The first mask image M is generated based on the pipeline masking strategy. e The encoder is based on the first mask image M e Identify visible and invisible video blocks, and extract features from the partially visible video blocks; and

[0008] Generating the second mask image M based on the motion unit masking strategy d The decoder determines the second mask image M based on the second mask image M. d Reconstruct the features of partially invisible video blocks.

[0009] In one embodiment of the present invention, the encoder is specified to use a visual transformer model, wherein one or more of the following parameters of the visual transformer model are extended:

[0010] Video block size, number of transformer layers, number of feature channels, number of extended layers of multilayer perceptron, and number of multi-head attention heads.

[0011] In one embodiment of the present invention, it is specified that an unlabeled multi-source video dataset is constructed for pre-training, wherein constructing the unlabeled multi-source video dataset includes extracting one or more of the following datasets:

[0012] Extract videos from the Kinetics dataset, where the Kinetics-400 / 600 / 700 datasets are merged, and duplicate videos and videos appearing in the Kinetics validation set are removed;

[0013] Extract videos from the SSv2 dataset;

[0014] Extract videos from the AVA dataset, specifically the video segments from minute 15 to minute 30 of the movie videos in the AVA dataset, and generate video clips by splitting them into 300-frame segments;

[0015] Extracting videos from the WebVid dataset; and

[0016] Videos are randomly extracted from an internally collected, non-public video dataset.

[0017] In one embodiment of the present invention, a first-stage pre-training is performed using the unlabeled multi-source dataset, and a labeled hybrid dataset is constructed, followed by a supervised second-stage pre-training on the labeled hybrid dataset.

[0018] In one embodiment of the present invention, constructing a labeled hybrid dataset includes merging the training dataset and validation dataset of the Kinetics-400 / 600 / 700 dataset, wherein identical labels in the Kinetics-400 / 600 / 700 dataset are merged, and duplicate videos and videos appearing in the validation set are removed.

[0019] In one embodiment of the present invention, the video mask autoencoder method further includes fine-tuning the pre-trained encoder, wherein a linear head is added after the encoder for action classification, and supervised fine-tuning is performed on the labeled mixed dataset.

[0020] In one embodiment of the present invention, supervised fine-tuning includes the following steps:

[0021] The encoder extracts video features from the input video and transmits these features to the linear head.

[0022] The linear head predicts the probability that the input video belongs to different categories;

[0023] Supervise model parameter updates using cross-entropy loss; and

[0024] Once the model converges, the encoder generates the complete video base model.

[0025] This invention also proposes a video mask auto-encoding system, comprising:

[0026] Encoder;

[0027] Decoder;

[0028] The pre-training module, configured to mask the encoder and decoder based on a dual-masking strategy, includes the following steps:

[0029] Use cubic embedding to process the input video and generate video blocks;

[0030] The first mask image M is generated based on the pipeline masking strategy. e The encoder is based on the first mask image M e Identify visible and invisible video blocks, and extract features from the partially visible video blocks; and

[0031] Generating the second mask image M based on the motion unit masking strategy d The decoder determines the second mask image M based on the second mask image M. d Reconstructing features of partially invisible video blocks; and

[0032] A supervised fine-tuning module is configured to fine-tune the pre-trained encoder, wherein a linear head is added after the encoder for action classification, and supervised fine-tuning is performed on the labeled mixed dataset.

[0033] The present invention also proposes a computer-readable storage medium having machine-readable instructions stored thereon, which, when executed by a processor, perform the steps according to the method.

[0034] The present invention also proposes a computer system, comprising:

[0035] A processor, configured to execute machine-readable instructions; and

[0036] A memory that stores machine-readable instructions that, when executed by a processor, perform the steps according to the method.

[0037] The present invention has at least the following beneficial effects: The present invention proposes a video mask autoencoder method and system, wherein pre-training based on a dual masking strategy can reduce pre-training overhead while maintaining model performance, thereby effectively saving computer computing resources; The present invention expands the model size and data scale, and through progressive training for pre-training, it can improve the model's ability to extract features, reduce the risk of overfitting during model pre-training, and improve the model's generalization ability, thereby improving the efficiency of downstream tasks such as action recognition, spatiotemporal detection, and temporal action detection. Attached Figure Description

[0038] To further illustrate the advantages and other features of the various embodiments of the present invention, a more specific description of the embodiments of the present invention will be presented with reference to the accompanying drawings. It is understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. In the drawings, identical or corresponding parts will be indicated by the same or similar reference numerals for clarity.

[0039] Figure 1 A computer system implementing the system and / or method according to the present invention is shown.

[0040] Figure 2 The diagram illustrates a flowchart of a video masking self-encoding system using a dual-masking strategy, according to one embodiment of the present invention. Detailed Implementation

[0041] It should be noted that the components in the various figures may be shown exaggeratedly for illustrative purposes and are not necessarily to scale. In each figure, the same reference numerals are used for components that are identical or have the same function.

[0042] In this invention, unless otherwise specified, "arranged on," "arranged above," and "arranged on" do not exclude the possibility of an intermediate element between them. Furthermore, "arranged on or above" merely indicates the relative positional relationship between two components, and in certain cases, such as when the product orientation is reversed, it can also be converted to "arranged below or under," and vice versa.

[0043] In this invention, the various embodiments are merely intended to illustrate the solutions of the invention and should not be construed as limiting.

[0044] In this invention, unless otherwise specified, the quantifiers “a” and “one” do not exclude scenarios involving multiple elements.

[0045] It should also be noted that, in the embodiments of the present invention, only a portion of the components or parts may be shown for clarity and simplicity. However, those skilled in the art will understand that, under the teachings of the present invention, necessary components or parts can be added as needed for specific scenarios. Furthermore, unless otherwise stated, features in different embodiments of the present invention can be combined with each other. For example, a feature in the second embodiment can replace a corresponding or functionally identical or similar feature in the first embodiment, and the resulting embodiment will also fall within the scope of disclosure or description of this application.

[0046] It should also be noted that, within the scope of this invention, the terms "same," "equal," and "equal to" do not imply that the two values ​​are absolutely equal, but rather allow for a certain reasonable margin of error. In other words, the terms also encompass "substantially the same," "substantially equal," and "substantially equal to." Similarly, in this invention, the directional terms "perpendicular to," "parallel to," etc., also encompass the meanings of "substantially perpendicular to" and "substantially parallel to."

[0047] Furthermore, the numbering of the steps in the methods of the present invention does not limit the execution order of the method steps. Unless otherwise specified, the method steps may be executed in different orders.

[0048] Other prior art related to this invention will be described below.

[0049] Visual foundation models: Foundation models refer to powerful models pre-trained on large datasets that can generalize to various downstream tasks. Early research in the field of vision primarily focused on pre-training CNN (Convolutional Neutral Network) and Transformer models on large-scale labeled datasets (ImageNet-1k, ImageNet-22k, JFT, etc.). Recently, unsupervised pre-training based on contrastive learning and Siamese learning frameworks has also emerged. Simultaneously, drawing on successful experiences in natural language processing, some researchers have introduced masked autoencoding methods into the field of vision, completing self-supervised pre-trained image foundation models such as BEiT, SimMIM, and MAE. Other researchers have utilized multimodal data to guide models to align with large-scale noisy image-text pairs crawled from the network, completing the pre-training of visual-language foundation models; representative models include CLIP, ALIGN, and Florence. These visual-language models possess strong zero-shot transfer capabilities.

[0050] The development of video foundational models has lagged far behind that of image foundational models, partly due to the relatively smaller size of video datasets and the greater computational cost of video models. Following the advent of the Kinetics dataset, traditional video foundational models gradually adopted a supervised pre-training paradigm. This involved supervised pre-training on Kinetics first, followed by transfer to other smaller datasets for action recognition tasks. Models using this pre-training paradigm include 2D convolutional neural networks (TSN, TSM), 3D convolutional neural networks (I3D, R(2+1)D, SlowFast), and transformers (TimeSformer, VideoSwin, Uniformer). Recently, unsupervised pre-trained video models have also emerged. These works typically directly transfer image-domain masking autoencoders to the video domain, including BEVT, MaskFeat, VideoMAE, and MAE-ST, among others.

[0051] Masked visual modeling: Early work on masked visual modeling treated masking methods as a form of denoising autoencoder or content inpainting. Inspired by the great success of masking modeling methods in natural language processing, the ViT (VisionTransformer) model made initial attempts at masked feature modeling. Recently, masked visual modeling methods have achieved great success on Transformer-based architectures. MAE and SimMIM employ a pre-training paradigm of masking image pixels and reconstructing the recovered image. MaskFeat pre-trains on video data by reconstructing HOG (Histogram of Oriented Gradient) features. VideoMAE and MAE-ST extend MAE to the video domain, completing self-supervised video model pre-training and achieving excellent results in action recognition.

[0052] Visual model scaling: Numerous works have attempted to scale CNN models to achieve better performance. EfficientNet, designed for CNNs, proposes a scaling strategy that balances model depth, width, and input resolution. Other works have achieved excellent performance by scaling the computational capacity and training data size of larger CNN networks. Recent works have attempted to scale visual transformer models to the billion-parameter level, including supervised pre-training on the JFT-3B dataset and unsupervised pre-training on the IN-22K-ext-70M dataset. VideoMAE and MAE-ST have also trained video transformer models with hundreds of millions of parameters. MAE-ST also attempted pre-training on a dataset of one million IG-uncurated video clips, but did not achieve better performance.

[0053] Existing video foundational models are typically limited by the amount of pre-training data and model size. More importantly, their transferability to downstream tasks is usually limited to action recognition and cannot be extended to more types of video downstream tasks, such as temporal action localization, failing to systematically consider various video-related tasks. The purpose of this invention is to extend the video Transformer model to the billion-parameter level, enabling it to achieve superior performance on a wider range of video downstream tasks.

[0054] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0055] Figure 1 A computer system 100 implementing the system and / or method according to the present invention is shown. Unless otherwise specified, the method and / or system according to the present invention can be implemented in... Figure 1 The invention may be implemented in the computer system 100 shown to achieve the objectives of the invention, or the invention may be implemented in a distributed manner in multiple computer systems 100 according to the invention via a network, such as a local area network or the Internet. The computer system 100 of the invention may include various types of computer systems, such as handheld devices, laptop computers, personal digital assistants (PDAs), multiprocessor systems, microprocessor-based or programmable consumer electronic devices, network PCs, minicomputers, mainframes, network servers, tablet computers, etc.

[0056] like Figure 1 As shown, the computer system 100 includes a processor 111, a system bus 101, a system memory 102, a video adapter 105, an audio adapter 107, a hard disk drive interface 109, an optical drive interface 113, a network interface 114, and a Universal Serial Bus (USB) interface 112. The system bus 101 can be any of several bus architecture types, such as a memory bus or memory controller, a peripheral bus, and a local bus using various bus architectures. The system bus 101 is used for communication between the various bus devices. In addition to... Figure 1Other bus devices or interfaces besides those shown are also conceivable. System memory 102 includes read-only memory (ROM) 103 and random access memory (RAM) 104, wherein ROM 103 may, for example, store basic input / output system (BIOS) data for basic routines used to implement information transfer at startup, while RAM 104 provides the system with faster access speed running memory. Computer system 100 also includes hard disk drive 109 for reading and writing to hard disk 110, optical drive interface 113 for reading and writing to optical media such as CD-ROM, etc. Hard disk 110 may, for example, store operating system and application programs. Drives and their associated computer-readable media provide non-volatile storage for computer-readable instructions, data structures, program modules, and other data for computer system 100. Computer system 100 may also include a video adapter 105 for image processing and / or image output, which is used to connect output devices such as display 106. The computer system 100 may also include an audio adapter 107 for audio processing and / or audio output, which is used to connect output devices such as speakers 108. Furthermore, the computer system 100 may also include a network interface 114 for network connectivity, wherein the network interface 114 can be connected to the Internet 116 via a network device such as a router 115, wherein the connection can be wired or wireless. Additionally, the computer system 100 may also include a Universal Serial Bus (USB) interface 112 for connecting peripheral devices, wherein the peripheral devices include, for example, a keyboard 117, a mouse 118, and other peripheral devices such as microphones, cameras, etc.

[0057] When the present invention is Figure 1 When implemented on the computer system 100, pre-training can be performed based on a dual-mask strategy, which can reduce pre-training overhead while maintaining model performance, thereby effectively saving computer computing resources. Furthermore, it can expand the model size and data scale, and improve the model's feature extraction ability, reduce the risk of overfitting during model pre-training, and improve the model's generalization ability through progressive training, thereby improving the efficiency of downstream tasks such as action recognition, spatiotemporal detection, and temporal action detection.

[0058] Furthermore, the embodiments can be provided as computer program products that may include one or more machine-readable media on which machine-executable instructions are stored, which, when executed by one or more machines such as a computer, computer network, or other electronic equipment, may cause one or more machines to perform operations according to the embodiments of the present invention. Machine-readable media may include, but are not limited to, floppy disks, optical disks, CD-ROMs (compact disc read-only memory) and magneto-optical disks, ROMs (read-only memory), RAMs (random access memory), EPROMs (erasable programmable read-only memory), EEPROMs (electrically erasable programmable read-only memory), magnetic or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions.

[0059] Furthermore, various embodiments can be downloaded as computer program products, wherein the program can be transmitted from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a communication link (e.g., a modem and / or a network connection) using one or more data signals implemented and / or modulated by a carrier wave or other propagation medium. Therefore, the machine-readable medium used herein may include such a carrier wave, but this is not required.

[0060] In this invention, the modules of the system according to the invention can be implemented using software, hardware, firmware, or a combination thereof. When a module is implemented using software, its function can be implemented through computer program flow. For example, the module can be implemented using code segments (such as code segments in languages ​​like C and C++) stored in a storage device (such as a hard disk, memory, etc.), wherein the corresponding function of the module can be implemented when the code segment is executed by a processor. When a module is implemented using hardware, its function can be implemented by setting a corresponding hardware structure. For example, the module's function can be implemented by hardware programming a programmable device such as a field-programmable gate array (FPGA), or by designing an application-specific integrated circuit (ASIC) that includes multiple transistors, resistors, capacitors, and other electronic devices. When a module is implemented using firmware, the module's function can be written into a read-only memory such as an EPROM or EEPROM in the form of program code, and the corresponding function of the module can be implemented when the program code is executed by a processor. In addition, some functions of the module may need to be implemented by separate hardware or by working in cooperation with the hardware. For example, the detection function is implemented by the corresponding sensor (such as a proximity sensor, accelerometer, gyroscope, etc.), the signal transmission function is implemented by the corresponding communication device (such as a Bluetooth device, infrared communication device, baseband communication device, Wi-Fi communication device, etc.), the output function is implemented by the corresponding output device (such as a display, speaker, etc.), and so on.

[0061] This invention proposes a video mask auto-encoding method (VideoMAEv2), which includes the following steps:

[0062] During the pre-training phase, the encoder and decoder are masked based on a dual-masking strategy, where cubic embedding is used to process the input video to generate video blocks.

[0063] The first mask image M is generated based on the pipeline masking strategy. e The encoder is based on the first mask image M e Identify visible and invisible video blocks, and extract features from the partially visible video blocks; and

[0064] Generating the second mask image M based on the motion unit masking strategy d The decoder determines the second mask image M based on the second mask image M. d Reconstructing features of partially invisible video blocks; and

[0065] The pre-trained encoder is fine-tuned, wherein a linear head is added after the encoder for action classification, and supervised fine-tuning is performed on the labeled mixed dataset.

[0066] The following description, in conjunction with specific embodiments, provides further details.

[0067] This invention further improves and extends VideoMAE. Specifically, based on a dual-masking strategy, data scale expansion, and progressive training, a simple and efficient VideoMAEv2 is proposed, using a ViT network with billions of parameters as its backbone. This system can train a video foundation model with billions of parameters and surpasses the performance limits of the original video ViT model in various downstream video tasks such as action recognition, temporal action detection, and spatiotemporal detection.

[0068] During the pre-training phase, VideoMAEv2 improves pre-training efficiency through a dual-masking strategy. Figure 2 This diagram illustrates a flowchart of a video masking self-encoding system using a dual-masking strategy, according to one embodiment of the present invention. Figure 2 As shown, VideoMAEv2 uses an asymmetric encoder-decoder framework and employs cubic embedding to process the input video. The dual-masking strategy refers to generating two mask images M for the encoder and decoder respectively, based on different masking strategies and ratios. e and M d The mask encoder in the dual-masking strategy is consistent with VideoMAE, using a pipeline masking strategy with a very high mask ratio to generate the mask map M. e And according to the mask image M eThis approach extracts features from the visible video blocks, while the mask decoder only reconstructs the invisible video blocks. Unlike VideoMAE, which reconstructs all invisible video blocks, this scheme reduces the input length of the decoder, decreases computational overhead, and achieves model performance close to that of the complete reconstruction scheme.

[0069] In existing technologies, VideoMAE's decoder needs to reconstruct all invisible video blocks, resulting in low efficiency. MAR (Masked Action Recognition) uses partial video blocks as model input. VideoMAE's encoder uses a pipeline masking strategy, which can better suppress information leakage but also increases the reconstruction difficulty. The decoder's masking strategy needs to supplement information in the reconstructed partial video blocks, encouraging the coverage of as many video blocks as possible during the reconstruction process. After comparing random masking strategies, random video frame masking strategies, and motion unit masking strategies, the inventors use a motion unit masking strategy in this invention. The motion unit masking strategy, proposed by MAR, refers to dividing a video segment into multiple units, where each unit includes four video blocks, and selecting different video blocks from the same spatial location at different time points according to a certain pattern. Experiments show that the motion unit masking strategy can improve training speed by 50% with only a small performance loss. Furthermore, the motion unit masking strategy can also be replaced by other effective masking strategies.

[0070] In embodiments of this invention, the model size is expanded to achieve superior performance. VideoMAEv2, referencing VideoMAE, uses a concise, original ViT (Vision Transformer) model as its backbone network. Following the ViT expansion rules, the model is expanded to ViT-g with one billion parameters, making it the first video model with a billion-parameter count. To reduce pre-training overhead, the VideoMAEv2 autoencoder system uses a shallower, narrower Transformer model as its decoder. Furthermore, the concise original ViT can be replaced with a ViT variant model or other models that can be pre-trained using masked video modeling methods.

[0071] The following details the process by which VideoMAEv2 extends ViT-H to ViT-g with a billion parameters.

[0072] The ViT model consists of a video patch embedding module and multiple identical Transformer layers stacked together. Different scales of ViT have different patch sizes (with the input video size remaining constant, a smaller patch size results in more embedded video patches, leading to higher computational cost and generally better performance), Transformer layer depth, feature width, MLP expansion width, and the number of multi-head attention heads. The largest traditional VideoMAEv1 model is ViT-huge, with values ​​of 16, 32, 1280, 5120, and 16 respectively; after scaling to ViT-giant, these values ​​become 14, 40, 1408, 6144, and 16 respectively. The expanded structure is shown in Table 1.

[0073] Table 1

[0074]

[0075] Furthermore, in embodiments of the present invention, the data scale is expanded to achieve superior performance. Traditional VideoMAE, pre-trained on small datasets, performs poorly on many downstream tasks. VideoMAEv2, by constructing a more general video foundation model, enables the model to achieve superior performance on a wide range of downstream tasks. The expansion of the data scale was explored, extending the unlabeled pre-training dataset to the millions. While expanding the pre-training data scale, considering the issue of data diversity, a pre-training dataset including multiple video sources such as the internet, YouTube, Instagram, movies, and personal recordings was constructed. Specifically, by compiling several publicly available academic video datasets (Kinetics, Something-Something, AVA, and WebVid) and collecting a portion of unedited videos from Instagram, an unlabeled multi-source video dataset containing 1.3 million video clips was constructed, named UnlabeledHybrid. The composition of the UnlabeledHybrid dataset is shown in Table 2.

[0076] Table 2

[0077]

[0078] In processing the aforementioned datasets, for the Kinetics dataset, the Kinetics-400 / 600 / 700 versions were merged, while duplicate videos and videos appearing in the Kinetics validation set were removed. This resulted in 658k videos. For the SSv2 dataset, all videos from the SSv2 training set were directly used, totaling 169k videos. For the AVA dataset, all videos were movies; only the 15th to 30th minute segments were used. All videos in the AVA training set were divided into 300-frame segments (10 seconds each), resulting in 21k video clips. For the WebVid dataset, which has two versions: 2M (2 million videos) and 1OM (10 million videos), 250k videos were randomly selected from WebVid-2M in this embodiment. For the self-collected video dataset, 250k videos were randomly selected from an internally collected, non-public video dataset in this embodiment.

[0079] VideoMAEv2 was the first to propose pre-training on such a large-scale multi-source video dataset, revealing the impact of data scale and diversity on video mask pre-training. Furthermore, using other datasets to construct the large-scale multi-source pre-training dataset of this invention is also conceivable.

[0080] The unlabeled multi-source pre-trained video dataset constructed in this invention features both large scale and multiple sources. A large number of diverse videos allows the model to learn a stronger ability to extract features. Traditional VideoMAEv1 is only pre-trained on its corresponding dataset, where the something-somethingv2 dataset contains a small number of videos (169k) and short durations (1-2 seconds). Pre-training only on these videos limits the feature extraction capabilities of VideoMAEv1. In fact, VideoMAEv2 shows significant improvements over SSV2.

[0081] Meanwhile, a large number of videos can reduce the risk of overfitting during model pre-training. Overfitting in mask modeling pre-training refers to the model memorizing the characteristics of videos in the dataset during training, enabling it to reconstruct the pixels of training videos well, but failing to reconstruct the pixels of unseen videos well. When the model memorizes the unique features of training videos, rather than the common features of videos, the model's performance drops sharply. With an increased number of videos, the model is less likely to memorize the unique features of training videos, but instead learns the common features of all videos, allowing the model to iterate more times. When the number of model iterations increases without overfitting, the model's performance usually increases. VideoMAEv2 expanded the size of the training dataset, allowing for more iterations during pre-training than VideoMAV1, resulting in superior model performance.

[0082] Meanwhile, multi-source videos can improve the model's generalization ability. Generalization ability refers to the model's ability to maintain performance on unfamiliar video types. For example, when Videomaev1 is pre-trained only on the Kinetics dataset from YouTube, its ability to extract features from videos from Instagram decreases. Videomaev2 uses multi-source datasets during pre-training, which improves the model's generalization ability, enabling the model trained with Videomaev2 to achieve excellent results on a wider range of tasks.

[0083] The transfer learning process is a crucial step in adapting pre-trained large video models to downstream tasks. Typically, masked autoencoder pre-training helps models learn invariant features and achieve good initialization. However, traditional VideoMAE only provides supervised fine-tuning of the pre-trained model on the target dataset. Due to limited supervision, this direct transfer approach may not fully unlock the potential of the pre-trained large model. To mitigate overfitting risks and unleash the potential of pre-trained models, VideoMAEv2 adopts a progressive training paradigm, utilizing multi-source, highly semantic supervision at multiple stages to gradually transfer the pre-trained large model to downstream tasks. Based on this, VideoMAEv2 proposes a progressive training paradigm.

[0084] VideoMAEv2 first performs masked video pre-training on the UnlabeledHybrid dataset, then constructs a labeled hybrid dataset, and performs a supervised post-pre-training phase on this labeled hybrid dataset. The labeled hybrid dataset consists of various versions of Kinetics, merging identical labels and removing duplicate videos and videos appearing in the validation set; therefore, it is named LabeledHybrid or Kinetics-710. The training and validation sets of the labeled hybrid dataset are obtained by mixing the Kinetics-400 / 600 / 700 datasets (400, 600, and 700 refer to the number of video labels (action categories) in the dataset; video labels include, for example, "skiing," "running on a treadmill," "playing the piano," etc.). The data processing may include: merging the training and validation datasets of the three Kinetics (-400 / 600 / 700) datasets respectively, to create the training and validation sets for the LabeledHybrid dataset; removing duplicate videos from the training and validation sets; removing duplicate videos from the validation set (since there are three versions of Kinetics, there will be identical videos); removing training videos that appear in the validation set; and merging identical category labels from each version of the dataset, resulting in 710 video labels. Ultimately, this yields 658k training videos and 67k validation videos.

[0085] Finally, VideoMAEv2 undergoes a fine-tuning phase on the target dataset, transferring general semantics to task-centric knowledge. Furthermore, this progressive training paradigm can be conceived for application to other foundational models.

[0086] The pre-trained model's encoder serves as the encoder (or backbone) in the supervised fine-tuning phase. A linear head is added after the encoder for action classification (this linear head maps video features to the probabilities of the 710 LabeledHybrid classes). Supervised fine-tuning is performed on the LabeledHybrid training set until the model converges on that dataset (model convergence means the model's classification accuracy on the validation set no longer improves). The fine-tuning process specifically includes: extracting video features using the encoder for a given input video; feeding the video features into the linear head to predict the probability of the video belonging to each class; and supervising the model's parameter updates using cross-entropy loss. Once the model converges, the encoder forms the trained video base model.

[0087] This invention also proposes a video mask auto-encoding system, comprising:

[0088] Encoder;

[0089] Decoder;

[0090] The pre-training module, configured to mask the encoder and decoder based on a dual-masking strategy, includes the following steps:

[0091] Use cubic embedding to process the input video and generate video blocks;

[0092] The first mask image M is generated based on the pipeline masking strategy. e The encoder is based on the first mask image M e Identify visible and invisible video blocks, and extract features from the partially visible video blocks; and

[0093] Generating the second mask image M based on the motion unit masking strategy d The decoder determines the second mask image M based on the second mask image M. d Reconstructing features of partially invisible video blocks; and

[0094] A supervised fine-tuning module is configured to fine-tune the pre-trained encoder, wherein a linear head is added after the encoder for action classification, and supervised fine-tuning is performed on the labeled mixed dataset.

[0095] In the embodiments of this invention, experiments were conducted to verify the feasibility of the dual-masking strategy design, and corresponding experiments were performed on the proposed data scale expansion, model size expansion, and progressive training to verify its effectiveness. Experiments were conducted on various downstream tasks to verify the excellent performance of the trained video base model.

[0096] Specifically, in one embodiment of this invention, an ablation experiment was conducted using a dual-masking strategy. The ViT-B model was used as the backbone network, and 800 rounds of pre-training and fine-tuning were performed on the Something-Something V2 dataset. The experiment showed that without a mask decoder, the Top-1 classification accuracy was 70.28%, with a pre-training time of 28.4 hours; with a random mask decoder (mask ratio of 50%), the classification accuracy was 64.87%; with a random video frame mask decoder (mask ratio of 50%), the classification accuracy was 69.76%; and with a motion unit mask decoder (mask ratio of 50%), the classification accuracy was 70.15%, with a pre-training time of 15.9 hours. The experimental data demonstrate that the dual-masking design using a motion unit mask strategy as the decoder mask strategy can reduce computational overhead and accelerate pre-training while maintaining similar performance.

[0097] In one embodiment of the invention, experiments were conducted to expand the pre-training data size, verifying the fine-tuned classification performance of the model on the Kinetics-400 and Something-Something V2 datasets. Experiments on the Kinetics-400 dataset showed that, for the ViT-B, ViT-L, and ViT-H models, using the expanded pre-training dataset resulted in 0.0%, 0.2%, and 0.3% higher classification accuracy compared to direct pre-training on Kinetics-400, respectively. Experiments on the Something-Something V2 dataset showed that, for the ViT-B, ViT-L, and ViT-H models, using the expanded pre-training dataset resulted in 0.4%, 1.4%, and 2.0% higher classification accuracy compared to direct pre-training on Something-Something V2, respectively. The experimental data demonstrate that expanding the pre-training data size brings performance benefits to the model, and these benefits increase with the model size. Furthermore, the benefits of expanding the data size are greater on the smaller Something-Something V2 dataset.

[0098] In one embodiment of the present invention, experiments were conducted to expand the model size, verifying the fine-tuned classification performance on the Kinetics-400 and Something-Something V2 datasets after model size expansion. Experiments on the Kinetics-400 dataset showed that when pre-trained on the UnlabeledHybrid dataset and fine-tuned directly on the Kinetics-400 dataset, the classification accuracies of the ViT-B, ViT-L, ViT-H, and ViT-g models were 81.5%, 85.4%, 86.9%, and 87.2%, respectively. Experiments on the Something-Something V2 dataset showed that when pre-trained on the UnlabeledHybrid dataset and fine-tuned directly on the Something-Something V2 dataset, the classification accuracies of the ViT-B, ViT-L, ViT-H, and ViT-g models were 71.2%, 75.7%, 76.8%, and 77.0%, respectively. These experiments demonstrate that model performance increases with model size expansion, but this benefit gradually decreases as the model size increases.

[0099] In one embodiment of the invention, an experiment on progressive training was conducted to verify the impact of progressive training on the classification performance of the model on the Kinetics-400 dataset. The direct fine-tuning training mode fine-tunes the model directly on the Kinetics-400 dataset after pre-training, while the progressive training mode fine-tunes the model first on the LabeledHybrid dataset to unlock its performance potential after pre-training, and then fine-tunes it on the Kinetics-400 dataset. Using the direct fine-tuning training mode, the ViT-H and ViT-g models achieved classification accuracies of 86.9% and 87.2% on Kinetics-400, respectively. Using the progressive training mode, the ViT-H and ViT-g models achieved classification accuracies of 88.6% and 88.5% on Kinetics-400, respectively. This demonstrates that progressive training can significantly unlock the model's performance potential. The experiment also found that the accuracy of ViT-g after progressive training was not higher than that of ViT-H. This may be because its classification performance on Kinetics-400 is close to saturation. The experiment also found that when the model classification performance was tested using a single view, the accuracy of ViT-H was 85.0, which was lower than the accuracy of ViT-g (85.6).

[0100] In one embodiment of the invention, experiments were conducted on an action recognition task, which was performed on the Kinetics-400 / 600 and Something-SomethingV1 / V2 datasets, achieving classification accuracies of 88.6% / 88.8% and 68.3% / 77.0%, respectively. On the Kinetics dataset, the VideoMAEv2 autoencoder system achieved the highest performance among all models trained without using the internal dataset. On the Something-Something dataset, the VideoMAEv2 autoencoder system achieved the highest performance among all models, especially on the Something-Somethingv1 dataset, where its classification accuracy exceeded the second-best by 10%.

[0101] In one embodiment of the present invention, experiments were conducted on a spatiotemporal detection task, specifically on the AVA and AVA-Kinetics datasets, achieving mAP (mean average precision) of 41.0 and 42.5 respectively, which are the best performance among all models.

[0102] In one embodiment of this invention, experiments were conducted on the temporal action detection task, specifically on the THUMOS14 and FineAction datasets, achieving mAP of 69.6 and 17.6 respectively, both representing the best performance among all models. The ActionFormer detection framework was used for the temporal action detection task.

[0103] Although various embodiments of the invention have been described above, it should be understood that they are presented by way of example only and not as limitations. It will be apparent to those skilled in the art that various combinations, modifications, and alterations can be made without departing from the spirit and scope of the invention. Therefore, the breadth and scope of the invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely by the appended claims and their equivalents.

Claims

1. A video mask auto-encoding method, characterized in that, Includes the following steps: During the pre-training phase, cubic embedding is used to process the input video to generate video chunks; The first mask image M is generated based on the pipeline masking strategy. e The encoder is based on the first mask image M e Identify visible and invisible video blocks, and extract features from some of the visible video blocks; as well as Generating the second mask image M based on the motion unit masking strategy d The decoder determines the result based on the second mask image M. d Reconstruct the features of partially invisible video blocks.

2. The video masking auto-encoding method according to claim 1, characterized in that, The encoder uses a vision transformer model, wherein one or more of the following parameters of the vision transformer model are extended: Video block size, number of transformer layers, number of feature channels, number of extended layers of multilayer perceptron, and number of multi-head attention heads.

3. The video mask auto-encoding method according to claim 1, characterized in that, Construct an unlabeled multi-source video dataset for pre-training, wherein constructing the unlabeled multi-source video dataset includes extracting one or more of the following datasets: Extract videos from the Kinetics dataset, where the Kinetics-400 / 600 / 700 datasets are merged, and duplicate videos and videos appearing in the Kinetics validation set are removed; Extract videos from the SSv2 dataset; Extract videos from the AVA dataset, specifically the video segments from minute 15 to minute 30 of the movie videos in the AVA dataset, and generate video clips by dividing them into 300 frames each; Extract videos from the WebVid dataset; as well as Videos are randomly extracted from an internally collected, non-public video dataset.

4. The video mask self-encoding method according to claim 3, characterized in that, The first stage of pre-training is performed using the unlabeled multi-source video dataset, and a labeled hybrid dataset is constructed for supervised second-stage pre-training on the labeled hybrid dataset.

5. The video mask self-encoding method according to claim 4, characterized in that, Constructing the labeled hybrid dataset involves merging the training and validation datasets of the Kinetics-400 / 600 / 700 datasets, where identical labels in the Kinetics-400 / 600 / 700 datasets are merged, and duplicate videos and videos appearing in the validation set are removed.

6. The video mask self-encoding method according to claim 5, characterized in that, It also includes fine-tuning the pre-trained encoder, wherein a linear head is added after the encoder for action classification, and supervised fine-tuning is performed on the labeled mixed dataset.

7. The video mask self-encoding method according to claim 6, characterized in that, Oversight and fine-tuning include the following steps: The encoder extracts video features from the input video and transmits these features to the linear head. The linear head predicts the probability that the input video belongs to different categories; Use cross-entropy loss to supervise model parameter updates; as well as Once the model converges, the encoder generates the complete video base model.

8. A video mask auto-encoding system, characterized in that, include: Encoder; Decoder; The pre-training module, configured to mask the encoder and decoder based on a dual-masking strategy, includes the following steps: Use cubic embedding to process the input video and generate video blocks; The first mask image M is generated based on the pipeline masking strategy. e The encoder is based on the first mask image M e Identify visible and invisible video blocks, and extract features from some of the visible video blocks; as well as Generating the second mask image M based on the motion unit masking strategy d The decoder determines the second mask image M based on the second mask image M. d Reconstruct features of partially invisible video blocks; as well as A supervised fine-tuning module is configured to fine-tune the pre-trained encoder, wherein a linear head is added after the encoder for action classification, and supervised fine-tuning is performed on the labeled mixed dataset.

9. A computer-readable storage medium having machine-readable instructions stored thereon, the machine-readable instructions, when executed by a processor, performing the steps of the method according to any one of claims 1-7.

10. A computer system, comprising: A processor, configured to execute machine-readable instructions; as well as A memory that stores machine-readable instructions that, when executed by a processor, perform the steps of the method according to any one of claims 1-7.

Images