Asymmetric mask distillation method for small mask autoencoder pre-training

Abstract

The present application relates to an asymmetric mask distillation method for small mask autoencoder pre-training, comprising: performing asymmetric masking on video inputs of a student encoder and a teacher encoder to obtain a plurality of tubular tokens as feature inputs of the teacher encoder and the student encoder, wherein a masking rate of the video inputs of the teacher encoder is less than a masking rate of the video inputs of the student encoder; and performing feature alignment between the student encoder and the teacher encoder.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an asymmetric mask distillation method for pre-training small mask autoencoders. Background Technology

[0002] Self-supervised learning has achieved tremendous success in recent years, surpassing many supervised learning methods. Recovering original images from damaged images has recently been introduced as an effective pre-training paradigm. Early works employed similar approaches for image denoising or inpainting. Following the success of Transformers in computer vision, recent work has attempted to apply Vision Transformers (ViTs) to Masked Autoencoders (MAEs). With the success of proposing generative tasks as pre-training objectives in natural language processing, recent work has proposed ViT-based masked image modeling, treating image patches as words. This simple masking and reconstruction framework has demonstrated excellent performance in downstream tasks such as image classification, object detection, semantic segmentation, and action recognition. Reconstruction objectives can be categorized into high-level and low-level features. A two-stage process is performed in high-level feature reconstruction because the reconstruction objective is discrete tokens, requiring a pre-trained token generator. For low-level reconstruction, work has proposed a single-stage pre-training method using oriented gradient histograms as the reconstruction objective. The reconstruction objective for MAE is pixels; this landmark work proposes an asymmetric autoencoder architecture that leverages a high mask rate to reduce computational overhead and enables scalability. Due to its excellent scalability, recent work has extended MAE from images to videos. The Video Mask Autoencoder (VideoMAE) applies a tube-style masking strategy to video data and employs joint spatiotemporal attention in the Transformer block to extract video features, which performs well on the Human Everyday Life Dataset (Something-in-Something V2 dataset).

[0003] Masking autoencoding frameworks still face several challenges. First, encoders typically operate on a small subset of visible markers with high masking rates (e.g., 75% for images, 90% for videos). This high masking rate can increase the difficulty of pre-training tasks and may encourage the encoder to capture more useful high-level information for reconstruction. This high masking rate can also lead to the loss of important and detailed structural information, resulting in pre-trained models capturing incomplete and biased visual information. Second, masking autoencoding often requires high-capacity ViT backbones (e.g., ViT-L and ViT-H) to unleash the power of masking pre-training. These large models incur high computational and memory costs when fine-tuning for downstream tasks, a cost particularly acute for video inputs where video converters take multiple frames as input.

[0004] Knowledge distillation, first proposed by Hinton et al., is an effective method for compressing models. A common distillation technique utilizes the classification prediction output of the teacher model as a medium for knowledge transfer. To introduce more dark knowledge from the teacher model, a temperature factor is introduced to align soft labels with the Kullback-Leibler divergence loss. However, distillation methods based on classification prediction output can only be applied to fine-tuning models, which impairs the generalization ability of unsupervised pre-trained models. In addition to utilizing the classification prediction output of the teacher model, researchers have also used intermediate features of the model for feature distillation. ViTKD combines classification prediction output alignment with feature alignment and performs well on fine-tuned models using supervised information. Recent work has successfully applied feature alignment to self-supervised learning methods based on contrastive learning. Analysis of optimization-friendly properties concludes that MAE can hardly benefit from direct feature distillation. Masked Image Pre-training of Language-Auxiliary Representations (MILAN) utilizes the high semantic information of contrastive language-image pre-training (CLIP) for feature alignment and designs an improved masking strategy for MAE.

[0005] The Masked Autoencoder supports an efficient Knowledge Distiller (DMAE) that employs a symmetric masking method based on MAE for feature alignment distillation during student model pre-training, and its technical validation has been performed solely on an image dataset. DMAE uses a symmetric masking strategy, meaning the teacher and student models apply the same masking method. DMAE allows both the teacher and student models to receive symmetric unmasked image patches, enabling direct feature alignment and reducing the computational complexity of the teacher model. However, the identical masking for both teachers and students limits the teacher model's ability to gather more contextual information from samples and exposes it to the risk of information leakage. Summary of the Invention

[0006] To address at least some of the problems mentioned above in the prior art, the present invention provides an asymmetric mask distillation method for pre-training a small mask autoencoder, comprising:

[0007] Asymmetric masking is applied to the video inputs of the student encoder and teacher encoder to obtain multiple tube tokens as feature inputs to both the teacher and student encoders. The masking rate of the teacher encoder's video input is less than that of the student encoder's video input.

[0008] Perform feature alignment between the student encoder and the teacher encoder.

[0009] Furthermore, multiple tubular tokens form the input sequence, and the length of the input sequence for the teacher encoder is greater than the length of the input sequence for the student encoder; and

[0010] The features of the student encoder are a subset of the features of the teacher encoder.

[0011] Furthermore, the asymmetric masking of the video inputs to the student encoder and the teacher encoder includes:

[0012] Generate mask matrices for the student encoder and teacher encoder, and then obtain the visible token index of the student encoder. and the visible token index of the teacher encoder in It is the token index of the student encoder. It is the token index of the teacher encoder. It is the length of the input sequence of the student encoder. is the length of the input sequence of the teacher encoder, and N is the number of tube tokens in the entire input sample.

[0013] Furthermore, the visible token index of the student encoder is a subset of the visible token index of the teacher encoder.

[0014] Furthermore, a serial alignment method is used to align features between the student encoder and the teacher encoder, wherein the serial alignment method includes direct alignment and generative alignment performed sequentially, wherein direct alignment is used for features common to both the student encoder and the teacher encoder, and generative alignment is used for features that exist in the teacher encoder but not in the student encoder.

[0015] Furthermore, the feature alignment between the student encoder and the teacher encoder using the serial alignment method includes:

[0016] For a specific layer l of the student encoder, the corresponding layer of the teacher encoder is l. * The features of the student encoder and the teacher encoder were extracted as follows:

[0017] The first step in direct alignment is to apply the projection function φ(·) to the student encoder to align the feature dimensions of the teacher encoder, where a linear projection layer is used to obtain the projected features of the student encoder. For direct alignment; and

[0018] The projected features of the student encoder are continuously used to generate features through the generator for alignment.

[0019] Furthermore, the generation of features via a generator includes:

[0020] Where z m This represents a masked token, where PE is a one-dimensional positional encoding.

[0021] Furthermore, when aligning multi-layer features, the direct alignment loss function is:

[0022] Where φ l This represents the projection function of the l-th layer. Let z(p) be the length of the input sequence of the student encoder, and z(p) represent the p-th token extracted from the input sequence as a feature; and

[0023] When aligning multi-layer features, the generated alignment loss function is:

[0024] in This represents the generator of the l-th layer. It is the length of the sequence that the teacher encoder inputs more than the student encoder, and z(p) represents the p-th token extracted from the input sequence as a feature.

[0025] Furthermore, it also includes reconstructing the pixel values ​​of the masked portion of the video during feature alignment, where the overall loss function for the feature alignment task and the video reconstruction task is: L total =L recon +L dir +L gen L recon It is the loss function for reconstructing the video, L dir It is the loss function for direct alignment, L gen It is the loss function for generating alignment.

[0026] Furthermore, a parallel alignment method is used instead of a serial alignment method. The parallel alignment includes direct alignment and generative alignment performed simultaneously. In the direct alignment process, a linear projection layer is used to obtain the feature maps of the student encoder for alignment. In the generative alignment process, a linear projection layer is used to obtain the feature maps of the student encoder and generate features through these feature maps to align with the features unique to the teacher encoder. The parameters of the linear projection layer in the direct alignment and the linear projection layer in the generative alignment are not shared.

[0027] The present invention also provides a computer-readable storage medium having a computer program stored thereon, the computer program performing the steps according to the method described above when executed by a processor.

[0028] This invention offers at least the following advantages: The disclosed asymmetric mask distillation method for pre-training small-scale masked autoencoders applies an asymmetric masking scheme to both the student encoder and the teacher encoder. The mask rate of the teacher encoder's video input is lower than that of the student encoder's video input, and the unmasked image blocks of the student encoder are a subset of those of the teacher encoder. This asymmetric mask distillation maintains the difficulty of the reconstruction task for the student during pre-training and also allows the teacher to receive more contextual information that can be passed to the student.

[0029] An asymmetric masking scheme is applied between the student encoder and the teacher encoder, resulting in two types of features: features present in both the student and teacher encoders, and features present only in the teacher encoder. A serial alignment method is used to align these two types of features. For the first type of features, direct alignment is used. For the second type of features, the projected features from the student encoder are fed to a decoder-like generator to generate features, which are then aligned with features visible only to the teacher encoder. The length of the generator's input sequence is based on the number of visible tokens encoded by the teacher, meaning the student encoder does not need to generate all mask tokens like a decoder. By incorporating positional encoding during feature generation, the generator ensures the student's predictive ability while reducing computational overhead. This serial alignment appropriately reduces the difficulty of the generation alignment task.

[0030] This asymmetric mask distillation method can be used to construct more efficient and effective small models that can be effectively adapted to downstream tasks. The computational cost and memory consumption of small models are reduced compared to large models. Attached Figure Description

[0031] To further illustrate the above and other advantages and features of the various embodiments of the present invention, a more specific description of the various embodiments of the present invention will be presented with reference to the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope.

[0032] Figure 1 A flowchart of an asymmetric mask distillation method for pre-training a small mask autoencoder according to an embodiment of the present invention is shown;

[0033] Figure 2 A schematic diagram illustrating an asymmetric masking strategy and a symmetric masking strategy according to an embodiment of the present invention is shown; and

[0034] Figure 3 A schematic diagram illustrating direct alignment, generative alignment, parallel alignment, and serial alignment according to an embodiment of the present invention is shown. Detailed Implementation

[0035] It should be noted that the components in the accompanying drawings may be shown exaggerated for illustrative purposes and may not be to scale.

[0036] In this invention, the various embodiments are merely intended to illustrate the solutions of the invention and should not be construed as limiting.

[0037] In this invention, unless otherwise specified, the quantifiers “a” and “one” do not exclude scenarios involving multiple elements.

[0038] It should also be noted that, in the embodiments of the present invention, only a portion of the parts or components may be shown for clarity and simplicity. However, those skilled in the art will understand that, under the teachings of the present invention, the required parts or components can be added as needed for specific scenarios.

[0039] It should also be noted that within the scope of this invention, the terms "same", "equal", and "equal to" do not mean that the two values ​​are absolutely equal, but allow for a certain reasonable error. In other words, the terms also cover "substantially the same", "substantially equal", and "substantially equal to".

[0040] It should also be noted that in the description of this invention, the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not explicitly or implicitly suggest that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0041] Furthermore, the numbering of the steps in the methods of the present invention does not limit the execution order of the method steps. Unless otherwise specified, the method steps may be executed in different orders.

[0042] Masked Autoencoders (SdAEs) support efficient Knowledge Distillers (DMAEs) that employ a MAE-based symmetric masking method for feature alignment distillation during student model pre-training, and have been technically validated only on image datasets. DMAE uses a symmetric masking strategy, where the teacher and student models apply the same masking scheme. DMAE allows both models to receive symmetrical unmasked image patches, enabling direct feature alignment and reducing the computational complexity of the teacher model. However, the identical masking for both models limits the teacher model's ability to gather more contextual information from samples and exposes it to the risk of information leakage. At the asymmetric level, similar distillation methods have emerged in recent MAE self-distillation works. These methods construct student and teacher models with a two-stream structure, typically designed with complementary masking schemes for the student and teacher models. Among them, the Self-Distilling Masked Autoencoder (SdAE) successfully accelerates MAE pre-training by analyzing information bottlenecks and applying a multiple masking strategy to the teacher model. SdAE is a method in the self-distillation field; however, directly applying the complementary masking scheme used in SdAE to knowledge distillation may result in excessively large feature gaps, making alignment difficult.

[0043] Inspired by these works, the technical solution of this invention employs an asymmetric masking strategy to perform feature-based distillation during the pre-training phase of MAE.

[0044] The technical solution of this invention is based on VideoMAE, so let’s first review the workflow of VideoMAE.

[0045] VideoMAE extends the mask autoencoder from the image domain to the video domain. Each video input is randomly sampled into a clip with T frames. Where T is the number of frames sampled, H is the pixel height of each frame, and W is the pixel width of each frame. The sampling stride τ is specifically set for the dataset.

[0046] 1. Image Patch Embedding: Due to the additional temporal dimension of video data, VideoMAE treats a 2×16×16×3 cube as a patch, i.e., joint spatiotemporal cube embedding. Then, a 3D convolutional neural network is used to process the patch, performing non-overlapping convolutions to obtain the total... Each image token has a dimension mapped to D dimensions, where This allows image tokens to be processed sequentially, with a total number of N image tokens.

[0047] 2. Masking Strategy: Due to information redundancy in video data, VideoMAE applies a higher masking rate (e.g., 90%). To further reduce information leakage, VideoMAE uses cube embedding to mask multiple frames. Specifically, a random binary mask mapping is first generated using image tokens as units. To ensure that a given marker in the spatial dimension is masked in all temporal dimensions, VideoMAE simply... Repeated in the time dimension This is used to obtain the final mask mapping. Then M is flattened into a binary one-dimensional sequence. Where 1 indicates that the marker needs to be masked, and 0 indicates that the marker is visible, let P vis Indicates the index of the visible marker.

[0048] 3. Encoder – Feature Extractor: The encoder is a standard ViT. The input to the encoder is a sequence of visible tokens with fixed one-dimensional positional encodings. To allow interaction between any two image tokens in the entire input sequence, VideoMAE applies a joint spatiotemporal attention mechanism. The latent features extracted by the encoder are represented as follows: When VideoMAE is applied to downstream tasks, the encoder acts as a feature extractor.

[0049] 4. Decoder – Pixel Reconstructor: The decoder's task is to reconstruct the input, which requires restoring the masked patches in pixel form. The decoder is typically shallower and narrower than the encoder. A subsequent linear layer maps the dimensions of the latent features to the width of the decoder, then concatenates the latent features with learnable masking labels under positional guidance, adding fixed one-dimensional positional encodings to the latent features and learnable masking features obtained from the encoder. The decoder is also a regular ViT with joint spatiotemporal attention; the output is aligned with the original video dimensions through a projection layer to obtain…

[0050] Objective Function: Following MAE, the pre-training task of VideoMAE is pixel reconstruction. It's important to note that it doesn't reconstruct all pixels, but only the masked parts. The loss function applied to pixel reconstruction is the mean squared error (MSE) loss, and the reconstruction target is normalized at the token level. The objective function is expressed as:

[0051] in This indicates the mask marker index.

[0052] Figure 1 A flowchart of an asymmetric mask distillation method for pre-training a small mask autoencoder according to an embodiment of the present invention is shown.

[0053] like Figure 1 As shown, the asymmetric mask distillation method for pre-training small mask autoencoders includes the following steps:

[0054] Step 1: Asymmetric masking is performed on the video inputs of the student encoder and teacher encoder to obtain multiple tube tokens as feature inputs to both the teacher encoder and student encoder. The masking rate of the teacher encoder's video input is less than that of the student encoder's video input. The multiple tube tokens form the input sequence. The length of the teacher encoder's input sequence is greater than the length of the student encoder's input sequence. The features of the student encoder are a subset of the features of the teacher encoder. The visible token indices of the student encoder are a subset of the visible token indices of the teacher encoder. The visible token index is the index of the tube token, referring to the i-th tube token, where i ≥ 1.

[0055] Downsampling video editing The input is video. After performing 3D patch embedding, an asymmetric masking strategy is applied to V to generate tube tokens for the student encoder and teacher encoder, respectively. The length of the input sequence for the student encoder is shorter than that for the teacher encoder, and the visible token indices of the student encoder are a subset of those of the teacher encoder.

[0056] Step 2: Perform feature alignment between the student encoder and the teacher encoder using a serial alignment method. The serial alignment method includes direct alignment and generative alignment performed sequentially. Direct alignment is used for features common to both the student encoder and the teacher encoder, while generative alignment is used for features present in the teacher encoder but not in the student encoder.

[0057] Step 3: While performing feature alignment, the decoder reconstructs the pixel values ​​of the masked portion of the video. The decoder uses the original features from the student encoder as well as trainable mask features to reconstruct the video pixels.

[0058] Feature alignment and video reconstruction are performed simultaneously. The overall loss function for the feature alignment and video reconstruction tasks is:

[0059] L total =L recon +L dir +L gen L recon It is the loss function for reconstructing the video, L dir It is the loss function for direct alignment, L gen It is the loss function for generating alignment.

[0060] Figure 2 A schematic diagram of an asymmetric masking strategy and a symmetric masking strategy according to an embodiment of the present invention is shown.

[0061] like Figure 2 As shown, the symmetric masking strategy applies the same mask rate to both the teacher encoder and the student encoder.

[0062] The asymmetry of the asymmetric masking strategy lies in the difference between the video inputs of the student encoder and the teacher encoder. Different masking methods are applied to the student encoder and the teacher encoder, and the unmasked patch of the student encoder is a subset of that of the teacher encoder. Specifically, the masking rate r of the video input of the teacher encoder... tea The mask rate r of the video input to the student encoder must be less than that of the student encoder. stu First, a mask matrix M is generated for the student encoder and the teacher encoder, and then the visible token index of the student encoder is obtained. in It is the token index of the student encoder. Where is the length of the input sequence for the student encoder, and N is the number of tubular tokens in the entire input sample. Meanwhile, the processing of the teacher encoder is similar to that of the student encoder, and the visible token index of the teacher encoder is... in It is the token index of the teacher encoder. is the length of the input sequence of the teacher encoder, and N is the number of tube tokens in the entire input sample. The visible token indices of the student encoder are a subset of the visible token indices of the teacher encoder. Therefore, the teacher encoder can obtain more contextual information while accurately preserving what the student encoder can receive. Since the student branch has a reconstruction task, this asymmetric masking strategy allows the teacher encoder to collect more contextual information for the student encoder's distillation, while ensuring the reconstruction difficulty of the student branch is maintained.

[0063] The following section provides a detailed introduction to the direct alignment method and the generative alignment method.

[0064] Direct alignment: Assuming that under a symmetric masking strategy, the input sequence lengths of the teacher encoder and the student encoder are the same, and they share the same set of visible token indices P. vis The features of the student encoder and teacher encoder obtained from a pair of corresponding layers are respectively represented as follows: Where D represents the dimension of the feature, l represents the layer index of the student encoder, and l * This represents the layer index of the teacher encoder corresponding to the student layer (e.g., a middle or last layer). Teacher encoders typically have a large feature dimension, thus requiring a projection function φ for alignment. d(·). This projection function can be a simple linear layer or a multilayer perceptron (MLP) with an activation function. The loss function for direct alignment of a single layer can be defined as:

[0065]

[0066] Where z(p) represents the p-th token extracted from the input sequence as a feature.

[0067] Generative alignment: Generative alignment can be applied when the input sequence lengths of the student encoder and the teacher encoder do not match. The length difference between the input sequences of the student encoder and the teacher encoder is represented as... The difference between the token indexes can be represented as: Use respectively This represents the features of the student encoder and teacher encoder obtained from a pair of corresponding layers. Since the feature dimensions of the student encoder and teacher encoder are not aligned, a simple linear layer φ is used. g (·) is used to map the dimensions of the student encoder's features. Since the teacher encoder has more features than the student encoder, it's necessary to utilize the student encoder's features to generate more features consistent with the teacher encoder's features. Specifically, a feature generator... It is a decoder-like structure with multi-head self-attention (MHA). Similar to a decoder, it requires input features (features from the student encoder) to be combined with a mask token z. m Connect them together and add a fixed one-dimensional position code (PE). The generation process can be described as follows:

[0068]

[0069]

[0070] It's worth noting that in a normal decoder structure, the mask token is repeated the same number of times as all invisible tokens. However, in the generated alignment, only the mask token is repeated. This reduces the generation of redundant features during the alignment process. Under the alignment strategy, only additional features from the teacher encoder are aligned, meaning features that exist only in the teacher encoder and not in the student encoder. MSE is used as the loss function for single-layer feature generation alignment.

[0071]

[0072] Serial alignment is where direct alignment and generative alignment share the same projection function, allowing the feature dimensions of the student encoder to match those of the teacher encoder. Specifically, for a specific layer l of the student encoder, the corresponding layer of the teacher encoder is l. *In the forward computation, the features of the student encoder and the teacher encoder can be extracted as follows: The first step in direct alignment is to apply the projection function φ(·) to the student encoder to align the feature dimensions of the teacher encoder. In practice, a linear projection layer was used to obtain the projected features of the student encoder. For direct alignment. Then, the projected features from the student encoder are continuously used to generate features through a generator for alignment, wherein the features generated by the generator include:

[0073] Where z m This represents a masked token, where PE is a one-dimensional positional encoding.

[0074] This serial alignment can appropriately reduce the difficulty of generating the alignment task.

[0075] An asymmetric masking scheme is applied between the student encoder and the teacher encoder, resulting in two types of features: features present in both the student and teacher encoders, and features present only in the teacher encoder. A serial alignment method is used to align these two types of features. For the first type of features, direct alignment is used. For the second type of features, the projected features from the student encoder are fed to a decoder-like generator to generate features, which are then aligned with features visible only to the teacher encoder. The length of the generator's input sequence is based on the number of visible tokens encoded by the teacher, meaning the student encoder does not need to generate all mask tokens like a decoder. By incorporating positional encoding during feature generation, the generator ensures the student's predictive ability while reducing computational overhead. This serial alignment appropriately reduces the difficulty of the generation alignment task.

[0076] When performing feature alignment, one or more layers can be selected. In one embodiment of the invention, features from the middle and last layers are selected, see [link to relevant documentation]. Figure 1 Because the feature distributions differ across layers, the alignment parameters are not shared between layers. Therefore, when aligning multi-layer features, the direct alignment loss function can be rewritten as:

[0077] Where φ l This represents the projection function of the l-th layer.

[0078] When aligning multi-layer features, the alignment loss function is rewritten as follows:

[0079] in This represents the generator of the l-th layer.

[0080] Figure 3A schematic diagram illustrating direct alignment, generative alignment, parallel alignment, and serial alignment according to an embodiment of the present invention is shown.

[0081] like Figure 3 There are four different feature alignment strategies for asymmetric masking, with serial alignment being the default.

[0082] If the direct alignment method is used to align features between the student encoder and the teacher encoder, only features common to both encoders will be aligned. The direct alignment method uses a two-layer MLP for alignment, where the linear projection layer has the same dimension as the teacher encoder. It is worth noting that the features to be aligned have not yet been normalized.

[0083] If a generative alignment method is used to align features between the student encoder and the teacher encoder, features are first generated using the existing features of the student encoder. Then, the generated features are aligned with the additional features of the teacher encoder relative to the student encoder. The generative alignment method employs a decoder-like generator to align the teacher encoder's features, where the number of mask tokens is greater than the number of tube tokens possessed by the teacher encoder than by the student encoder. Before inputting the student encoder's features into the generator, a linear projection layer (which can be simply called a linear layer) is needed to align the teacher's dimensions. The features used to calculate the alignment loss are also the additional features of the teacher encoder relative to the student encoder.

[0084] Parallel alignment is a simple combination of direct alignment and generative alignment, where the direct alignment part uses only a simple linear projection layer. It's important to note that the linear projection layers for direct and generative alignment do not share parameters. Specifically, direct and generative alignment are performed simultaneously. In direct alignment, the linear projection layer is used to obtain the projected features of the student encoder for alignment. In generative alignment, the linear projection layer is used to obtain the projected features of the student encoder and generate features from these features to align with the features unique to the teacher encoder. The parameters of the linear projection layers for direct and generative alignment are not shared.

[0085] For serial alignment, direct alignment and generative alignment are combined serially, and the linear projection layers of both alignments share parameters, which reduces the difficulty of generative alignment. Specifically, direct alignment and generative alignment are performed sequentially. Direct alignment is performed first, during which the projection features of the student encoder are obtained. Subsequently, features are generated using these projection features to align with the features unique to the teacher encoder.

[0086] An asymmetric mask distillation architecture (AMD) for a small-scale mask autoencoder was established using the aforementioned asymmetric mask distillation method. Figure 1As shown, the overall framework of AMD is a two-stream distillation structure with a student branch and a teacher branch, where the teacher branch is a larger MAE pre-trained model. Notably, AMD only distills with the encoder portion of the teacher branch, while the student branch requires the decoder to complete the reconstruction task. Feature distillation occurs between the corresponding layers of the student and teacher encoders, with AMD responding to asymmetric masks using both direct alignment and generative alignment in a serial manner.

[0087] To verify the feasibility of the aforementioned asymmetric mask distillation method, extensive experiments were conducted on 32 A100-80G GPUs. The teacher encoder used was the ViT-Large model from VideoMAE.

[0088] (1) Experimental results on the Something-Something V2 dataset: When both ViT-Small models are used as inference models, the model trained using the above asymmetric mask distillation method achieves an action classification accuracy of 70.2%, outperforming VideoMAE by 3.4%. When both ViT-Base models are used as inference models, the above asymmetric mask distillation method achieves an action classification accuracy of 73.3%, outperforming VideoMAE by 3.7%, and only 1.0% lower than the teacher model.

[0089] (2) Experimental results on the Kinetics-400 dataset: When both ViT-Small models are used as inference models, the model trained using the above asymmetric mask distillation method achieves an action classification accuracy of 80.1%, which is 1.1% better than VideoMAE. When both ViT-Base models are used as inference models, the model trained using the above asymmetric mask distillation method achieves an action classification accuracy of 82.2%, which is 2.2% better than VideoMAE.

[0090] (3) Transfer performance on the AVA 2.2 dataset: Using the ViT-Base model as the inference model, after fine-tuning on Kinetics-400, the model trained using the above asymmetric mask distillation method can achieve 33.5% mAP, which is 1.7% better than VideoMAE.

[0091] Compared with the MAE pre-trained distillation scheme of DMAE, the above-mentioned asymmetric mask distillation method has the following advantages:

[0092] (1) DMAE uses a symmetric masking method, which reduces the inference cost of the teacher encoder, but is suboptimal for the distillation target. The asymmetric mask distillation method mentioned above uses an asymmetric masking method, which lowers the masking rate of the teacher encoder. This allows the teacher encoder to integrate more contextual information from the video to benefit the distillation process. In 800 training rounds on the SSV2 dataset, the asymmetric masking method outperforms the symmetric masking method by 0.4%.

[0093] (2) When using a multi-layer alignment strategy, DMAE uses the same MLP structure to handle different alignment layers, while the above-mentioned asymmetric mask distillation method uses multiple sets of alignment structures that do not share parameters to align different numbers of layers. This approach is beneficial for different distributions of different layers.

[0094] (3) DMAE has not verified the addition of distillation structure during MAE pre-training for longer training rounds, and has only performed a rough verification in the image domain. In contrast, the above-mentioned asymmetric mask distillation method has been fully experimentally verified in the video domain for training rounds, and has only lagged behind the teacher encoder by 1% at 800 training rounds.

[0095] Furthermore, the embodiments can be provided as computer program products that may include one or more machine-readable media on which machine-executable instructions are stored, which, when executed by one or more machines such as a computer, computer network, or other electronic equipment, may cause one or more machines to perform operations according to the embodiments of the present invention. Machine-readable media may include, but are not limited to, floppy disks, optical disks, CD-ROMs (compact disc read-only memory) and magneto-optical disks, ROMs (read-only memory), RAMs (random access memory), EPROMs (erasable programmable read-only memory), EEPROMs (electrically erasable programmable read-only memory), magnetic or optical cards, flash memory, or other types of media / machine-readable media suitable for storing machine-executable instructions.

[0096] Furthermore, various embodiments can be downloaded as computer program products, wherein the program can be transmitted from a remote computer (e.g., a server) to a requesting computer (e.g., a client) via a communication link (e.g., a modem and / or a network connection) using one or more data signals implemented and / or modulated by a carrier wave or other propagation medium. Therefore, the machine-readable medium used herein may include such a carrier wave, but this is not required.

[0097] While some embodiments of the present invention have been described in this application, those skilled in the art will understand that these embodiments are merely illustrative. Numerous variations, alternatives, and improvements will arise in those skilled in the art under the teachings of this invention without departing from its scope. The appended claims are intended to define the scope of the invention and thereby cover methods and structures within the scope of the claims themselves and their equivalents.

Claims

1. An asymmetric mask distillation method for pre-training a small mask autoencoder, characterized in that, include: Asymmetric masking is applied to the video inputs of the student encoder and the teacher encoder to obtain multiple tube tokens as feature inputs to the teacher encoder and the student encoder, wherein the masking rate of the video input of the teacher encoder is less than that of the video input of the student encoder; the multiple tube tokens form an input sequence, and the length of the input sequence of the teacher encoder is greater than the length of the input sequence of the student encoder; the features of the student encoder are a subset of the features of the teacher encoder. The visible token index of the student encoder is a subset of the visible token index of the teacher encoder. ; as well as Perform feature alignment between the student encoder and the teacher encoder; Feature alignment between the student encoder and the teacher encoder is performed using a serial alignment method, wherein the serial alignment method includes direct alignment and generative alignment performed sequentially. Direct alignment is used for feature alignment for features common to both the student encoder and the teacher encoder, while generative alignment is used for feature alignment for features that exist in the teacher encoder but not in the student encoder. The feature alignment between the student encoder and the teacher encoder using the serial alignment method includes: For a specific layer of the student encoder The layer corresponding to the teacher encoder is The features of the student encoder and the teacher encoder were extracted as follows: ; The first step in direct alignment is to use the projection function. This is applied to the student encoder to align the feature dimensions of the teacher encoder, where a linear projection layer is used to obtain the projected features of the student encoder. Used for direct alignment; as well as The projected features of the student encoder are continuously used to generate features through the generator for alignment.

2. The asymmetric mask distillation method for pre-training a small mask autoencoder according to claim 1, characterized in that, The asymmetric masking of the video inputs of the student encoder and the teacher encoder includes: Generate mask matrices for the student encoder and teacher encoder, and then obtain the visible token index of the student encoder. and the visible token index of the teacher encoder ,in , , It is the token index of the student encoder. It is the token index of the teacher encoder. It is the length of the input sequence of the student encoder. is the length of the input sequence of the teacher encoder, and N is the number of tubular tokens in the entire input sample. This represents the mask rate of the video input to the teacher encoder. This represents the mask rate of the video input to the student encoder.

3. The asymmetric mask distillation method for pre-training a small mask autoencoder according to claim 2, characterized in that, The generation of features via a generator includes: ,in This represents a masked token, where PE is a one-dimensional positional encoding.

4. The asymmetric mask distillation method for pre-training a small mask autoencoder according to claim 3, characterized in that, When aligning multi-layer features, the direct alignment loss function is: ,in Indicates the first l The projection function of the layer, It is the length of the input sequence of the student encoder. This represents the p-th token extracted from the input sequence as a feature; as well as When aligning multi-layer features, the generated alignment loss function is: ,in Indicates the first l Layer generator, It is the length of the sequence that the teacher encoder inputs compared to the student encoder. This represents the p-th token extracted from the input sequence as a feature.

5. The asymmetric mask distillation method for pre-training a small mask autoencoder according to claim 2, characterized in that, This also includes reconstructing the pixel values ​​of the masked portion of the video during feature alignment, where the overall loss function for both the feature alignment and video reconstruction tasks is: ,in It is the loss function for reconstructing the video. It is the loss function for direct alignment. It is the loss function for generating alignment.

6. A computer-readable storage medium having a computer program stored thereon, the computer program implementing the steps of the method as claimed in any one of claims 1-5 when executed by a processor.

Images