Motion guided mask method and pre-training method of visual transformer model

Abstract

The application discloses a motion-guided mask method, which selects a frame in a video segment as a reference frame, obtains a mask graph of the reference frame by using a mixed Gaussian distribution, extracts optical flow to both sides respectively from the reference frame as a starting point, then generates the mask graph of the video segment frame by frame based on the optical flow from the reference frame, finally divides the mask graph according to preset video cubic block size, sums the values in each video cubic block, and regards the position with the maximum mask ratio as a mask position and the rest as visible non-mask positions in the time sequence dimension, so as to complete video masking. The method can better inhibit information leakage by dynamically adjusting the mask positions of different frames according to object motion.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a motion-guided masking method and a pre-training method for a visual transformer model using the motion-guided masking method. Background Technology

[0002] The Vision Transformer (ViT) model has been widely applied to a range of tasks in image and video domains, including image classification, object detection, semantic segmentation, object tracking, action recognition, and spatiotemporal detection, achieving excellent performance. Due to the high computational power of ViT, these models typically require pre-training on large-scale datasets to reduce the risk of overfitting. Furthermore, since existing video datasets are usually relatively small, the efficiency requirements for video model pre-training methods are even higher. Early video models often directly adopted image-based Transformer model pre-training, where the model was first pre-trained on large-scale image datasets and then transferred to video models. This pre-training approach limited the spatiotemporal representation capabilities of the video models.

[0003] Masking modeling is a mainstream unsupervised pre-training method in natural language processing and computer vision. To improve the spatiotemporal representation capabilities of video models, some research has introduced masked auto-encoders (MAEs) into the video domain. MAEs are schemes that can be used to mask and reconstruct image input blocks, thereby enabling the pre-training of Vision Transformer (ViT) models, and have achieved great success in the image domain. However, due to the additional temporal dimension of video, exhibiting temporal redundancy and correlation, transferring MAEs to the video domain in practice requires additional design considerations, such as using extremely high masking ratios, such as 90%, to improve pre-training performance.

[0004] VideoMask Autoencoder (VideoMAE) is a video masking scheme based on MAE that introduces masking modeling into the pre-training of video models. Considering the redundancy and temporal correlation of video information, VideoMAE proposes a tube masking strategy. Tube masking involves masking image patches at the same location across all frames of a video segment. This approach aims to mask the same objects in adjacent frames, thereby reducing information leakage, increasing the difficulty of pre-training, and improving model performance. However, due to object or camera motion, the same object often appears in different locations across different frames, causing the tube masking strategy to fail to suppress information leakage in many cases. Summary of the Invention

[0005] To address some or all of the problems in the prior art, the first aspect of the present invention provides a motion guidance masking method, comprising:

[0006] Select one frame from the video segment as the reference frame;

[0007] The mask image of the reference frame is obtained using a Gaussian mixture distribution;

[0008] Starting from the reference frame, optical flow is extracted to both sides respectively;

[0009] Based on the optical flow, starting from the reference frame, a mask image of the video segment is generated frame by frame; and

[0010] The mask image is divided according to the preset video cube size, and the values ​​in each video cube are summed. In the temporal dimension, the position with the largest mask ratio equal to the preset value is taken as the masked position, and the remaining positions are regarded as visible and unmasked positions.

[0011] Furthermore, the intermediate frame of the video segment is used as the reference frame.

[0012] Further, obtaining the mask image of the reference frame includes:

[0013] Random selection For each patch, a Gaussian distribution is generated centered at the midpoint of each patch to obtain a mixture Gaussian distribution;

[0014] Regularize the mixture Gaussian distribution; and

[0015] Obtain the probability density function of the regularized Gaussian mixture distribution, and use the probability density function as a mask.

[0016] Furthermore, The value of is calculated using the following formula:

[0017]

[0018] Where ρ is the mask ratio, H is the height of the frame image, and W is the width of the frame image.

[0019] Furthermore, the size of the image patch is 16 pixels × 16 pixels.

[0020] Furthermore, the optical flow is obtained by recursively transforming the entire field using a small model version, starting from the reference frame and proceeding in reverse to both sides.

[0021] Furthermore, the mask image of the video segment is obtained by twisting the mask image of the reference frame forward and backward.

[0022] Furthermore, the method also includes:

[0023] For holes caused by reverse distortion, fill in the corresponding values ​​on the mask map of the reference frame.

[0024] Furthermore, the preset video cube size is 2 pixels × 16 pixels × 16 pixels.

[0025] Furthermore, the preset value for the mask ratio is 90%.

[0026] Based on the motion-guided masking method described above, a second aspect of the present invention provides a pre-training method for a visual transformer model, comprising:

[0027] The video clip is divided into non-overlapping cubic blocks, and the cubic blocks are embedded to obtain video embedding blocks;

[0028] The video embedding block is masked using the motion-guided masking method described above.

[0029] The unmasked video embedding block is fed into the encoder to extract features;

[0030] The obtained features and learnable mask blocks are input into the decoder to recover and reconstruct the masked video cube; and

[0031] Calculate the mean squared error loss between the reconstructed video cube and the real video cube, and calculate the parameter gradient to update the model.

[0032] This invention provides a motion-guided masking method that dynamically adjusts the mask position in different frames based on object motion, thereby better suppressing information leakage. The pre-training method for the visual transformer model using this motion-guided masking method increases the difficulty of pre-training, reduces the risk of overfitting during model pre-training, and ultimately improves the pre-training effect of the video model, enhancing its performance, such as classification accuracy. Attached Figure Description

[0033] To further illustrate the above and other advantages and features of the various embodiments of the present invention, a more specific description of the various embodiments of the present invention will be presented with reference to the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. In the drawings, identical or corresponding parts will be indicated by identical or similar reference numerals for clarity.

[0034] Figure 1 A flowchart illustrating a motion-guided masking method according to an embodiment of the present invention is shown.

[0035] Figure 2 This diagram illustrates a process of a motion-guided masking method according to an embodiment of the present invention.

[0036] Figure 3 This diagram illustrates the effect of a motion-guided masking method according to an embodiment of the present invention.

[0037] Figure 4 A schematic diagram showing the results of ablation experiments under different mask ratios is provided.

[0038] Figures 5a-5d The images shown are the original video frame image, the masked frame image obtained by the pipeline masking strategy, the random masking strategy, and a motion-guided masking method according to an embodiment of the present invention.

[0039] Figure 6 A flowchart illustrating a pre-training method for a visual transformer model according to an embodiment of the present invention is shown; and

[0040] Figure 7 This diagram illustrates the process of pre-training a visual transformer model using an embodiment of the present invention. Detailed Implementation

[0041] In the following description, the invention is described with reference to various embodiments. However, those skilled in the art will recognize that the embodiments may be practiced without one or more specific details or in conjunction with other alternatives and / or additional methods or components. In other instances, well-known structures or operations are not shown or described in detail so as not to obscure the inventive points of the invention. Similarly, for illustrative purposes, specific numbers and configurations are set forth to provide a comprehensive understanding of embodiments of the invention. However, the invention is not limited to these specific details.

[0042] In this specification, references to "an embodiment" or "this embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. The phrase "in one embodiment" appearing throughout this specification does not necessarily refer to the same embodiment in all instances.

[0043] It should be noted that the embodiments of the present invention describe the method steps in a specific order; however, this is only for illustrating the specific embodiment and not for limiting the order of the steps. On the contrary, in different embodiments of the present invention, the order of the steps can be adjusted according to actual needs.

[0044] VideoMAE, as an unsupervised pre-training method for video models, can improve pre-training effectiveness to some extent by addressing temporal redundancy and correlation in videos, while also suppressing some information leakage, increasing pre-training difficulty, and ultimately improving pre-training results. However, the pipeline masking strategy used in VideoMAE is based on the assumption that "objects are relatively stationary or have only minor motion in adjacent frames," so VideoMAE cannot handle videos with high-speed motion and cannot effectively suppress information leakage. Based on this, this invention improves upon VideoMAE, resulting in a motion-guided masking video mask autoencoder, MGMAE. Specifically, it replaces the pipeline masking strategy in VideoMAE with a motion-guided masking method. This motion-guided masking method dynamically adjusts the mask position for each frame based on the object's motion information, thereby better suppressing information leakage, increasing pre-training difficulty, and ultimately improving the pre-training effect of the video model.

[0045] The present invention will be further described below with reference to the accompanying drawings of the embodiments.

[0046] Figure 1 and Figure 2 The figure illustrates a flowchart and process diagram of a motion-guided masking method according to an embodiment of the present invention. As shown in the figure, a motion-guided masking method includes:

[0047] First, in step 101, a reference frame is selected. A frame from the video clip is chosen as the reference frame. Due to object or camera motion, objects appearing in the reference frame may disappear in a later frame. When warping the mask image based on optical flow, holes may appear in the warped mask image due to pixels mapping out of bounds. Therefore, to better reduce information loss, in one embodiment of the present invention, the middle frame of the video clip is used as the reference frame, and the image is warped in both forward and backward directions starting from the middle frame. This can minimize the information loss caused during image warping. However, it should be understood that in other embodiments of the present invention, any frame from the video clip can also be selected as the reference frame. To verify the effect of the middle frame, an ablation experiment was conducted on the selection of the reference frame. The experiment showed that the model pre-trained using the first frame as the reference frame achieved a classification accuracy of 70.5% on the SSV2 (Something-Something V2) dataset, lower than the 71.0% accuracy when the middle frame was used as the reference frame.

[0048] Next, in step 102, an initial mask image is generated. The mask image of the reference frame is obtained as the initial pixel-level mask image. In existing technologies, such as pipeline masks, random masks, and single-time masks, cubic-level binary initialization is used, meaning that within each 2-pixel × 16-pixel × 16-pixel cube, the mask image value is either 0 or 1. This cubic-level initialization method disrupts the continuity of object textures. To address this problem, in one embodiment of the present invention, a Gaussian mixture distribution is used to obtain the mask image of the reference frame, specifically including:

[0049] Random selection A Gaussian distribution is generated centered on the midpoint of each map patch to obtain a Gaussian mixture distribution. In one embodiment of the present invention, The value is determined by the following:

[0050] The formula yields the following results:

[0051]

[0052] Where ρ is the mask ratio, H is the height of the frame image, and W is the width of the frame image. In one embodiment of the present invention, the size of the patch is 16 pixels × 16 pixels;

[0053] Regularize the mixture Gaussian distribution; and

[0054] Obtain the probability density function of the regularized Gaussian mixture distribution, and use the probability density function as the mask image of the reference frame.

[0055] However, it should be understood that in other embodiments of the present invention, other methods, such as random masking, can also be selected for mask image initialization. To verify the effect of using Gaussian mixture distribution for mask image initialization, the present invention conducted a comparative experiment with other initialization methods. The experiment found that the classification accuracy of the model obtained by random 0-1 binary initialization on image blocks and subsequent masking and pre-training was 70.9% on SSV2, and the classification accuracy of the model obtained by random 0-1 binary initialization on pixels and subsequent masking and pre-training was 70.8%, both lower than the 71.0% of the Gaussian mixture distribution initialization scheme.

[0056] Simultaneously, in step 103, optical flow is extracted. Starting from the reference frame, optical flow is extracted to both sides. In one embodiment of the invention, Recurrent All-Pairs Field Transformers (RAFT) is used to extract the optical flow. Furthermore, for lightweight and efficient operation, a smaller version of RAFT, i.e., RAFT-small, is preferred, extracting optical flow to both sides starting from the reference frame. Since in this embodiment of the invention, the image is preferably warped using a reverse warping method, it is preferable to extract the optical flow in reverse, resulting in:

[0057] as well as

[0058]

[0059] Among them, I i υ represents the i-th frame of the input video clip. i→i-1 υ i→i+1 These represent the optical flow from the i-th frame to the previous frame and the next frame, respectively.

[0060] Furthermore, in embodiments of the present invention, the impact of optical flow extraction quality on the masking scheme can be simulated by changing the number of iterations of RAFT. In embodiments of the present invention, the default number of iterations is 12, and the final masked and pre-trained model achieves a classification accuracy of 71.0% on SSV2. When the number of iterations is 6 and 24, the classification accuracies on SSV2 are 71.3% and 71.0%, respectively. It can be seen that because the training dataset of RAFT and the SSV2 dataset have different data distributions, the impact of optical flow quality on the results is relatively small. More iterations introduce noise, while fewer iterations actually improve classification accuracy.

[0061] Next, in step 104, the mask image is distorted. Based on the optical flow, starting from the reference frame, a mask image for the video segment is generated frame by frame. Specifically, the mask image obtained from the reference frame is distorted forward and backward using optical flow. Image distortion is divided into forward distortion and backward distortion. Forward distortion encounters serious hole and conflict problems. The hole problem refers to the situation where no optical flow can map a pixel to a certain position during the distortion process, thus forming a hole. The conflict problem refers to the situation where multiple optical flow vectors map multiple pixels to the same position during the distortion process. Reverse distortion seeks a one-to-one correspondence between pixels in two images, so there is no conflict problem. Although there are still holes caused by pixels being mapped out of bounds, the number of holes generated is usually less than that of forward distortion, and they are mainly concentrated at the boundaries of the images, resulting in less damage to the distribution of image information. To verify this, the present invention conducted ablation experiments on the choice of distortion method. The experiments showed that the model obtained by using forward distortion and then subsequent masking and pre-training achieved a classification accuracy of 70.5% on SSV2, which is lower than the 71.0% of the model obtained by backward distortion. Based on this, in embodiments of the present invention, it is preferable to use a mask image of the video segment obtained by twisting the mask image of the reference frame in both forward and backward directions:

[0062]

[0063] in:

[0064] φ B Indicates the reverse twist operation;

[0065] υ i→i-1 υ i→i+1 Let i represent the optical flow from the i-th frame to the previous frame and the next frame, respectively; and

[0066] M i+1 M i-1 These represent the mask images for frames i+1 and i-1, respectively.

[0067] In one embodiment of the present invention, for holes caused by reverse distortion, the corresponding values ​​on the reference frame mask are filled in according to the pipe masking strategy. It should be understood that in other embodiments of the present invention, random values, 0 (representing invisible), or 1 (representing visible) can also be filled into the holes. However, the subsequent masking and pre-training of the resulting model on SSV2 yields lower classification accuracy than the values ​​at the corresponding positions on the reference frame mask. Experiments show that the schemes filling with random values, 0, and 1 achieve final classification accuracies of 70.8%, 70.7%, and 70.8%, respectively, lower than the default scheme (filling with values ​​at the corresponding positions on the reference frame mask) at 71.0%.

[0068] Finally, in step 105, visible video blocks are sampled. After obtaining the mask image of the entire input video segment, the mask image is divided according to a preset video cube size, and the values ​​within each video cube are summed. In the temporal dimension, the position with the largest mask ratio equal to the preset value is taken as the occlusion position, and the remaining positions are considered as visible non-occluded positions. In one embodiment of the present invention, the preset video cube size is 2 pixels × 16 pixels × 16 pixels. To determine the optimal preset mask ratio value, the present invention conducted an ablation experiment on the mask ratio, and the experimental results are as follows. Figure 4 As shown in the figure, a masking ratio of 85% results in the highest classification accuracy of the subsequently trained model on SSV2, but the difference in classification accuracy compared to a 90% masking ratio is not significant. However, as the masking ratio decreases, the computational cost during model pre-training also increases. Therefore, in the embodiments of this invention, a high masking ratio of 90% is preferred.

[0069] Figure 3 This diagram illustrates the effect of a motion-guided masking method according to an embodiment of the present invention. As shown, the mask image of the video segment obtained using the motion-guided masking method allows the mask position in each frame to change according to the motion information of the object, such as the bottle cap shown in the figure. Furthermore, Figures 5a-5d The figures show the original video frame image, the masked frame image obtained using a pipeline masking strategy, a random masking strategy, and a motion-guided masking method according to an embodiment of the present invention. As shown in the figures, compared with other masking strategies, the masking method in this embodiment of the present invention can change according to the trajectory of the ball in different frames, thereby better suppressing information leakage.

[0070] Based on the motion-guided masking method described above. Figure 6 and Figure 7 The following figures illustrate the flowchart and process diagram of a pre-training method for a visual transformer model according to an embodiment of the present invention. As shown in the figures, a pre-training method for a visual transformer model includes:

[0071] First, in step 601, cubic embedding. The video segment I of size T×3×H×W is divided into non-overlapping cubic blocks. By embedding these cubes into a cube, a video embedding block is obtained. Where D is the number of channels in the embedded block;

[0072] Next, in step 602, motion-guided masking. Using the motion-guided masking method described above, the mask position of each frame is dynamically adjusted based on the object motion information represented by optical flow, and the masking position of the video embedding block T, which is 90% masked, is denoted as M;

[0073] Next, in step 603, feature extraction occurs. For efficient computation, in this embodiment of the invention, only the visible video embedding block is fed into the encoder for feature extraction, and the encoder outputs the visible video embedding block T. v The corresponding feature Z. The visible video embedding block is... That is, all video embedding blocks that are not occluded. In one embodiment of the invention, the encoder employs the original ViT model with a joint attention mechanism;

[0074] Next, in step 604, the reconstruction is performed. The visible video embedding block T obtained in step 603 is then used. v The corresponding feature Z and the learnable mask block are input into the decoder to recover and reconstruct the occluded video cube. In one embodiment of the invention, the decoder is a narrower and shallower ViT model compared to the encoder, and employs a transformer structure with the same joint attention mechanism as the encoder; and

[0075] Finally, in step 605, the model is updated. The mean squared error loss between the reconstructed video cubes and the real video cubes is calculated, and the parameter gradients are computed to update the model. The purpose of pre-training is to minimize the reconstructed cubes. The mean squared error loss between the actual video cube C and the real video cube C is calculated only on the occluded cube in this embodiment of the invention, and is ultimately used to calculate the parameter gradient and update the model. The formula for calculating the mean squared error is as follows:

[0076] To verify the effectiveness of the motion-guided masking method and the visual transformer model pre-training method used in MGMAE of this invention, experiments were conducted on the Kinetics-400 and Something-Something V2 datasets using the ViT-base model as the encoder, and the results were compared with VideoMAE. The experiments showed that MGMAE's loss during pre-training was greater than VideoMAE, indicating that its motion-guided masking method increased the difficulty of pre-training and reduced the risk of model overfitting. The experiments also found that the model pre-trained by MGMAE, after fine-tuning on the downstream action recognition task, outperformed VideoMAE, demonstrating that its motion-guided masking method effectively improves model performance.

[0077] The specific experimental results are as follows: MGMAE, after 800 pre-training epochs on the Kinetics-400 dataset, achieved a pre-training loss of 0.6462, higher than VideoMAE's 0.5875. After fine-tuning, its classification accuracy reached 81.2%, higher than VideoMAE's 80.0%. After 1600 pre-training epochs on the Kinetics-400 dataset, its pre-training loss was 0.6378, higher than VideoMAE's 0.5890. After fine-tuning, its classification accuracy reached 81.8%, higher than VideoMAE's 81.5%. After 800 pre-training epochs on the Something-Something V2 dataset, its pre-training loss was 0.5820, higher than VideoMAE's 0.5278. After fine-tuning, its classification accuracy reached 71.0%, higher than VideoMAE's 69.6%. On the Something-Something V2 dataset… The dataset was pre-trained for 2400 epochs on the V2 dataset, with a pre-training loss of 0.5659, which is higher than VideoMAE's 0.5122. After fine-tuning, the classification accuracy was 72.3%, which is higher than VideoMAE's 70.8%.

[0078] Although various embodiments of the invention have been described above, it should be understood that they are presented by way of example only and not as limitations. It will be apparent to those skilled in the art that various combinations, modifications, and alterations can be made without departing from the spirit and scope of the invention. Therefore, the breadth and scope of the invention disclosed herein should not be limited by the exemplary embodiments disclosed above, but should be defined solely by the appended claims and their equivalents.

Claims

1. A motion-guided masking method, characterized in that, Including the following steps: Select one frame from the video segment as the reference frame; The mask image of the reference frame is obtained using a Gaussian mixture distribution; Starting from the reference frame, optical flow is extracted to both sides respectively; Based on the optical flow, starting from the reference frame, a mask image of the video segment is generated frame by frame; and The video segment's mask image is divided according to a preset video cube size, and the values ​​of the mask image corresponding to each video cube are summed. In the temporal dimension, the position in the mask image corresponding to each video cube with a mask ratio equal to a preset value and the largest mask value is taken as the occlusion position, and the remaining positions are regarded as visible non-occluded positions, so as to complete the video masking.

2. The motion-guided masking method as described in claim 1, characterized in that, The middle frame of the video segment is used as the reference frame.

3. The motion-guided masking method as described in claim 1, characterized in that, The mask image of the reference frame is obtained according to the following steps: Random selection For each patch, a Gaussian distribution is generated centered at the midpoint of each patch to obtain a mixture Gaussian distribution; Regularize the mixture Gaussian distribution; and Obtain the probability density function of the regularized Gaussian mixture distribution, and use the probability density function as a mask.

4. The motion-guided masking method as described in claim 3, characterized in that, The The value of is calculated using the following formula: ， in, For mask ratio, The height of the frame image, and The width of the frame image.

5. The motion-guided masking method as described in claim 3, characterized in that, The size of the image block is 16 pixels × 16 pixels.

6. The motion-guided masking method as described in claim 1, characterized in that, The optical flow is extracted from the reference frame in reverse order to both sides through a recursive full-pair field transformation of a small model version.

7. The motion-guided masking method as described in claim 1, characterized in that, The mask image of the video segment is obtained by twisting the mask image of the reference frame forward and backward.

8. The motion-guided masking method as described in claim 7, characterized in that, It also includes the following steps: For holes caused by reverse distortion, fill in the corresponding values ​​on the mask map of the reference frame.

9. The motion-guided masking method as described in claim 1, characterized in that, The preset video cube size is 2 pixels × 16 pixels × 16 pixels.

10. The motion-guided masking method as described in claim 1, characterized in that, The preset value for the mask ratio is 90%.

11. A pre-training method for a visual transformer model, characterized in that, include: The video clip is divided into non-overlapping cubic blocks, and the cubic blocks are embedded to obtain video embedding blocks; The video embedding block is masked using the motion-guided masking method as described in any one of claims 1 to 10; The unmasked video embedding block is fed into the encoder to extract features; The extracted features and learnable mask blocks are input into the decoder to recover and reconstruct the masked video cube. as well as Calculate the mean squared error loss between the reconstructed video cube and the real video cube, and calculate the parameter gradient to update the model.

Images