Ultra-high-definition video classification method based on dynamic adaptive masking and word lemma sparsity

Abstract

The present invention belongs to the technical field of ultra-high-definition video image processing and proposes an ultra-high-definition video classification method based on dynamic adaptive masking and word-element sparsity. The method comprises: using a random mask to randomly mask ultra-high-definition video frames according to a certain ratio, inputting the video frames after random masking into an encoder to obtain intermediate visual features; classifying the intermediate visual features obtained by the encoder to complete fine-tuning of the ultra-high-definition video frames; removing unimportant visual words in the current transformation layer according to decision probabilities and dynamic masks, passing the remaining important visual words to subsequent transformation layers for processing, and completing the classification of a video frame in the ultra-high-definition video; after completing the classification of all frames in a video, directly merging and summarizing the results obtained for each frame individually, and voting to select the class with the most categories as the final ultra-high-definition video classification result. The present invention can reduce the computational complexity of the ultra-high-definition video processing process and ensure computational accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of ultra-high-definition video image processing, and in particular to an ultra-high-definition video classification method based on dynamic adaptive masking and word-unit sparsity. Background Art

[0002] At present, due to the widespread use of urban roads, traffic congestion on urban roads is becoming more and more serious.

[0003] In recent years, visual transformers (VMTs) have become a key research topic in both the computer vision industry and academia due to their impressive performance. Compared to the previously popular convolutional neural network (CNN) model, ViT benefits from the introduction of a self-attention mechanism, enabling it to better establish global semantic dependencies within image patches, thereby achieving higher model accuracy. However, this also comes with higher computational complexity than CNNs. Furthermore, with the current development of the ultra-high-definition (UHD) video industry, UHD video has become an increasingly important source of data in the video field. Compared to lower-resolution standard-definition (SD) video, UHD video, with a resolution of 3840x2160 pixels or higher, consumes significantly more computational resources. Therefore, how to effectively improve the processing speed of existing ViT models for UHD video while maintaining high model accuracy is a key challenge in the application of deep neural network technology in UHD video processing.

[0004] Until now, convolutional neural networks (CNNs) have been the standard paradigm for processing images and videos. CNNs are distinguished by their local connections and weight sharing mechanisms. This design enables them to effectively identify and capture spatial features in images, gradually extracting high-level, abstract features through multiple layers of convolution and pooling operations. For example, CNN filters can start with pixel-level details and gradually learn mid-level features such as edges, shapes, and textures, ultimately recognizing more complex objects and scenes.

[0005] In recent years, with advances in deep learning, the Visual Transformer (ViT) has gradually emerged as a prominent force. ViT's unique use of a self-attention mechanism allows the model to capture interactions between image elements globally, rather than confined to local regions, thereby achieving a deep understanding of the image's global semantics. This capability has enabled ViT to achieve superior classification accuracy compared to CNNs in certain visual tasks. However, this enhanced performance does not come without a price. Compared to CNNs, ViT typically comes with higher computational complexity. For example, two common models, ViT-Base and ResNet-34, may achieve superior results on certain classification tasks, but ViT-Base requires significantly more computing resources and time for training and inference. This can be a significant burden on typical hardware. This issue becomes particularly acute when working with large data sources such as ultra-high-definition (UHD) video. UHD video's larger data size and high resolution quadratically increase the number of pixels contained in each frame, exacerbating ViT's computational complexity. This means that even state-of-the-art systems face significant challenges in real-time processing of UHD video. Therefore, seeking methods that can maintain or improve classification accuracy while being superior in computational efficiency has become one of the key issues that need to be urgently addressed in the field of computer vision.

[0006] Due to the Transformer model's unique self-attention mechanism, its overall computational complexity is quadratically related to the length of the input sequence. Consequently, existing ViT models consume significant resources when processing ultra-high-definition video. The most direct way to improve this problem is to sparsify the input image's token sequence, removing unimportant image tokens. This alleviates the computational performance bottleneck caused by the excessively long token sequences in ultra-high-definition video. However, while token sparsity-based acceleration can directly accelerate the network model, it can easily lead to a decrease in model accuracy due to varying degrees of information loss. Therefore, existing ViT-based technologies struggle to achieve both fast and good inference performance when processing ultra-high-definition video. Summary of the Invention

[0007] The purpose of the present invention is to provide an ultra-high-definition video classification method based on dynamic adaptive masking and word sparseness, aiming to solve the bottleneck problems of computing resources and computing time faced by the ViT-based video frame classification method in processing ultra-high-definition videos.

[0008] The present invention solves the technical problem and adopts the following technical solution:

[0009] The ultra-high-definition video classification method based on dynamic adaptive mask and word sparseness includes the following steps:

[0010] Random masks are used to randomly mask ultra-high-definition video frames at a certain ratio, and the masked video frames are input into the encoder to obtain intermediate visual features. This completes the unsupervised pre-training of the dynamic adaptive masked autoencoder transformer, enabling the dynamic mask branch to distinguish important visual words.

[0011] Classify the intermediate visual features obtained by the encoder to complete the fine-tuning of ultra-high-definition video frames;

[0012] Remove unimportant visual words from the current transformation layer according to the decision probability and dynamic mask, and pass the remaining important visual words to the subsequent transformation layer for processing to complete the classification of a video frame in the ultra-high-definition video;

[0013] After completing the classification of all frames in a video, the results obtained for each frame are directly merged and summarized, and the class with the most categories is voted as the final ultra-high-definition video classification result.

[0014] As a further optimization, before randomly masking the ultra-high-definition video frames according to a certain ratio using a random mask, the method further includes:

[0015] Suppose a video sample in the ultra-high-definition video dataset is ,use represent A frame in which Represents the number of video frames, Represent the height, width and number of channels of the input video frame respectively;

[0016] Sample the video frame Grid-slice the data into a patch sequence ,in, Represents the pixel size of the video frame patch, Represents the number of patches, that is, the length of the input sequence;

[0017] Will Mapped through a linear layer dimensional space .

[0018] As a further optimization, the random mask is used to randomly mask the ultra-high-definition video frame according to a certain ratio, and the masked video frame is input into the encoder to obtain the intermediate visual features, which means:

[0019] Let the random mask of the pre-training input be ,By randomly masking the video frame according to a certain ratio, the masked video frame is obtained ;

[0020] The masked video frame is input into the encoder composed of 12 transformer layers to obtain the intermediate visual features. .

[0021] As a further optimization, the dynamic adaptive mask layer is pre-set and inserted into the position of the corresponding Transformer layer in the Encoder to implement the dynamic adaptive mask strategy during the pre-training process;

[0022] During the pre-training process, Gumbel-Softmax is used to sample the score distribution to obtain the importance decision mask of the word unit. .

[0023] As a further optimization, during the pre-training process, masked words are not involved in the attention calculation process, and the multi-head self-attention formula is used to block unimportant visual words. The multi-head self-attention formula is expressed as:

[0024]

[0025] in, , Represents the query, key and value of the standard transformer layer, is the weight of the corresponding subspace mapping of the multi-head attention, is the dimension of each subspace.

[0026] As a further optimization, the intermediate visual features obtained Before inputting into the decoder, it is mapped to the original video frame length to obtain :

[0027]

[0028] in, Represents position coding information, Represents a special placeholder.

[0029] As a further optimization, we get Then, the intermediate visual features Input to the decoder, decoder output :

[0030] ;

[0031] In getting After that, the following dynamic adaptive pre-training loss function is used To train:

[0032]

[0033]

[0034]

[0035] in, is the number of dynamic mask branches, Represents the corresponding The target pruning rate of the dynamic mask branch, Treated as regularization for dynamic mask branches.

[0036] As a further optimization, during the fine-tuning of the ultra-high-definition video frame, a fully connected layer ECN is introduced to classify the intermediate visual features obtained by the encoder. The process is expressed as follows:

[0037]

[0038] Among them, avg represents the average pooling operation, and softmax represents the normalized exponential function;

[0039] In getting After that, the objective function of the fine-tuning stage for:

[0040]

[0041] in, Represents a tag, represents reconstruction losses, A token representing the predicted passive mask.

[0042] As a further optimization, the unimportant visual words in the current transformation layer are removed according to the decision probability and the dynamic mask, and the remaining important visual words are passed to the subsequent transformation layer for processing to complete the classification of a video frame in the ultra-high-definition video, including:

[0043] Order The cropping rate of the dynamic mask layer is , then the decision probability obtained Sort them and get the index of their word importance from large to small;

[0044] To the front Select the elements of to get the dynamic mask for the current transformation layer input , the specific formula is as follows:

[0045]

[0046] Based on dynamic mask , directly obtain the remaining word sequences with higher importance, and pass them to the subsequent transformation layer to complete the classification of a video frame in the ultra-high-definition video.

[0047] The beneficial effects of the present invention are: through the above-mentioned ultra-high-definition video classification method based on dynamic adaptive mask and word sparsity, on the one hand, an adaptive word sparsity technology is used to dynamically remove unimportant image words from each transformation layer to achieve the purpose of acceleration and reduce computing resources and computing time; on the other hand, in order to compensate for the loss of model accuracy caused by dynamic sparsity and to improve the final task accuracy as much as possible, the present invention proposes a ViT training method based on dynamic mask image modeling, which achieves a high-definition video image processing effect that is faster and more accurate than the original ViT. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flowchart of an ultra-high-definition video classification method based on dynamic adaptive masking and word sparseness in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0050] Example

[0051] See also Figure 1 This embodiment provides an ultra-high-definition video classification method based on dynamic adaptive masking and word sparseness, wherein the flow chart thereof is shown in FIG. Figure 1 , wherein the method comprises the following steps:

[0052] S1. Use random masks to randomly mask ultra-high-definition video frames at a certain ratio, and input the masked video frames into the encoder to obtain intermediate visual features. This completes unsupervised pre-training of the dynamic adaptive masked autoencoder transformer, enabling the dynamic mask branch to distinguish important visual words.

[0053] S2, classify the intermediate visual features obtained by the encoder and complete the fine-tuning of the ultra-high-definition video frame;

[0054] S3: Remove unimportant visual words from the current transformation layer according to the decision probability and dynamic mask, and pass the remaining important visual words to the subsequent transformation layer for processing to complete the classification of a video frame in the ultra-high-definition video;

[0055] S4. After completing the classification of all frames in a video, the results obtained for each frame are directly merged and summarized, and the class with the most categories is voted as the final ultra-high-definition video classification result.

[0056] It should be noted that this embodiment is mainly used for efficient video frame classification of ultra-high-definition videos with a resolution of 3840x2160 and above. This embodiment mainly includes an unsupervised pre-training phase, a supervised fine-tuning phase, and a testing phase. The following describes the functions and specific steps of each phase:

[0057] 1. Pre-training stage:

[0058] In this stage, the dynamic adaptive mask autoencoder transformer is unsupervisedly pre-trained to achieve the discriminative ability of the dynamic mask branch for important visual words and provide a trained initialization weight for the subsequent fine-tuning step.

[0059] Before pre-training, a video sample in the ultra-high-definition video dataset can be set as ,use represent A frame in which Represents the number of video frames, Represent the height, width and number of channels of the input video frame respectively. In order to facilitate the processing of the converter, the video frame samples are Grid-like slicing is performed to make it a patch sequence similar to the token sequence in natural language processing .in, Represents the pixel size of the video frame patch, represents the number of patches, i.e. the length of the input sequence. Mapped through a linear layer dimensional space .

[0060] In this embodiment, the goal of unsupervised pre-training is to randomly mask the input video frame and then predict the masked part of the video frame, thereby improving the network's perception of ultra-high-definition images. This is achieved through a deeper encoder and a shallower decoder. Here, let the random mask of the pre-training input be ,By randomly masking the video frame according to a certain ratio, the masked video frame is obtained The masked video frame is input into the encoder composed of 12 transformer layers to obtain the intermediate visual features. :

[0061]

[0062] Among them, the dynamic adaptive mask layer can be pre-set and inserted into the position of the corresponding Transformer layer in the Encoder, thereby realizing a dynamic adaptive mask strategy during the pre-training process.

[0063] Specifically, the dynamic adaptive mask layer uses the intermediate visual features obtained by the previous layer to make token importance decisions. Each dynamic adaptive mask layer is a small 3-layer fully connected network that is used to obtain the importance distribution of the input token of the current layer. During the training process, in order to ensure that the dynamic masking process is differentiable, the re-parameterization technique will be used. Gumbel-Softmax can be used to sample the score distribution to obtain the token importance decision mask. .

[0064] In order to prevent the masked tokens from participating in the attention calculation process during training, the following modified multi-head self-attention formula is used to block non-important tokens:

[0065]

[0066] in:

[0067] Here They represent the queries, keys, and values of the standard transformer layer. is the weight of the corresponding subspace mapping of the multi-head attention, is the dimension of each subspace.

[0068] The visual features obtained Before inputting into the decoder, it is mapped to the original video frame length to obtain This operation can be described by the following formula:

[0069]

[0070] in, Represents position coding information, Represents a special placeholder token. Next, the decoder will output :

[0071]

[0072] In getting After that, the following dynamic adaptive pre-training loss function is used To train:

[0073]

[0074]

[0075]

[0076] in, is the number of dynamic mask branches, Represents the corresponding The target pruning rate of the dynamic mask branch, This can be considered a regularization of the dynamic mask branch. By constraining the predictions of the dynamic mask branch, it enables the dynamic mask branch to distinguish the importance of tokens. Furthermore, thanks to the compact and flexible dynamic mask branch, it does not add too much additional computational effort.

[0077] It should be noted that the encoder of the network model in this embodiment is composed of multiple transformer layers, and the decoder is also composed of multiple transformer layers. However, the decoder has fewer layers than the encoder. On the one hand, due to the use of a masked training strategy, the encoder only needs to process a small number of tokens during training. On the other hand, because the decoder is a smaller network, it consumes fewer computing resources than a wider decoder network design, thus effectively suppressing the overall computational complexity.

[0078] 2. Supervised fine-tuning stage:

[0079] This stage introduces supervised signals based on the weights of the pre-training stage to achieve ultra-high-definition video classification.

[0080] In this embodiment, a new fully connected layer (FCN) is introduced in the fine-tuning stage to classify the intermediate features obtained by the encoder. The process can be expressed as follows:

[0081]

[0082] Among them, avg represents the average pooling operation, and softmax represents the normalized exponential function. After that, the objective function of the fine-tuning stage for:

[0083]

[0084] here Represents a tag, represents reconstruction losses, A token representing the predicted passive mask.

[0085] It is worth noting that compared to the pre-training stage, no random masks are applied to the initial video frames during fine-tuning. . Therefore, the output of the decoder is , which is consistent with the decoder output during pre-training To distinguish them.

[0086] 3. Testing phase:

[0087] This stage describes how the dynamic adaptive layer removes tokens during the test phase, thereby directly reducing the computational complexity of the model. During the test phase, some unimportant tokens will be removed according to the current decision probability. The remaining tokens are then passed to the subsequent transformation layers for processing, thereby reducing the performance consumption of the ViT model when processing ultra-high-definition video.

[0088] In this embodiment, the The cropping rate of the dynamic mask layer is , then the decision probability obtained Sort by tokens and get the index of their importance from large to small. Select the elements of to get the dynamic mask for the current transformation layer input The specific formula is as follows:

[0089]

[0090] Based on this mask , the remaining token sequences with higher importance can be directly obtained and passed to the subsequent transformation layer to achieve the classification of a certain video frame in the ultra-high-definition video.

[0091] Finally, after processing all the frames in a video Finally, the results obtained for each frame are directly combined and summarized, and the class with the most categories is voted as the final video classification result.

[0092] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. An ultra-high-definition video classification method based on dynamic adaptive masking and word sparseness, characterized by: The steps include: Random masks are used to randomly mask ultra-high-definition video frames at a certain ratio, and the masked video frames are input into the encoder to obtain intermediate visual features. This completes the unsupervised pre-training of the dynamic adaptive masked autoencoder transformer, enabling the dynamic mask branch to distinguish important visual words. Classify the intermediate visual features obtained by the encoder to complete the fine-tuning of ultra-high-definition video frames; Remove unimportant visual words from the current transformation layer according to the decision probability and dynamic mask, and pass the remaining important visual words to the subsequent transformation layer for processing to complete the classification of a video frame in the ultra-high-definition video; After completing the classification of all frames in a video, the results obtained for each frame are directly merged and summarized, and the class with the most categories is voted as the final ultra-high-definition video classification result; Before using random mask to randomly mask the ultra-high-definition video frame according to a certain ratio, the method further includes: assuming that a certain video sample in the ultra-high-definition video data set is ,use represent A frame in which Represents the number of video frames, Represent the height, width and number of channels of the input video frame respectively; the video frame samples Grid-slice the data into a patch sequence ,in, Represents the pixel size of the video frame patch, Represents the number of patches, that is, the length of the input sequence; Mapped through a linear layer dimensional space ; During the fine-tuning process of the ultra-high-definition video frame, a fully connected layer FCN is introduced to classify the intermediate visual features obtained by the encoder. The process is expressed as follows: Among them, avg represents the average pooling operation, and softmax represents the normalized exponential function; In getting After that, the objective function of the fine-tuning stage for: in, Represents a tag, represents reconstruction losses, represents the predicted passive masked word, Refers to the importance decision mask generated by the i-th dynamic adaptive mask layer, is the number of dynamic mask branches, is the regularization term representing the i-th dynamic mask branch.

2. The ultra-high-definition video classification method based on dynamic adaptive masking and word sparseness according to claim 1 is characterized in that: The method of randomly masking the ultra-high-definition video frames according to a certain ratio using random masks and inputting the masked video frames into the encoder to obtain intermediate visual features refers to: Let the random mask of the pre-training input be ,By randomly masking the video frame according to a certain ratio, the masked video frame is obtained ; The masked video frame is input into the encoder composed of 12 transformer layers to obtain the intermediate visual features. .

3. The ultra-high-definition video classification method based on dynamic adaptive masking and word sparseness according to claim 2 is characterized in that: The dynamic adaptive mask layer is pre-set and inserted into the position of the corresponding Transformer layer in the Encoder to implement the dynamic adaptive mask strategy during the pre-training process; During the pre-training process, Gumbel-Softmax is used to sample the score distribution to obtain the importance decision mask of the word unit. .

4. The ultra-high-definition video classification method based on dynamic adaptive masking and word sparseness according to claim 3 is characterized in that: During the pre-training process, the masked words are not involved in the attention calculation process, and the multi-head self-attention formula is used to block the unimportant visual words. The multi-head self-attention formula is expressed as: in, , Represents the query, key and value of the standard transformer layer, is the weight of the corresponding subspace mapping of the multi-head attention, is the dimension of each subspace.

5. The ultra-high-definition video classification method based on dynamic adaptive masking and word sparseness according to claim 2 is characterized in that: The visual features in the middle will be obtained Before inputting into the decoder, it is mapped to the original video frame length to obtain : in, Represents position coding information, Represents a special placeholder.

6. The ultra-high-definition video classification method based on dynamic adaptive masking and word sparseness according to claim 5 is characterized in that: get Then, the intermediate visual features Input to the decoder, decoder output : ； In getting After that, the following dynamic adaptive pre-training loss function is used To train: in, is the number of dynamic mask branches, Represents the corresponding The target pruning rate of the dynamic mask branch, Treated as regularization for dynamic mask branches.

7. The ultra-high-definition video classification method based on dynamic adaptive masking and word sparseness according to claim 1 is characterized in that: The method removes unimportant visual words in the current transformation layer according to the decision probability and the dynamic mask, passes the remaining important visual words to the subsequent transformation layer for processing, and completes the classification of a video frame in the ultra-high-definition video, including: Order The cropping rate of the dynamic mask layer is , then the decision probability obtained Sort them and get the index of their word importance from large to small; To the front Select the elements of to get the dynamic mask for the current transformation layer input , the specific formula is as follows: Based on dynamic mask , directly obtain the remaining word sequences with higher importance, and pass them to the subsequent transformation layer to complete the classification of a video frame in the ultra-high-definition video.

Images