ViT model lightweight method for video action recognition

Abstract

The invention discloses a ViT model lightweight method for video action recognition, which comprises the following steps of: performing motion estimation in a data processing stage, calculating to obtain motion intensity and time redundancy scores of tokens, and performing window division on video frames according to the scores. Afterwards, a window-global token merging strategy is adopted to effectively merge redundant tokens, the first three layers of the model process local information through adaptive window merging, and from the fourth layer, the model repeatedly executes global merging according to the depth of the layers so as to realize gradually enhanced feature representation; therefore, the number of tokens is reduced, the complexity of the ViT model in space-time self-attention calculation is reduced, and the problem of high calculation resource consumption caused by space-time information redundancy of the current ViT model in a video task is solved. Calculation overhead and energy consumption can be remarkably reduced in tasks such as video action recognition, action detection and time sequence event understanding, and meanwhile the characterization capacity of a key dynamic area is kept.

Description

Technical Field

[0001] This invention relates to a lightweight method for ViT models used in video action recognition, belonging to the field of lightweight model technology. Background Technology

[0002] In recent years, Vision Transformer (ViT)-based models have received widespread attention in the field of computer vision, especially in image and video processing tasks. ViT effectively captures global information through its self-attention mechanism, but its complex structure and high computational resource requirements limit its deployment in practical applications. Furthermore, when processing video data, the temporal repetition and spatial importance of the video data lead to spatiotemporal information redundancy in the ViT model, resulting in unnecessary computation. As the scale of tasks such as video action recognition, action detection, and behavior understanding continues to expand, the computational bottleneck of ViT in video becomes more prominent, making it difficult to directly deploy in real-world application scenarios or on edge devices.

[0003] To address this issue, researchers have begun exploring lightweight methods to reduce the computational complexity of models. Among these, token merging-based lightweight methods have gradually become a research hotspot. Token merging aims to reduce the number of tokens input into the ViT model, thereby reducing the computational burden. By processing the input tokens, pruning or merging unimportant redundant tokens, and retaining representative feature tokens, researchers have been able to significantly reduce computational resource consumption while maintaining model performance. This approach not only improves inference speed but also reduces storage space requirements, making ViT models more suitable for deployment on edge devices.

[0004] However, existing token merging methods still have limitations in handling spatiotemporal information redundancy. How to effectively select and merge tokens to capture important temporal dynamic information remains a problem to be solved. Furthermore, most existing pruning / merging methods require fine-tuning or retraining of the original model, making it difficult to apply them directly to different video Transformer models in a plug-and-play manner. They also cannot ensure that action semantics and temporal consistency are not compromised without training. Therefore, designing a general, lightweight solution that can be directly integrated into any ViT model, supports training-free deployment, and maintains stable performance across various video understanding tasks has become a key requirement for the practical application of current video models. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a lightweight method for ViT models for video action recognition, which effectively selects and merges tokens to capture important temporal dynamic information, reduces the complexity of the ViT model in spatiotemporal self-attention calculation, and thus solves the problem of high computational resource consumption caused by spatiotemporal information redundancy in video tasks.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: A lightweight method for ViT models in video action recognition includes the following steps: Step 1: Obtain the video dataset for the video action recognition task, and use the optical flow estimation method to estimate the dense optical flow field of two adjacent video frames in the video dataset; Step 2: Using a weighted average repair occlusion detection and repair method, the occlusion area in each frame of video is detected and repaired to obtain the repaired optical flow field. Step 3: Divide each frame into several image blocks of the same size, calculate the average displacement of each image block based on the repaired optical flow field, calculate the motion intensity score based on the average displacement, and generate a cross-frame image block soft mapping matrix based on the average displacement. Step 4: Divide each frame into several tokens of the same size. The size of the token is the same as the size of the image block, that is, the token and the image block correspond one-to-one. Based on the cross-frame image block soft mapping matrix, obtain the matching tokens of each token in the next frame. Calculate the similarity score between each token in the current frame and the token that matches each token in the next frame. Step 5: For each token, calculate the temporal redundancy score of the token in all frames based on the similarity score, and improve the temporal redundancy score using a weighted variance penalty term to obtain the improved temporal redundancy score. Step 6: Preset the window selection ratio and token merging rate of the ViT model. The first three layers of the ViT model only perform one merging operation, that is, only one of the first three layers has a non-zero token merging rate, while the other two layers have a zero token merging rate. When all frames pass through each layer of the ViT model, the tokens in each frame are merged based on the preset token merging rate. Step 7: When all frames pass through the first three layers of the ViT model where the token merging rate is not 0, perform binary matching merging on the tokens in the first frame. At the same time, use an adaptive greedy algorithm to divide other frames into several windows, and perform adaptive window merging on the remaining frames according to the window selection ratio and the token merging rate. Step 8: When all frames pass through all layers of the ViT model except the first three layers, perform binary matching and merging on the tokens in the first frame, and perform global merging on the tokens in the remaining frames; until the merging of the last layer of the ViT model is completed, the final merged token sequence is obtained, which is used for subsequent video action recognition tasks.

[0007] Compared with the prior art, the present invention, employing the above technical solution, has the following technical effects: 1. This invention not only estimates inter-frame motion using optical flow in the temporal dimension to reduce temporal redundancy, but also reduces spatial redundancy by dividing the window using motion intensity and saliency scores in the spatial dimension. This spatiotemporal combined optimization strategy can more effectively utilize inter-frame and intra-frame spatiotemporal information, thus improving efficiency.

[0008] 2. This invention performs multi-dimensional joint merging of tokens by calculating temporal redundancy scores and token importance scores. This strategy not only reduces redundant information but also preserves key features, ensuring that the model maintains high performance even after lightweighting.

[0009] 3. This invention organically combines windowing and global merging strategies, dynamically selecting either an adaptive window merging strategy or a global merging strategy based on the number of model layers. This dynamic selection mechanism can flexibly adjust the merging strategy according to different stages of the model, further improving the efficiency of lightweight design. The adaptive dynamic local window design better adapts to video motion information, mitigating the information loss problem caused by window edge cutting.

[0010] 4. This invention provides a solution that eliminates the need to retrain the ViT backbone network and inserts a lightweight preprocessing module only at the front end or intermediate layers. This module merges redundant tokens by reading patch embeddings or selecting token embeddings from intermediate layers, and then returns the merged token sequence to subsequent Transformer layers for further inference. For ViT-based video models, including VideoMAE, TimeSformer, and their variants, those skilled in the art can select appropriate feature extraction layers and insertion positions based on the specific network structure to achieve integration without adjusting backbone parameters. By reducing redundant tokens at the model front end, this invention significantly reduces computational overhead and energy consumption in tasks such as video action recognition, action detection, and temporal event understanding, while maintaining the ability to represent key dynamic regions. This enables efficient, low-power video Transformer inference, suitable for different deployment scenarios in the cloud, mobile, and edge devices. Attached Figure Description

[0011] Figure 1 This is a flowchart of a lightweight method for ViT model for video action recognition according to the present invention; Figure 2 This is a schematic diagram of the adaptive window merging process proposed in this invention; Figure 3 This is a schematic diagram of the global merging process proposed in this invention. Detailed Implementation

[0012] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0013] This invention proposes a lightweight method for ViT models used in video action recognition, comprising: assuming the existence of a video dataset containing... Frame-by-frame video, each frame has a resolution of [resolution value missing]. The Vision Transformer (ViT) model is used for processing. The ViT model divides each frame into multiple tokens, each token representing a local region of the frame. Assume each frame is divided into... There are 1 token, each token having a size of 1. , This is the side length of the token. For both the video frame and the token, the... Frame representation ,in Each frame Classified as The token, the first Each token is represented as ,in .

[0014] In the implementation of this invention, the first three layers of the Transformer model ( to The fourth layer performs a one-time adaptive window initialization and merging operation, which completes the dynamic window division and local merging of frames based on motion intensity and saliency score. Starting from [point number], the model enters the global merging phase, primarily performing global token merging operations across windows and frames. During the global merging phase, the model can repeatedly call upon the generated window partitioning results to perform supplementary merging within a local range, but it will not re-partition the windows or repeat the complete adaptive window merging process. The global merging process integrates information from the entire image, merges similar features from different regions, and updates the representation of each token. Due to the depth of these layers, the model can utilize global information to improve its understanding of the overall image. The number of merging layers depends on the depth of the layer and its corresponding role. During the processing of each layer, the layer index... This is used as a key parameter to determine the merging type. Through this design, the window partitioning operation is only executed once in the early stages of the model, while merging within a window can be repeatedly called at different levels, achieving stability of the low-level structure and continuity of semantic aggregation at the high level, thereby ensuring temporal consistency and matching of complexity analysis.

[0015] like Figure 1 As shown, the specific steps are as follows: Step 1, assuming the video frame sequence is ,in This represents the total number of video frames, with each frame having a resolution of [resolution value missing]. Initialize the merge rate list. (Including the merging rate or number of tokens per tier in ViT) and window selection ratio To obtain inter-frame motion information and improve the situation where optical flow estimation errors may still occur in complex situations such as non-rigid motion, drastic displacement, strong occlusion, or sudden changes in illumination, this invention provides a modular optical flow estimation framework that supports flexible switching between multiple optical flow methods and calculates optical flow between adjacent frames. and Dense optical flow field To adapt to different deployment environments and accuracy requirements.

[0016] This invention supports the following three types of optical flow estimation methods, which are determined by configuration parameters. Control options allow users to set the desired optical flow estimation mode during system initialization based on the dynamic characteristics of the video task, computational resource limitations, or accuracy requirements.

[0017] (1) Improved Farneback method (default scheme): Optical flow estimation is based on multi-scale pyramid and local polynomial expansion, which is suitable for resource-constrained and lightweight deployment scenarios.

[0018] First, each frame Convert to grayscale image : , Then, construct a multi-scale pyramid, and Decomposed into images of multiple scales: , in, This represents the number of layers in the pyramid. Starting from the lowest resolution layer (bottom layer), the optical flow field is estimated layer by layer upwards, and then gradually restored to the original resolution layer (top layer) through upsampling. A polynomial expansion is performed around each pixel to calculate local motion information and update the optical flow field. , Finally, fine optical flow estimation results were obtained at the highest resolution layer. .

[0019] (2) RAFT method: Based on the reconstructed related volume and iterative update mechanism, it has stronger non-rigid body modeling and global optimization capabilities.

[0020] First, extract features from the input frame: , in, This represents the CNN feature extractor, and then an All-Pairs Correlation Volume is constructed, which represents the feature similarity between every pair of pixels (or locations) in two frames of images: , Initialize the optical flow field to zero field and conduct Round-by-round iterative optimization: , in, Indicates the first In each iteration, the GRU (Generic Representation Root) module fuses contextual features with relevant information, ultimately yielding the converged optical flow estimation result. .

[0021] (3) PWC-Net method: Based on the end-to-end deep network structure of feature pyramid, optical flow pyramid and deformable convolution, it balances accuracy and inference speed and is suitable for complex dynamic scenes.

[0022] First, construct the feature pyramid: , To improve matching accuracy and feature consistency, an optical flow-guided feature alignment mechanism is introduced at each pyramid level. Let the current level be the... Layer, first place the previous layer The optical flow estimation results are upsampled to the current resolution, and then the optical flow field is used to evaluate the frame. The feature map is backsampled (warping) to align it to the spatial location of the current frame: , It helps improve the matching accuracy at the current layer, providing high-quality alignment features for subsequent cost volume construction and optical flow residual estimation.

[0023] Construct the cost volume and fuse contextual information to estimate the optical flow residual: , , in, This represents the residual prediction network. Then, the optical flow field is updated: , The final output is the finest level of optical flow. .

[0024] Step 2: Since occlusion or local noise may exist within video frames, potentially leading to incorrect optical flow matching, a weighted average repair occlusion detection and repair method is introduced to effectively detect and repair occluded areas in the image, improving the accuracy and robustness of optical flow estimation. First, occlusion detection is performed, and the local gradient of the optical flow is calculated. , and The optical flow field is respectively in and Calculate the gradient in the direction, and then calculate the gradient magnitude. If the gradient magnitude There exists a pixel whose gradient magnitude value is greater than a set threshold. If the pixel is an occluded pixel, then it is considered an occluded pixel. , in, It is an adjustment factor. and Let represent the mean and variance of the optical flow field, respectively. The detection sensitivity is adjusted based on changes in the optical flow of the image. Occlusion repair is performed on each optical flow-occluded pixel, and a local neighborhood is selected around the occluded pixel. Perform weighted average repair: , in, It is a weighting function that weights the optical flow fields based on their similarity within the neighborhood. The calculation method is as follows: , The more similar the neighboring pixels, the greater their weight. This ensures that the repaired optical flow field has high smoothness and consistency.

[0025] Step 3, divide each frame into For each image patch, the central and corner regions are sampled using grids of different sizes, and their average displacement is calculated. : , in, Indicates the first The sampling grid region corresponding to each image patch Indicates the location The pixel displacement value (which can be obtained from the optical flow field) is on the screen. This represents the total number of pixels within the region. It is also more robust to handling occlusion and non-rigid motion. Based on this average displacement, the corresponding position of each image patch in the next frame is determined, and the similarity between the image patch in the 3×3 region centered on the corresponding position in the next frame is calculated. After normalization, a cross-frame image patch soft mapping matrix is ​​generated. ( (where is the number of image patches in a frame), each element in the matrix is ​​denoted as . , indicating the current frame number The first image patch and the second image patch in the next frame within a 3×3 region centered on the optical flow projection position of that image patch. The matching confidence among candidate image patches, and the generated soft mapping matrix The matching confidence relationships of tokens between different frames were recorded.

[0026] For each image patch, calculate its motion intensity score. The exercise intensity score is obtained by calculating the average displacement of the tokens, using the following formula: , in, It is the first The average displacement of each image patch is calculated by taking the average of the displacements of the center region and the four surrounding corner regions of the sampled image patch. Subsequently, the motion intensity score needs to be normalized to the [0,1] interval. , in, and These are the exercise intensity score matrices. The maximum and minimum values ​​in the range. Finally, the motion intensity scores of all tokens are calculated. Composition of motion intensity fraction matrix .

[0027] Step 4, transfer the frame Divided into Token. Frame Divided into The size is The token, the division is done by the frame The image is segmented according to a fixed grid size, with each token corresponding to a specific image patch. Next, an embedding operation is performed on each token. Each token is mapped to a high-dimensional feature vector. , The feature dimension of the token is defined. This embedding process is achieved through a linear transformation, mapping the pixel value of each token to a high-dimensional feature space. Finally, all tokens are concatenated into a token sequence. This provides input for subsequent steps.

[0028] For frames Each token in The soft mapping matrix of the image patch obtained in step 2 Get its frame Mapping confidence of matching token regions .in The numbering can be defined in order from left to right or from top to bottom, or any fixed mapping method can be used. Those skilled in the art can adjust the numbering according to the specific implementation. The arrangement is then replaced with an equivalent one. Then, the weighted token pairs are calculated. cosine similarity : , in, This represents the dot product of two tokens. and These are their norms. To improve the semantic consistency and discriminative ability of cross-frame similarity estimation, this invention calculates the similarity term... In this case, the token embedding output from the intermediate layer is selected as the feature representation. This layer's features have stronger semantic abstraction capabilities than those of shallow layers, while still retaining some spatial positioning information, enabling a more accurate characterization of cross-frame semantic consistency.

[0029] In one implementation, cross-frame similarity calculation uses token embedding of an intermediate Transformer layer as a feature representation to balance semantic abstraction and feature stability. In another implementation, similarity calculation can be performed solely based on the patch bedding layer or other feature layers with equivalent representation capabilities. Those skilled in the art can make substitutions according to the specific ViT structure, all of which are equivalent variations of this invention.

[0030] Step 5: To avoid recursive dependency between the time redundancy score calculation and the current merging result, this invention uses token embeddings taken from the feature representations cached before the previous merging round in the merging decision, ensuring the independence of the redundancy measurement and the stability of the algorithm. For each token... Calculate its temporal redundancy score across all frames. The temporal redundancy score is calculated by progressively averaging the similarity scores of the token across all adjacent frames. The formula is: , in, It is the total number of frames in the video. The summation variable is the index of the frame containing the token. Iterate through all adjacent frame pairs. Indicates token In frame and Similarity score between them Take as the weight of the corresponding match Such a weighted progressive averaging can capture the non-homogeneity of motion changes. However, using only weighted averaging may not accurately reflect the semantic stability of tokens over time, especially in the presence of local mutations or short-term actions, easily misclassifying high-frequency semantic regions as redundant. To enhance the model's ability to perceive temporal fluctuations, this invention further introduces a weighted variance penalty term to construct a more robust measure of temporal redundancy: , The final improved time redundancy score is: , in, The volatility penalty weight controls the degree to which token temporal instability affects redundancy calculation. In online inference or streaming scenarios, the aforementioned temporal redundancy score can be approximated by a fixed-length sliding window. This improved score not only characterizes the average stability of the token's representation in the time series but also considers its inter-frame volatility, enabling the model to retain key regions that appear frequently but have large semantic fluctuations. This is particularly suitable for time-sensitive tasks such as fine-grained action recognition.

[0031] Time redundancy score of all tokens Composition of time redundancy fraction matrix This matrix is ​​used for subsequent token merging operations. Simultaneously, a token size matrix is ​​initialized. Its initial value is 1 (the initial size of each token is 1).

[0032] Step 6: Preset the window selection ratio and token merging rate of the ViT model. The first three layers of the ViT model only perform one merging operation, that is, only one of the first three layers has a non-zero token merging rate, while the other two layers have a zero token merging rate. When all frames pass through each layer of the ViT model, the tokens in each frame are merged based on the preset token merging rate.

[0033] Step 7, in this embodiment, only the latter... Motion intensity score and temporal redundancy score are pre-calculated for each frame. The first frame is not involved in this pre-calculation process, so it is processed separately using a binary matching and merging method. Let the token data of the first frame be... Divide the tokens into two sets alternately and calculate the similarity matrix between the tokens in the two sets. This is used to select matching token pairs. Based on the similarity matrix. Select the first one with the highest similarity The proportional tokens are merged. The merging operation uses a weighted average method, assuming that on the... Layer, for each pair of matching tokens Their sizes are respectively and Merged tokens The eigenvectors are calculated as follows: , No. Token size after merging in the layer for: , Update token size matrix : , During the token merging process, each token's size matrix value corresponds one-to-one with the token. The merging relationship is preserved during token merging. At the same time, the values ​​in the token size matrix are added together according to the same merging relationship.

[0034] The merged tokens will be updated with their corresponding time redundancy scores based on a weighted average, calculated as follows: , in, and These are the time redundancy scores of the tokens before the merger.

[0035] Step 7.1: To avoid misclassifying static but semantically crucial regions in the video as redundant regions, this invention introduces a semantic saliency map generated by a saliency detection model as an auxiliary indicator, along with a motion intensity score matrix. Construct a joint weighted window partitioning method. Significance matrix. Generated by the saliency detection module U²-Net. U²-Net takes the current video frame image as input and outputs a saliency probability map with the same resolution as the frame. It then maps this map to a token grid coordinate system using spatial mean pooling, ensuring that each token and its corresponding window have a corresponding saliency score. This is based on the motion intensity score matrix. and significance matrix Use an adaptive greedy algorithm to divide the frames Divided into multiple windows of different sizes The window partitioning strategy involves subdividing regions with high motion intensity and saliency into smaller windows, thereby enabling finer processing of semantic information. The initial window set is... For each window Simultaneously calculate its local motion intensity mean. Significance mean Define the window joint weighted scoring function: , By dynamically adjusting the adaptive threshold for each frame or local region This determines the window division. A threshold is set. ,in and These are the mean and standard deviation of the window joint weighted score, respectively. To adjust parameters, for each region, in addition to considering the average weighted score, the dispersion of the weighted score distribution is also taken into account to determine whether further subdivision is necessary. This is considered when the weighted score is greater than... The areas shown are more likely to be important and should be divided into windows. If... Greater than the set threshold Then the window Further divided into two sub-windows and Setting the minimum window size to 2×2, consisting of 4 tokens, avoids overly fragmented local information, which could compromise overall global semantic information. Through recursive partitioning, the final window set is obtained. This method can simultaneously focus on regions with high motion intensity and regions with high semantic saliency, achieving fine-grained preservation of static but key targets (such as faces, traffic signs, etc.), while suppressing irrelevant motion regions such as background disturbances (such as water surface ripples, leaf swaying) from being misclassified as important regions, thereby improving the information preservation capability and task accuracy in the lightweight compression process.

[0036] Step 7.2, for each window The similarity between all tokens within a window is calculated, and the average of these similarities is used to measure the overall similarity between tokens within the window. The similarity calculation formula is as follows: , in, It is a token and tokens The similarity between them is calculated using the same method as in step 4. The similarity value for each window is then calculated. Used for subsequent token merging operations.

[0037] Step 7.3, Select similarity value The highest front Token merging is performed using a proportional window. The merging rate for each window... From the list of merger rates Retrieved from the selected window. Within the range, perform a binary matching and merging operation. Assume that in the... Layer, for each pair of tokens within the window According to their size and Perform weighted merging: , No. Token size after merging in the layer for: , Update token size matrix : , The values ​​in the large and small matrices are in a one-to-one correspondence with the tokens. When tokens are merged, the merging relationship is preserved. The update of the token size matrix is ​​achieved by adding the sizes of the merged tokens using the same merging relationship as the token merging. The larger the value, the more merges the token performs, and the greater the weight of the weighted merge.

[0038] After each layer is merged, and before being passed to the next calculation, the data from the first frame is... and After integrating with other frame data, a complete frame data is obtained. Because the first frame is merged differently from the other frames, the final number of tokens obtained is also different. Therefore, the first frame... and Zero-padding is used to ensure consistency in token representation between the first and subsequent frames, avoiding impact on the unified expression of cross-frame information and subsequent spatiotemporal feature fusion. While zero-padding achieves formal uniformity in dimensionality, padded tokens lack a true semantic basis. If indiscriminately used in subsequent feature fusion, they can easily introduce representational biases, affecting the accuracy of cross-frame spatiotemporal modeling. Therefore, this invention further proposes a masked alignment strategy. While maintaining a consistent number of tokens across frames, it explicitly marks the padded positions using a token mask and masks these positions in the subsequent self-attention module, preventing them from participating in weight calculations and semantic propagation. This mechanism retains the advantages of a unified structure while effectively blocking the influence of tokens without semantic information, ensuring cross-frame semantic alignment and modeling stability.

[0039] To maintain consistent token counts across frames while avoiding interference from zero-padding tokens on attention and semantic propagation, this invention defines the position mask as follows: , Indicates a real token. This represents a zero-padding token. Column-to-row bidirectional masking and post-normalization are then performed on the self-attention and residual / feedforward paths, as follows: (1) Attention masking and normalization Let the standard attention logits be: , Denote constants For any Define the masked logits: , Among them, row masking comes from the fill query. The output, column masking, points to the fill key. Attention. Obtain masking softmax: , To maintain attention on the "real key only" normalization and to align with the Token Size redistribution of this invention, a one-time renormalization is performed after optional Size compensation: , in, For the corresponding token size, the actual token takes a positive value, and the padding bits are treated as... This is an equivalent implementation. The formula ensures that the weights of all pointers to the padding keys are always 0, and the weights of the real keys are renormalized to 1, thus satisfying both the unified structure and preventing semantic amplification and gradient interference of the padding bits.

[0040] (2) Query-side nullification and residual / feedforward gating For the output vector Apply gating: , That is, zeroing is performed row by row on the attention output, residual branch, and feedforward network output of the fill query rows. For element-wise multiplication, For length (A vector of all 1s). This gating only affects the signal at the filling position; the forward propagation path at the actual position remains unchanged.

[0041] (3) Forward consistency and pooling masking Use it in any step that requires average pooling or feature aggregation of the token set. As a weight masking item, for example: , The filling positions are not included in statistics or aggregation, thus maintaining consistency between the feature distribution and the masking strategy.

[0042] Step 8: When all frames pass through all layers of the ViT model except the first three layers, perform binary matching and merging on the tokens in the first frame, and perform global merging on the tokens in the remaining frames; until the merging of the last layer of the ViT model is completed, the final merged token sequence is obtained, which is used for subsequent video action recognition tasks.

[0043] Step 8.1: After token merging, the imbalance in token sizes leads to a shift in the softmax weight distribution in standard self-attention, thus weakening its expressive power in global modeling. Traditional proportional attention mechanisms directly add weights to softmaxlogits... While this approach achieves compensation, it is equivalent to injecting a fixed bias term into the unnormalized attention score, which implicitly alters the semantic relevance structure and introduces potential semantic bias, contradicting its design goal of "non-semantic amplification".

[0044] To address this, this invention proposes a normalized attention redistribution mechanism. This method redistributes attention proportionally based on token size after calculating the standard attention weights, rather than introducing bias at the logits stage, thus fundamentally avoiding semantic reinforcement effects. The specific calculation is as follows: , in, and For querying key vectors, For channel dimension, This is the token size vector. This is the numerical stability constant. It means that the standard softmax is calculated first, then the scaling is applied, and finally the result is divided by the sum of the scaled values, thus preserving the probability normalization property of attention.

[0045] The token size is initialized to 1. Each time a token merge operation is performed, the size of the merged tokens is added to the target token, and the update formula is: , in, Indicates all merged into tokens Other token sets, This refers to their original size. TokenSize represents the total number of patches covered by the token, not the number of times it is merged or its semantic weight. Using "number of merges" or "cumulative area" directly as the size might cause redundant region tokens to be continuously merged, thus gaining higher weight in the attention and causing the model to misfocus on informationless regions during training. To address this, our method incorporates two key constraints to avoid this phenomenon: Importance-driven one-way merging strategy: Merging operations always aggregate tokens of lower importance to tokens of higher importance, not the other way around. This means that the tokens being merged were originally determined to be redundant or have low semantic contribution, and their merging is only for "semantic filling" and will not participate in the decision-making process as an attention-driven factor.

[0046] Similarity-weighted merging mechanism: During merging, not only the spatial location between tokens is considered, but also a weighting factor is introduced based on semantic similarity. Tokens from non-important regions have a much smaller weighting effect than the number of patches represented by their size due to their significant semantic differences from important tokens, thus effectively suppressing interference from invalid information.

[0047] Therefore, although the Token Size value gradually increases with token merging, it is not equivalent to an amplified signal of "semantic importance" in this method. In the normalized attention redistribution mechanism, the role of Token Size is limited to structural proportional compensation of attention weights after softmax normalization, to offset the attention dilution effect caused by the reduction in the number of tokens, making the attention distribution closer to the original state before merging. Token Size is a purely structural normalization factor in this invention. Combined with a reasonable merging direction and semantic similarity weighting mechanism, this compensation method can maintain structural consistency while avoiding uninformative weights for non-informative regions in attention calculation, thereby ensuring the semantic stability and expressive accuracy of the model after compression.

[0048] Step 8.2 calculates the token importance score to measure the relative importance of each token in the model. In traditional token importance calculation methods based on the current layer's activation value, the importance score is negatively affected by the merging operation: merging changes the token's composition and activation pattern, thus affecting the importance assessment of subsequent layers, forming a feedback loop. This leads to unstable or even amplified importance scores, reducing the robustness of the merging strategy. To break this feedback loop, this invention introduces an importance score predictor in the importance scoring process. This predictor does not directly rely on the immediate activation value of the current layer, but rather predicts the importance score based on the token's cross-time historical information and current features within the same layer, using a lightweight multilayer perceptron (MLP). Historical information comes from the most recent... The forward statistics may come from cached statistics before the previous round of merging, thus avoiding inter-layer information access in a single forward pass.

[0049] For each token Record its nearest neighbor on the same layer Activation mean in the next forward computation With activation volatility (variance) Alternatively, the activation mean and variance of this layer obtained from the previous round of merging can be used to form a historical feature vector: , and Exponential moving average statistics can be used for iterative updates within the same layer. Let the decay coefficient be... , No. When making a merger decision ,in, Activate the scalar statistics for the tokens of this layer. This represents the average value along the corresponding token matching trajectory.

[0050] Get the embedding features of the token in the current layer and with historical characteristics splicing: , Input the fused features into a lightweight MLP predictor: , in, The importance score for prediction is normalized to [0,1]. By utilizing historical trajectory information to avoid direct reliance on the instantaneous activation value of the current layer, the merging feedback loop is fundamentally broken, improving the overall robustness and convergence of the merging strategy.

[0051] Step 8.3, Time Redundancy Score And importance score These belong to different semantic dimensions. Directly weighting and merging them into a single score for ranking would lead to semantic dimension confusion. Therefore, this invention proposes a multi-dimensional joint ranking (Pareto merging strategy), which introduces a multi-dimensional non-dominated ranking mechanism in the merging decision to avoid weight conflicts between different semantic indicators and improve the interpretability and robustness of the merging.

[0052] Construct a two-dimensional score vector for the token: , Define a dominance relation if and only if A token is called a token if at least one of the following conditions is strictly met. Domination Token , recorded as Based on this, we obtain the Pareto set of preferred frontiers: , This refers to a set that cannot be simultaneously surpassed by any other token in either of the two dimensions. During the merging process, this invention employs a two-stage screening strategy: 1. Redundancy Pre-screening: Select tokens with redundancy exceeding a threshold from all tokens. Candidate set: , 2. Importance ranking: In According to importance Sort in ascending order to obtain the final sorted set.

[0053] To obtain the final complete sorting result, this invention will not belong to... Other tokens according to their The value is appended to the end of the sorted list to obtain the final set covering all tokens. And its size is consistent with the number of input tokens.

[0054] Through the aforementioned multi-dimensional joint sorting mechanism, time redundancy and importance are considered independently during the merging process, effectively avoiding erroneous merging caused by semantic confusion in traditional weighted fusion methods, and ensuring a balance between compression efficiency and information retention.

[0055] Step 8.4, assuming the current layer is Based on the sorted set of tokens Merging rate with the current layer The tokens are categorized into two classes: important and unimportant. For a set of unimportant tokens and a set of important tokens, their similarity is calculated. ,Right now: , in, and These are unimportant tokens. and important tokens The high-dimensional feature vector, Indicates unimportant tokens and important tokens Similarity; for each unimportant token Find the most important token with the highest similarity and designate it as the unimportant token. The corresponding token, the important token with the highest similarity is matched with Merge; When at least one unimportant token is merged into the same important token In the middle, the unimportant tokens and important tokens that need to be merged will be... Perform an exponential transformation on the similarity: , Calculate the normalized denominator : , For the unimportant tokens and important tokens that need to be merged, calculate their weights separately: , Weighted merge of unimportant tokens into their corresponding important tokens. The merging formula is: , Update the token size matrix and time redundancy score matrix , the first frame of data and This data is then integrated with other frame data to obtain the complete data. In subsequent layers of ViT, the repeated global merge is performed until the final merge token sequence is obtained. This sequence is used for downstream video tasks. Through the motion perception and token merging strategy of this invention, while maintaining consistency in action semantics and temporal sequence, the final number of tokens is significantly reduced, resulting in an overall computational complexity reduction of nearly 50% compared to the original Transformer forward inference. In tasks such as video action recognition, it only leads to an accuracy decrease of approximately 0.7%-1%, thus achieving efficient and low-cost video Transformer inference. End of process.

[0056] The aforementioned definition of temporal redundancy score is based on global statistics of the entire video frame sequence and is applicable to offline video action recognition scenarios. In actual online or streaming inference scenarios, those skilled in the art can use a fixed-length time sliding window to estimate the temporal redundancy score only within a local window, which is an equivalent substitution to the present invention.

[0057] Optical flow estimation is a crucial step in temporal information modeling in this method. It primarily captures inter-frame motion information, thereby assisting in temporal redundancy evaluation and dynamic merging. To balance modeling accuracy, computational efficiency, and scene adaptability, the modular optical flow estimation framework of this invention supports three methods: improved Farneback, RAFT, and PWC-Net. Compared to the self-attention mechanism within the temporal window used in existing methods, its complexity is approximately [missing information - likely a percentage]. Our method exhibits superior scalability in terms of computational complexity. We incorporate occlusion repair and sparse mapping mechanisms into the traditional Farneback optical flow algorithm, significantly reducing the computational burden while preserving motion estimation accuracy. This improved optical flow estimation algorithm has a computational complexity of approximately [missing information - likely OCR errors]. ,in Let be the side length of the local optical flow window. It can be observed that at lower input resolutions (i.e., ... In the case of optical flow computation, the computational costs of the two are comparable, but in high-resolution scenes, due to the complexity of optical flow computation and the embedding dimension, the computational cost of optical flow computation increases. Irrelevant and limited by the local window size, its overall complexity is lower than that of fully connected temporal attention. The RAFT method relies on constructing a full-pair correlation volume across frames and iteratively optimizing the optical flow field on the correlation volume through gated recurrent units (GRUs) to achieve fine pixel-level displacement estimation. Its overall complexity can be approximated as... , Let be the number of channels in the GRU convolution. It is evident that in high-resolution scenes, since the relevant volume size is proportional to the square of the number of pixels within the frame, its computational resource consumption is far higher than that of local optical flow methods, making it particularly unsuitable for real-time inference deployment under high-resolution video or long temporal inputs. The PWC-Net method performs hierarchical optical flow estimation through a pyramid structure. It performs optical flow prediction at a lower resolution in each layer and refines it layer by layer through upsampling. The overall computational complexity can be approximated as... , The search radius is given. Therefore, the complexity of PWC-Net is on the same order of magnitude as the improved Farneback method, and both are limited by the local window size and feature dimension, not by the number of tokens in the entire frame or the attention structure. This characteristic makes it more suitable for lightweight deployments and resource-constrained environments, especially valuable for video modeling tasks with high real-time and computational efficiency requirements.

[0058] It is worth noting that although this invention employs adaptive window merging at the lower levels and global merging at the higher levels, unlike existing fixed-division strategies, its design is based on clear rationale and does not compromise semantic aggregation capabilities. Lower-level window merging dynamically partitions windows based on a combination of motion intensity and semantic saliency, ensuring high semantic consistency and spatial relevance of tokens within the same window. Furthermore, it performs only one local merge to eliminate highly redundant and low-saliency regions, avoiding the cutting of semantic entities across windows. Higher-level global merging, on the other hand, uses multi-dimensional joint ranking to filter based on temporal redundancy and importance indicators, and incorporates an importance score predictor to stabilize merging decisions, thus prioritizing the retention of key regions with discriminative value. Simultaneously, a normalized attention redistribution mechanism maintains consistent token weight distribution across different merging stages, preventing attention shifts caused by early compression. This design makes lower-level and higher-level merging complementary, significantly reducing early computational load while maintaining or even enhancing the semantic aggregation capabilities of higher levels, effectively avoiding the structural defects of traditional methods.

[0059] The following is a specific embodiment for illustration.

[0060] Taking the Kinetics-400 (K400) video dataset for video action recognition tasks as an example, this dataset contains approximately 400 action categories and a large number of short video clips. Assume any video in this dataset contains 8 frames, each with a resolution of 224×224. A 12-layer Vision Transformer (ViT) model is used for processing. In the first three layers of the Transformer model (i.e., ...) to Layer, layer number is In this context, an adaptive window merging process is performed only once. However, in [the context of the previous sentence], [the following is unclear and likely refers to a different process]. to Within a layer, a global merge process is executed, which can be repeated multiple times. This is specifically determined by the merge rate list. The data in the data determines which layer to perform adaptive window merging; 0 indicates no merging.

[0061] Step 1: Assume the video frame sequence is Each frame has a resolution of 224×224. This invention defaults to using the Farneback method for optical flow estimation, converting each frame into a grayscale image to obtain a grayscale image sequence. Build Layered pyramids yield image sequences at different scales: , Starting from the coarsest scale, the optical flow field is estimated layer by layer to obtain the optical flow field for each frame. .

[0062] Step 2: Perform occlusion detection and repair for each frame to improve the accuracy and robustness of optical flow estimation. First, occlusion detection is performed, and the gradients of the horizontal and vertical components of the optical flow field are calculated separately to obtain the local gradient of the optical flow. Then calculate the gradient magnitude. ,if Greater than the adaptive threshold The region is considered an occluded region, and the occluded region set is obtained. For each occluded area To perform occlusion repair, select a local neighborhood around the occluded area. A weighted average repair is performed, and the repaired optical flow values ​​are updated in the original optical flow field to obtain the repaired optical flow field. .

[0063] Step 3: Divide each frame into 196 image blocks, each block being 16×16 in size, where 16 is the side length of the image block. Let the... Frame is , For each image patch, assume the center point of the image patch is... The range of the 6×6 grid is arrive Assume the four corner points of the image patch are and The range of the 5×5 grid is arrive , arrive , arrive , arrive After sampling these grid regions, the average displacement of all sampled points was calculated, and its position on the 1st... The corresponding position in the frame. Then, the similarity between each image patch and all image patches within the 3×3 region is calculated, and a soft mapping matrix is ​​generated. Record the matching confidence relationship of tokens between different frames.

[0064] The motion intensity score for each image block is obtained by taking the modulus of the average displacement. After calculation, 196 image blocks are obtained per frame. The scores are then normalized to form a 14×14 motion intensity score matrix, which is used to obtain the matrix for the last 7 frames. .

[0065] Step 4: The ViT model will process each frame Divide the tokens into a fixed size of 16×16, resulting in 14×14 tokens. Let the [number]th [token] be [the token]. Frame is , ;No. Each token is , Each token is 16×16 in size. An embedding operation is performed on each token, mapping its pixel values ​​to a high-dimensional feature space through a linear transformation. Assuming a feature dimension of 768, each token is mapped to a 768-dimensional feature vector. Concatenate the feature vectors of all 196 tokens into a sequence. Ultimately, 8 token sequences were obtained. .

[0066] With frames and For example, through the token soft mapping matrix Find the matching token in the second frame for each token in the first frame, and the 3x3 region centered on that matching token, for example, token pairs. This indicates that the first token in frame 1 matches the region centered on the third token in frame 2, and the cosine similarity is calculated. A similarity matrix of 196×196 is obtained by calculating the similarity of all token pairs. Perform similar operations sequentially across the remaining frames. Step 5: Calculate the temporal redundancy score for each token. In frames For example, for each of the tokens According to the token soft mapping matrix Find the matching tokens and token regions in the previous 7 frames, and then obtain the similarity score of the tokens from the similarity matrix of each frame, for example... Calculate the redundancy score And calculate the weighted variance penalty term. The final time redundancy score is obtained as follows: Repeat the above steps to calculate the time redundancy score for all tokens. Finally, the time redundancy fraction matrix is ​​obtained. Its size is 196×1. Similar operations are performed sequentially on other frames. Simultaneously, the token size matrix is ​​initialized. The initial value is 1.

[0067] Step 6: Preset the window selection ratio of the ViT model and the token merging rate at each level When based on the preset token merging rate When the calculated target number of merged tokens exceeds the actual number of tokens that can be merged in the current frame under the minimum window constraint, the actual number of merged tokens can be truncated to the maximum number of mergeable tokens, or the effective merging rate of this layer can be reduced accordingly. Those skilled in the art can set an upper limit for the merging rate based on specific application scenarios to avoid feature loss due to excessive merging. The first three layers of the ViT model perform only one merging operation; that is, only one of the first three layers has a non-zero token merging rate, while the other two layers have a zero rate. When all frames pass through each layer of the ViT model, the tokens in each frame are merged based on the preset token merging rate.

[0068] Step 7: Since only the motion intensity score and temporal redundancy score of the last 7 frames can be calculated, the first frame must be processed separately for each token merging step, performing a binary matching merge. The token data for the first frame is... Merger rate This indicates the proportion of tokens that need to be merged in the first frame. Tokens are alternately divided into two sets, and the similarity between the tokens in the two sets is calculated. The tokens with the highest similarity are then found. The token pairs are weighted and merged according to the token size matrix. The token data of the first frame after merging is as follows: The number of tokens was reduced by 39, and the updated token size matrix has a shape of 157×1, with values ​​updated according to the merged token size.

[0069] In the first three layers of the model where merging operations are required, an adaptive greedy algorithm is used to divide all frames except the first frame into several windows. Adaptive window merging is then performed on the remaining frames based on the window selection ratio and the token merging rate. For example, the motion intensity score of each token is stored in a matrix. In the calculation of the significance matrix Initialize window collection The initial window is the entire frame, and the average motion intensity of the window is calculated. Significance score of the game Calculate the joint weighted score of the window. Then, the mean and variance of the score are calculated to obtain the adaptive threshold. ,like Then the window The window is divided into two sub-windows. and The new window set is { The process continues by dividing the window into smaller sub-windows using a binary recursive method until the average motion intensity of all windows is less than or equal to the adaptive threshold. Perform similar operations sequentially on other frames.

[0070] For the results obtained through the above operations Window collection Select a specific child window For example, It contains 4 tokens. Calculate the token similarity within the window to obtain a similarity matrix. Calculation window average similarity The average similarity of each window is calculated sequentially, and a similar operation is performed on other frames.

[0071] Based on the average similarity of each window, and according to the window selection ratio Select the first one with the highest similarity Token compression is performed using a proportional window. Assuming the current frame has 10 windows, the calculated average similarities are as follows: Select the first one with the highest similarity The proportional window, i.e., the first 5 windows, perform token merging. From the list. Get the merging rate of the current adaptive window merging layer. This indicates that 20% of the tokens need to be merged. Within the selected window, a binary matching merge is performed. For the window... Each token pair in ( and ), according to their size and Perform a weighted merge. Assume the number of tokens per frame is reduced by 30 after merging, and the token data is... Updated token size matrix The shape is 166, and the value is updated based on the size of the merged tokens, also using a weighted merge to update the time redundancy score. Similar operations are performed sequentially on other frames. Before being passed to subsequent layers, the data of the first frame is aligned using a masking mechanism. and Align and integrate with other frame data to obtain complete frame data. And a token size matrix of size 166 .

[0072] Step 8: When all frames pass through all layers of the ViT model except the first three layers, perform binary matching and merging on the tokens in the first frame, and perform global merging on the tokens in the remaining frames; until the merging of the last layer of the ViT model is completed, the final merged token sequence is obtained, which is used for subsequent video action recognition tasks.

[0073] To address the imbalance in attention distribution after token merging and further improve the model's ability to represent attention to large tokens, the attention mechanism in the model is replaced with a normalized attention redistribution mechanism. After adaptive window merging, the token data for each frame is... Each frame contains 166 tokens, and their importance scores are predicted using a lightweight multilayer perceptron (MLP) to obtain an importance score matrix for each token. .

[0074] The temporal redundancy score for each token in each frame has been calculated. And importance score These scores are then used for multidimensional joint sorting. First, a two-dimensional score vector of the tokens is constructed. The Pareto set of precedence is obtained from the defined dominance relation. A two-stage screening strategy is employed for merging. Assuming a redundancy threshold of 0.6, a pre-screening process is first performed, selecting candidate tokens with a redundancy level higher than 0.6 from all tokens. Then according to Prioritize them by importance, and those that do not belong to Other tokens according to their The value is appended to the end of the sorted list to obtain the final set covering all tokens. .

[0075] From the list of merger rates Get the merging rate of the current layer 15. Therefore, based on the merging rate, the top 85% of tokens are selected as important tokens, and the remaining 15% are considered unimportant tokens. This results in 141 important tokens and 25 unimportant tokens. The similarity between important and unimportant tokens is calculated, and the important token with the highest similarity to the unimportant tokens is selected for global merging.

[0076] After merging, the number of tokens per frame decreased by 25, and the token data is as follows: Updated token size matrix The shape is 141, and the value is updated based on the size of the merged tokens, also using a weighted merge update time redundancy score. .

[0077] In subsequent layers of ViT, step 8 is repeated until the final merged token sequence is obtained. This sequence is used for downstream video tasks. In this example, token merging reduces the number of tokens by 48%, significantly lowering the overall computational complexity compared to the original Transformer forward inference, thus achieving efficient and low-cost video Transformer inference. End of process.

[0078] This invention effectively utilizes temporal and spatial redundancy, optimizing in both the temporal and spatial dimensions to significantly reduce redundant information, thereby reducing computational load and memory consumption. By combining windowing and global merging strategies, it provides a customized merging strategy for video processing tasks, significantly improving computational efficiency and model performance. Compared to other lightweight methods, this invention not only drastically reduces floating-point computation but also maintains high task accuracy.

[0079] Based on the same inventive concept, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the aforementioned ViT model lightweighting method for video action recognition.

[0080] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0081] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0082] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0083] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0084] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A method for ViT model lightweight for video action recognition, characterized in that, The method comprises the following steps: Step 1, obtaining a video data set for a video action recognition task, and estimating a dense optical flow field of adjacent two frames of the video data set by using an optical flow estimation method; Step 2, detecting and repairing an occlusion area in each frame of video by using a weighted average repair occlusion detection and repair method to obtain a repaired optical flow field; Step 3, dividing each frame into a plurality of image blocks of the same size, calculating an average displacement of each image block according to the repaired optical flow field, calculating a motion intensity score based on the average displacement, and generating a cross-frame image block soft mapping matrix based on the average displacement; Step 4, dividing each frame into a plurality of tokens of the same size, the token size being the same as the image block size, that is, the token and the image block correspond one by one, obtaining a matching token of each token in the next frame based on the cross-frame image block soft mapping matrix, and calculating a similarity score between each token in the current frame and the token matched with each token in the next frame by weighting; Step 5, for each token, calculating a time redundancy score of the token in all frames based on the similarity score, and improving the time redundancy score by using a weighted variance penalty term to obtain an improved time redundancy score; Step 6, presetting a window selection ratio of a ViT model and a token merging rate of each layer, and performing a merging operation only once in the first three layers of the ViT model, that is, only the token merging rate of one of the first three layers is not 0, and the token merging rates of the other two layers are 0; When all frames pass through each layer of the ViT model, the tokens in each frame are merged based on the preset token merging rate; Step 7, when all frames pass through the layer of the ViT model in which the token merging rate is not 0, performing a binary matching merging on the tokens in the first frame, and dividing other frames into a plurality of windows by using an adaptive greedy algorithm, and performing adaptive window merging on the remaining frames according to the window selection ratio and the token merging rate; Step 8, when all frames pass through the layers of the ViT model other than the first three layers, performing a binary matching merging on the tokens in the first frame, and performing a global merging on the tokens in the remaining frames; until the merging of the last layer of the ViT model is completed, a final merged token sequence is obtained, which is used for subsequent video action recognition tasks.

2. The ViT model lightweight method for video action recognition of claim 1, characterized in that, In step 1, the optical flow estimation method is an improved Farneback optical flow method, a RAFT optical flow estimation method or a PWC-Net optical flow estimation method.

3. The ViT model lightweight method for video action recognition of claim 1, wherein, The specific process of step 2 is as follows: Computing local gradients of dense optical flow fields : ， wherein, is a dense optical flow field of adjacent frames and is a dense optical flow field, and are gradients of the dense optical flow field in and directions, respectively. Computing gradient magnitude If the gradient magnitude of a certain pixel in the gradient magnitude image is greater than a preset threshold value , the pixel is determined as an occlusion pixel, otherwise not. The formula is as follows: ， ， wherein, is an adjustment factor, and denote the mean and variance of the dense optical flow field respectively. Selecting a local neighborhood around the occluded pixel Weighted average inpainting is performed for the occluded pixel, with the following formula: ， wherein, is the occluded pixel coordinate, is the repaired optical flow, is the local neighborhood of pixel coordinates, is the weight function, , is the optical flow of each pixel in the local neighborhood , is the neighborhood center pixel optical flow, i.e. the occluded pixel optical flow.

4. The ViT model lightweight method for video action recognition of claim 1, characterized in that, The specific process of step 3 is as follows: dividing each frame into image blocks, for each image block, sampling a center region and a corner region of the image block, calculating an average displacement of the image block: ， wherein, is the average displacement of the th image block, represents all the sampling regions corresponding to the th image block, is the total number of pixels within the sampling region, , , is the resolution of each frame, is the side length of the image block; represents the pixel displacement value on the pixel coordinate within the sampling region, obtained from the repaired optical flow field; Based on the average displacement, the corresponding position of each image patch in the next frame is determined, and the similarity between the patch and the corresponding position in the next frame (within a 3×3 region) is calculated. After normalization, a cross-frame image patch soft mapping matrix is ​​generated. Each element in the matrix is ​​denoted as , indicating the current frame number The first image patch and the second image patch in the next frame within a 3×3 region centered on the optical flow projection position of that image patch. Matching confidence among candidate image patches; For each image patch, a motion intensity score is calculated based on the average displacement and normalized to the [0, 1] interval: ， ， wherein, and are the maximum and minimum values of the matrix of all motion intensity scores respectively, is the normalized motion intensity score, the normalized motion intensity score of all image blocks composing the motion intensity score matrix .

5. The ViT model lightweight method for video action recognition of claim 4, characterized in that, The specific process of step 4 is as follows: Each frame is divided into tokens, each token is mapped to a high-dimensional feature vector , represents the set of real numbers is the feature dimension for tokens; Based on Cross-frame image block soft mapping matrix Get the current frame China Token Matching token in the next frame The confidence level is used to calculate the current frame using weighted averages. China Token In the current frame and the next frame Similarity score between : ， wherein, denotes the dot product, denotes the current frame in which the token is mapped to a high-dimensional feature vector, denotes the next frame in which the token is matched to a token mapped to a high-dimensional feature vector, denotes the token is matched to a token in a 3x3 size region centered around the token in the next frame.

6. The ViT model lightweight method for video action recognition of claim 5, characterized in that, The specific process of step 5 is as follows: For each token a temporal redundancy score is computed by progressively averaging the similarity scores of the token in all adjacent frames : ， wherein, is the total number of frames of the video dataset, is the index of the frame where the token is located, the summation variable traverses all pairs of adjacent frames, ; Introducing a weighted variance penalty term A time redundancy score Improvements: ， ， wherein, is the improved temporal redundancy score, is the volatility penalty weight.

7. The ViT model lightweight method for video action recognition of claim 6, characterized in that, The specific process of the binary matching merging is as follows: Let the token data of the first frame be The token data of the first frame is alternately divided into two sets, the first set and the second set. The similarity of any token pair composed of a token in the first set and a token in the second set is calculated. The similarity is sorted in descending order, and the top The token pair corresponding to the proportional similarity is merged, is the token merging rate. The weighted average method is used for merging, that is, in the ViT model layer, for the token pairs , that need to be merged, the first frame is represented by the token that needs to be merged when the ViT model passes through the first layer, and , the sizes of and are and respectively, and the initial size of the token is 1. The merged token in the first layer is calculated as ， No. Token size after merging in the layer for: ， The merged token updates the corresponding time redundancy score based on the size of the token, and the calculation formula is: ， wherein, the merged tokens corresponding high-dimensional feature vectors, and the tokens and corresponding high-dimensional feature vectors, the time redundancy scores of the merged tokens in the layer, and the time redundancy scores of the tokens and before merging.

8. The ViT model lightweight method for video action recognition of claim 7, characterized in that, In step 7, other frames are divided into a plurality of windows by using an adaptive greedy algorithm, and adaptive window merging is performed on the remaining frames according to the window selection ratio and the token merging rate, and the specific process is as follows: Step 7.1, for the current frame When performing adaptive window merging, the current frame is input into the saliency detection model to generate a saliency matrix According to the motion intensity score matrix and the saliency matrix , the current frame is divided into multiple windows of different sizes, with the smallest window set to 2x2 size, i.e. composed of 4 tokens; considering the current frame as a window , compute the local motion intensity mean and the saliency mean , define the window joint weighted score function : ， wherein, and are weighting score function adjustment factors; the adaptive threshold is preset : wherein, and are the mean and standard deviation of the window joint weighted score respectively, is an adjustment parameter; if is greater than a preset threshold , the window is further divided into two sub-windows and , the joint weighted score function of the sub-windows is continuously calculated, if the joint weighted score function of the sub-windows is greater than a preset threshold , the sub-windows are further divided, through recursive division, until the joint weighted score function of all windows is not greater than the preset threshold , and the size of the minimum window is 2x2. Step 7.2, for each window , the similarity between all tokens within the window is calculated and averaged to get the similarity value for each window, calculated as: ， in, For window The similarity value, For the token in the window and tokens Similarity between them The calculation method is the same as the similarity score calculation method in step 4. For window The number of tokens included; Step 7.3, selecting a preset window selection ratio Selecting a window for token merging, i.e., selecting a similarity value The top The window with the highest token merging rate, i.e., the token merging rate of the current frame The token merging rate of the ViT model layer where the current frame is located, and the token merging method in the window adopts a weighted average method in the two-point matching merging.

9. The ViT model lightweight method for video action recognition of claim 8, characterized in that, In step 8, the specific process of the global merging is as follows: Step 8.1, for the current frame When performing global merging, the normalized attention reallocation mechanism is used to compute the attention score for each token in the current frame :​ ， wherein, and is a query and key vector, is a channel dimension, is a size vector of tokens, is a numerical stability constant; Step 8.2, calculating an importance score of each token by using a lightweight multi-layer perception; Step 8.3: Based on the attention score and importance score, sort the current frame according to the Pareto ranking algorithm. Select a subset of tokens from all tokens and sort them in ascending order of importance score to form the first set; sort the remaining tokens in ascending order of importance score to form the second set; combine the first set and the second set to form the final set. Step 8.

4. According to the current frame Token merging rate of the ViT model layer where the token is located Divide the tokens of the current frame into important and unimportant tokens, i.e. divide the tokens with a final set proportion into unimportant tokens and the rest into important tokens, and calculate the similarity between any unimportant token and any important token: ​ ， wherein, and are high-dimensional feature vectors of unimportant tokens and important tokens respectively, denotes the similarity of unimportant tokens and important tokens ; for each unimportant token , the important token with the highest similarity is found as the corresponding token of the unimportant token , and the important token with the highest similarity is merged with ; When there are at least one unimportant token merged into the same important token when the similarity between the unimportant token to be merged and the important token is exponent transformed: ， Computing the normalized denominator : ， For the unimportant tokens and important tokens to be merged, their weights are calculated respectively: ， combining unimportant tokens with corresponding important tokens with the combining formula: ， wherein, is the unimportant token to be merged with the important token , is a constant, is the weight of the unimportant token to be merged , is the weight of the important token , is the high-dimensional feature vector of the unimportant token to be merged , is the high-dimensional feature vector of the new token obtained after global merging.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, The processor implements the steps of the ViT model lightweight method for video action recognition according to any one of claims 1-9 when executing the computer program.

Images