Video action model pruning method based on semantic clustering local motion consistency

Abstract

This invention relates to the field of computer vision technology and discloses a video action model pruning method based on semantic clustering local motion consistency. It involves interval sampling of the original video to obtain a set of video frames, dividing each frame into patches, and outputting token sequence features after linear projection and positional encoding. A semantic clustering pruning module outputs retained token sequence features and redundant token sequence features. A frame neighborhood motion consistency module performs motion prediction on the redundant token sequence features, outputting dynamically reactivated token sequence features. Finally, the dynamically reactivated token sequence features are input into the trained video action model to output the video action category. This invention has the ability to characterize the spatial semantic structure of key actions and achieves redundant token pruning, reducing computational complexity while ensuring action recognition accuracy and real-time inference capabilities in complex scenes.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a video action model pruning method based on semantic clustering local motion consistency. Background Technology

[0002] Video action recognition, a core task in computer vision, aims to enable machines to identify and classify human actions or object movements occurring in videos. Unlike static image classification, video action recognition not only extracts spatial features from single frames but also captures the dynamic evolution patterns between frames to distinguish actions that are similar in a single frame but different along the time dimension. For example, sitting down and standing up may be similar in a middle frame but with opposite directions of movement. With the development of artificial intelligence and computer vision, video action recognition has been widely applied in many fields: in security monitoring, it can monitor abnormal behaviors such as fights and falls in real time, triggering alarms to compensate for the oversights of human monitoring; in intelligent transportation, it can identify dangerous behaviors such as pedestrians running red lights, drivers driving while fatigued, or using mobile phones while driving.

[0003] From a technological evolution perspective, video action recognition has undergone three transformations: from handcrafted features to deep learning and then to the Transformer architecture. Early methods relied on handcrafted feature descriptors, such as histograms of oriented optical flow (HOF) and motion boundary histograms (MBH), combined with support vector machines (SVM) for classification. While these methods were computationally efficient, they had weak generalization ability and struggled to handle complex backgrounds. With the rise of deep learning, convolutional neural networks (CNNs) became mainstream. Representative works include: two-stream networks that process RGB frames using spatial flow and optical flow using temporal flow, then fuse the results; 3D CNNs that directly perform convolutions on spatiotemporal cubes to explicitly model the temporal dimension, such as C3D and I3D; and temporal segmentation modules that divide long videos into segments, extract features from each segment, and then aggregate them, such as TSN and TSM. Although these methods improved accuracy, they were limited by the local receptive field of convolution operations. Inspired by Natural Language Processing (NLP), VisionTransformer (ViT) and its variants, such as Video Swin Transformer and TimeSformer, have become new development trends. Video action recognition models based on the Transformer architecture have achieved breakthrough accuracy on multiple public benchmark datasets thanks to their global context modeling capabilities and long-range spatiotemporal dependency capture. However, the high resolution and long temporal sequence characteristics of video data result in a large number of redundant features that contribute little to the action category, such as long-term static background areas and uniform textured surfaces lacking discriminative information. These redundant features have limited semantic representation of actions but still participate in the complete attention calculation. Since the complexity of the attention mechanism increases quadratically with the length of the token sequence, it restricts the inference speed of the model, making it difficult to meet the millisecond-level latency requirements of real-time monitoring and online interaction scenarios in the industry, and also raising the deployment threshold for edge devices.

[0004] Token pruning is a sparsity acceleration technique that aims to prune tokens with lower contributions to the final task during the model inference stage. Key tokens are retained for subsequent attention calculations and feature modeling, reducing computational complexity and memory usage while maintaining model accuracy, thus achieving a lightweight model and accelerated inference. Depending on the operation, token pruning can be divided into hard pruning and soft merging. Hard pruning directly deletes tokens with lower importance scores, while soft merging, unlike simple discarding, combines multiple similar tokens.

[0005] Currently, no token pruning method has been found specifically for video action recognition. Existing image methods mainly evaluate the importance of spatial tokens, but the dynamic spatiotemporal structure of video data makes it difficult to directly transfer image methods. Token pruning for video models faces two challenges: insufficient characterization of spatial semantic structure and lack of temporal motion consistency.

[0006] (1) The problem of insufficient spatial semantic structure refers to the fact that existing image token pruning methods focus on the semantic similarity or attention score between tokens, and mostly use attention weight or feature amplitude, without considering the influence of token position relationship on action semantics. (2) The problem of lack of temporal motion consistency refers to the failure to consider the real spatial position relationship and motion evolution law of inter-frame tokens, resulting in the inconsistency of pruning strategy along the time dimension. Summary of the Invention

[0007] The purpose of this invention is to address the shortcomings of existing methods by providing a video action model pruning method based on semantic clustering and local motion consistency. This method can achieve semantic perception sparsity of intra-frame tokens through Gaussian mixture modeling, and can determine whether to restore pruned tokens through frame neighborhood local measurement and motion trajectory predictor, i.e., dynamic reactivation, to ensure the consistency of token pruning. This reduces computational complexity while improving the model's action recognition accuracy and real-time inference capability in complex scenes.

[0008] The video action model pruning method based on semantic clustering local motion consistency provided by this invention includes the following steps: Step 1: Perform interval sampling on the original video to obtain a set of video frames. Divide each frame into a Patch, and generate Token sequence features after linear projection and position encoding. Step 2: Construct a semantic clustering pruning module, with the input being token sequence features and the output being redundant token sequence features and retained token sequence features; Step 3: Construct a frame neighborhood motion consistency module, with redundant token sequence features and retained token sequence features as inputs, and dynamically reactivated token sequence features as outputs; Step 4: Perform multi-head self-attention interaction and forward propagation on the dynamically reactivated token sequence features, calculate motion consistency loss and action classification loss, and optimize the video action model using the stochastic gradient descent algorithm until convergence. Step 5: For the new video sequence, input it into the trained video action model to obtain the predicted action category probability distribution, and obtain the action category accordingly.

[0009] Furthermore, step 1 specifically includes the following steps: Step 11: Perform interval sampling on the input raw video to obtain a representative set of video frames. ,in, Represents a video frame; Represents the total number of frames, serving as a scalar parameter in the time dimension; These represent the pixel resolution of each frame in the vertical and horizontal directions, respectively; the number of channels (3) corresponds to the RGB color space, conforming to standard image representation conventions; this sampling strategy avoids the computational burden caused by redundant frames while ensuring the temporal coverage integrity of action segments. Since videos of varying lengths exist in practical applications, they can be standardized to a fixed number of frames through zero-padding or truncation. And during training, a masking mechanism is used to ignore invalid parts.

[0010] Step 12: Transfer video frames Divide into non-overlapping regions of fixed size, i.e., patches, with a size of [size missing]. , The side length of the patch is usually taken as... The video frames are divided into Each of the given patches is flattened into a one-dimensional vector and mapped to the feature dimension via a learnable linear projection matrix. , to obtain the original number Frame number Token characteristics: in, Represents a learnable linear projection matrix; Indicates the first Frame number One patch; obtained at this stage It only contains partial visual content information and does not include spatial location information; Step 13: Add position encoding to the patch obtained in Step 12 That is, the first Frame number Token characteristics: In the early stages of training a video action model, the position encoding vector is typically initialized randomly following a Gaussian or uniform distribution and used as a model parameter in backpropagation for updates. Additionally, to support action classification tasks, a learnable classification token vector is added. After being processed by the encoder, the final output of this vector will be extracted for action classification; the vector is then concatenated along its spatial location to obtain the first... Frame Token Features Each row of this matrix corresponds to the first row. Visual units at specific locations within a frame constitute the standard input form of ViT.

[0011] Furthermore, step 2 specifically includes the following steps: Step 21: To characterize the latent semantic structure of tokens within a frame, from a statistical perspective, assume that the first... Frame Token obeys by A Gaussian Mixture Model (GMM) is composed of semantic clusters, where each semantic cluster represents a set of latent distributions of tokens with similar semantic patterns in the feature space, and its probability density function is defined as: in, Indicates the first Each semantic center vector, i.e. the semantic cluster center, represents the mean position of the semantic pattern in the feature space; Represent the covariance matrix; And satisfy The mixing coefficient; Indicates a Gaussian distribution; To ensure numerical stability, it is assumed that each semantic cluster has the same isotropic covariance, i.e. ,in, Indicates temperature parameter; Let represent the identity matrix; at this point, the Gaussian mixture model degenerates into a soft clustering model based on Euclidean distance, which is easier to solve using the expectation-maximization algorithm. Step 22: Since the parameters of the Gaussian mixture model cannot be solved analytically directly, the Expectation-Maximization (EM) algorithm is used for iterative optimization to approximate the maximum likelihood estimate. To avoid the EM algorithm getting trapped in local optima, the Max-Min Distance method is used. Initialize each semantic center vector; randomly initialize the first semantic center vector. ;for The following samples, in turn, are selected as the new semantic cluster centers: Ensure that the initial semantic cluster centers are as dispersed as possible in the feature space, thereby covering different semantic regions, such as foreground objects, background textures, and motion edges; Step 23: In the Expectation Step of the Expectation Maximization algorithm, fix the current... semantic center vectors Calculate the first Frame number Token characteristics Belongs to the The soft affiliation probability of each semantic cluster: Among them, the smaller the Euclidean distance between the token feature and the semantic center vector, the higher the soft attribution probability, representing the basic assumption that neighboring points in the feature space are more likely to belong to the same semantic pattern; the denominator is normalized to ensure... Obtain the soft affiliation probability of all tokens to each semantic cluster; Step 24: In the Maximization Step of the Expectation-Maximization algorithm, use the soft-attribution probability obtained from the E-step. Update each semantic center vector: Among them, the new semantic center vector It is the first The weighted average of frame token features is used, with each weight representing the soft-association probability of the semantic cluster. The semantic cluster center is gradually moved to a data-dense region to better fit the latent semantic structure. The E-step and M-step are repeated until the change in the semantic cluster center is less than a preset threshold or the maximum number of iterations is reached. The convergence threshold of the expectation-maximization algorithm is... The maximum number of iterations is Complete the iterative optimization of Gaussian mixture model parameters to obtain stable semantic cluster partitioning results; Step 25: To assess the dominance of a semantic cluster in the current frame, define a semantic cluster importance index: in, From a global perspective, the importance of a certain semantic cluster in the current frame is characterized. For some tokens that are not prominent locally but important globally, even if the soft attribution probability is not high, they can still get a high score because the overall activity of the semantic cluster they belong to is high. Step 26: To further distinguish between semantically clear regions and ambiguous boundaries, define the Token redundancy entropy: This metric is derived from Shannon entropy and is used to assess the uncertainty of token attribution: low entropy indicates that the probability is concentrated on a certain semantic cluster, with clear semantics, such as the interior of a clear object; high entropy indicates that the probability is evenly distributed among multiple semantic clusters, with ambiguous semantics, such as object edges, noisy areas, and complex textures; therefore, high entropy tokens are often located at semantic boundaries or background noise areas, and have limited contribution to action recognition, and should be preferentially removed. Step 27: Combine the semantic cluster importance obtained from Step 25 and Step 26 and Token Redundancy Entropy Construct a token importance scoring function: in, Used to introduce a global semantic context and avoid evaluating individual tokens in isolation; This represents the decay factor, which suppresses high-entropy token scores. Step 28: Set the threshold Generate a binary mask: in, This is a feature for redundant tokens; To preserve token characteristics, the mask implements a hard transformation from continuous ratings to discrete choices. Step 29: Based on the binary mask generated in step 28 , will the The frame token features are divided into two mutually exclusive subsets: a retention set and a holding set. With redundant sets Redundant token features constitute a redundant set: in, The number of redundant tokens is represented; the characteristics of the retained tokens constitute the retained set: in, This indicates the number of tokens to retain; pruning tokens reduces subsequent computational overhead, while retaining tokens with higher contributions as input to the frame neighborhood motion consistency module.

[0012] Furthermore, step 3 specifically includes the following steps: Step 31: The tokens in the redundant set obtained in Step 29 may have been incorrectly pruned. Construct a motion trajectory predictor. It is used to predict the feature representation of the token in the current frame in the next frame; a multi-layer perceptron (MLP) with at least one hidden layer is used, and its structure is as follows: in, and Represents the learnable weight matrix; and Represents the bias vector; This represents a non-linear activation function, such as ReLU or GELU; This represents the spatial coordinates of the token in the original image; Step 32: After completing the Gaussian mixture model parameter iterative update in steps 21-24, obtain the latent semantic clusters of token features for each frame. Soft attribution probability and the corresponding semantic center vector Introducing a local metric matrix (Local Metric Matrix); for each semantic cluster Learn a shared trainable parameter vector ,in, For the token feature dimension, in the early stages of model training, the parameter vector Typically, the model is randomly initialized following a Gaussian or uniform distribution and used as model parameters in backpropagation for updates; the Softplus activation function is then used to... Mapped to positive definite weights: The local metric matrix corresponding to this semantic cluster is: ,in, vector Convert to a diagonal matrix; for any matrix belonging to the first... Tokens of semantic clusters That is, satisfying Its local metric matrix is ; Step 33: Based on the motion trajectory predictor in step 31 , obtained the Tokens in the Frame Redundancy Set In the Frame prediction feature representation: By obtaining the first [unit / item] through real spatial location relationships. In the frame and the first Frame number Token characteristics The most closely related real token features in spatial location ,in, This is a spatial location mapping function used to establish the correspondence between token indices at the same spatial location between adjacent frames; combined with the local metric matrix learned in step 32 for the semantic cluster to which the token belongs. Calculate predictive features With true characteristics Feature spatial distance between: in, This indicates the degree to which the actual motion deviates from the smooth trend. A value significantly greater than zero indicates that the token's movement is sudden or discontinuous, and may carry key action information; Step 34: To further capture drastic changes in motion direction, such as sudden stops, turns, and reversals, local curvature constraints are introduced within a three-frame window; the spatial correspondence is used to obtain the first... In the frame and the first Frame number Token characteristics Corresponding historical token characteristics Calculate the local curvature of the feature space of the token at the same spatial location within a three-frame window: when or If the time frame is not specified, the calculation for that frame is skipped to avoid exceeding the timing boundary.

[0013] Step 35: The result obtained by combining steps 33 and 34 and This determines whether to restore the pruned token, i.e., dynamic reactivation; for redundant sets... The token in the middle, if If so, then its mask is reset to 1, where, This is the curvature weighting coefficient; The threshold for dynamic reactivation; dynamic reactivation ensures that only tokens that deviate from a smooth trend and have undergone drastic directional changes will be restored; Step 36: After dynamic reactivation, obtain the dynamic reactivation token feature set. : in, The number of sparse tokens output in step 29. This represents the number of tokens recovered after dynamic reactivation. This set not only retains semantically important tokens within the frame but also supplements key tokens for cross-frame motion anomalies, serving as input for subsequent ViT operations and improving action recognition robustness while ensuring computational efficiency.

[0014] Furthermore, step 4 specifically includes the following steps: Step 41: Concatenate the dynamically reactivated token features from Step 36 along their spatial locations. ,in, For the first The actual number of tokens retained in the frame is calculated by appending a learnable classification token vector with added positional encoding to the token matrix. ,Right now Input to the ViT encoder, which is composed of It consists of alternating layers of multi-head self-attention (MSA) and feedforward network (MLP), with each layer containing LayerNorm pre-normalization and residual connections: in, and They represent the first Layer and first The layer encoder outputs the complete result; Indicates the first The layer encoder's MSA sublayer output; retrieve the classification token vector. Used for action classification prediction, where, This indicates that the first row of the matrix is ​​extracted. Step 42: Obtain the classification token vector using the method from step 41. Through linear classification head Mapped to the Logits space, where, Given the total number of action categories, the predicted probability distribution for each category is obtained by normalizing using the Softmax function: Where, vector This indicates the action classification result of the video action model for the current video segment; the index corresponding to the highest probability is the action category. .

[0015] Furthermore, step 5 specifically includes the following steps: Step 51: During the training process, simultaneously optimize action recognition accuracy and cross-frame motion consistency, and construct a composite loss function: in, Classification loss Using standard cross-entropy loss Supervision of action category prediction, among which, Ground Truth One-Hot Label; Motion Consistency Loss Local metric matrix used to guide step 3 With motion trajectory predictor To better fit the laws of motion in real space, specifically in the following forms: in, It is a redundant set; when or If the time limit is exceeded, skip the calculation for that frame to avoid exceeding the time boundary. Step 52: In the actual deployment phase, the video action model performs a gradient-free forward propagation process, sequentially performing steps 1-4 on the video samples to obtain dynamic reactivation token sequence features, which are then input into the ViT encoder to obtain the action category.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. The video action model pruning method based on semantic clustering local motion consistency of the present invention utilizes a semantic clustering pruning module. From a statistical modeling perspective, it assumes that tokens within the same frame follow a Gaussian mixture distribution. It iteratively optimizes the semantic cluster center vector and the soft-attribution probability of tokens through the expectation-maximization algorithm. Furthermore, it constructs a token importance scoring function by combining semantic cluster importance and redundancy entropy, and achieves a hard conversion from continuous scoring to discrete selection through binary masking. By characterizing the spatial semantic structure of key actions, using semantic cluster importance to measure the global semantic context, and using redundancy entropy to distinguish semantically clear regions from fuzzy boundaries, it can effectively remove redundant tokens such as static backgrounds and uniform textures, reducing computational overhead.

[0017] 2. The video action model pruning method based on semantic clustering local motion consistency of the present invention utilizes the frame neighborhood motion consistency module to learn local metric matrices for different semantic clusters and construct a motion trajectory predictor to model the motion trend of tokens at the same spatial location in adjacent frames. By calculating the feature spatial distance and feature spatial local curvature between the predicted spatial location and the real spatial location, the key action tokens of incorrect pruning are dynamically reactivated to ensure the motion consistency of temporal pruning.

[0018] 3. The video action model pruning method based on semantic clustering local motion consistency of the present invention adopts an end-to-end joint optimization strategy to construct a composite loss function that takes into account both classification accuracy and motion consistency, ensuring that pruning decisions and the backbone network evolve in tandem. This method can reduce the computational complexity of attention while ensuring the model's action recognition accuracy and real-time reasoning ability in complex dynamic scenes. Attached Figure Description

[0019] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In the drawings: Figure 1 This is an overall flowchart of the video motion model pruning method based on semantic clustering local motion consistency of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are for illustrative purposes only and are not intended to limit the scope of protection of this invention.

[0021] Example 1 The video motion model pruning method based on semantic clustering local motion consistency provided in this embodiment refers to... Figure 1 As shown, it includes the following steps: Step 1: Perform interval sampling on the original video to obtain a set of video frames. Divide each frame into a Patch and generate Token sequence features after linear projection and position encoding.

[0022] In this embodiment, step 1 specifically includes the following steps: Step 11: Perform interval sampling on the input raw video to obtain a representative set of video frames. ,in, Represents a video frame; Represents the total number of frames, serving as a scalar parameter in the time dimension; These represent the pixel resolution of each frame in the vertical and horizontal directions, respectively; the number of channels 3 corresponds to the RGB three-color space, which conforms to standard image representation conventions; This sampling strategy avoids the computational burden caused by redundant frames while ensuring the temporal coverage integrity of action segments. Since videos of varying lengths exist in practical applications, they can be standardized to a fixed number of frames through zero-padding or truncation. And during training, a masking mechanism is used to ignore invalid parts.

[0023] Step 12: Transfer video frames Divide into non-overlapping regions of fixed size, i.e., patches, with a size of [size missing]. , The side length of the patch is usually taken as... The video frames are divided into Each of the given patches is flattened into a one-dimensional vector and mapped to the feature dimension via a learnable linear projection matrix. , to obtain the original number Frame number Token characteristics: in, Represents a learnable linear projection matrix; Indicates the first Frame number One patch; obtained at this stage It only contains partial visual content information and does not include spatial location information; Step 13: Add position encoding to the patch obtained in Step 12 That is, the first Frame number Token characteristics: In the early stages of training a video action model, the position encoding vector is typically initialized randomly following a Gaussian or uniform distribution and used as a model parameter in backpropagation for updates. Additionally, to support action classification tasks, a learnable classification token vector is added. After being processed by the encoder, the final output of this vector will be extracted for action classification; the vector is then concatenated along its spatial location to obtain the first... Frame Token Features Each row of this matrix corresponds to the first row. Visual units at specific locations within a frame constitute the standard input form of ViT.

[0024] Step 2: Using the semantic clustering pruning module, the input is the token sequence features, and the output is the retained token sequence features and the redundant token sequence features.

[0025] In this embodiment, step 2 specifically includes the following steps: Step 21: To characterize the latent semantic structure of tokens within a frame, from a statistical perspective, assume that the first... Frame Token obeys by A Gaussian Mixture Model (GMM) is composed of semantic clusters, where each semantic cluster represents a set of latent distributions of tokens with similar semantic patterns in the feature space, and its probability density function is defined as: in, Indicates the first Each semantic center vector, i.e. the semantic cluster center, represents the mean position of the semantic pattern in the feature space; Represent the covariance matrix; And satisfy The mixing coefficient; Indicates a Gaussian distribution; To ensure numerical stability, it is assumed that each semantic cluster has the same isotropic covariance, i.e. ,in, Indicates temperature parameter; Let represent the identity matrix; at this point, the Gaussian mixture model degenerates into a soft clustering model based on Euclidean distance, which is easier to solve using the expectation-maximization algorithm. Step 22: Since the parameters of the Gaussian mixture model cannot be solved analytically directly, the Expectation-Maximization (EM) algorithm is used for iterative optimization to approximate the maximum likelihood estimate. To avoid the EM algorithm getting trapped in local optima, the Max-Min Distance method is used. Initialize each semantic center vector; randomly initialize the first semantic center vector. ;for The following samples, in turn, are selected as the new semantic cluster centers: Ensure that the initial semantic cluster centers are as dispersed as possible in the feature space, thereby covering different semantic regions, such as foreground objects, background textures, and motion edges; Step 23: In the Expectation Step of the Expectation Maximization algorithm, fix the current... semantic center vectors Calculate the first Frame number Token characteristics Belongs to the The soft affiliation probability of each semantic cluster: Among them, the smaller the Euclidean distance between the token feature and the semantic center vector, the higher the soft attribution probability, representing the basic assumption that neighboring points in the feature space are more likely to belong to the same semantic pattern; the denominator is normalized to ensure... Obtain the soft affiliation probability of all tokens to each semantic cluster; Step 24: In the Maximization Step of the Expectation-Maximization algorithm, use the soft-attribution probability obtained from the E-step. Update each semantic center vector: Among them, the new semantic center vector It is the first The weighted average of frame token features is used, with each weight representing the soft-association probability of the semantic cluster. The semantic cluster center is gradually moved to a data-dense region to better fit the latent semantic structure. The E-step and M-step are repeated until the change in the semantic cluster center is less than a preset threshold or the maximum number of iterations is reached. The convergence threshold of the expectation-maximization algorithm is... The maximum number of iterations is Complete the iterative optimization of Gaussian mixture model parameters to obtain stable semantic cluster partitioning results; Step 25: To assess the dominance of a semantic cluster in the current frame, define a semantic cluster importance index: in, From a global perspective, the importance of a certain semantic cluster in the current frame is characterized. For some tokens that are not prominent locally but important globally, even if the soft attribution probability is not high, they can still get a high score because the overall activity of the semantic cluster they belong to is high. Step 26: To further distinguish between semantically clear regions and ambiguous boundaries, define the Token redundancy entropy: This metric is derived from Shannon entropy and is used to assess the uncertainty of token attribution: low entropy indicates that the probability is concentrated on a certain semantic cluster, with clear semantics, such as the interior of a clear object; high entropy indicates that the probability is evenly distributed among multiple semantic clusters, with ambiguous semantics, such as object edges, noisy areas, and complex textures; therefore, high entropy tokens are often located at semantic boundaries or background noise areas, and have limited contribution to action recognition, and should be preferentially removed. Step 27: Combine the semantic cluster importance obtained from Step 25 and Step 26 and Token Redundancy Entropy Construct a token importance scoring function: in, Used to introduce a global semantic context and avoid evaluating individual tokens in isolation; This represents the decay factor, which suppresses high-entropy token scores. Step 28: Set the threshold Generate a binary mask: in, This is a feature for redundant tokens; To preserve token characteristics, the mask implements a hard transformation from continuous ratings to discrete choices. Step 29: Based on the binary mask generated in step 28 , will the The frame token features are divided into two mutually exclusive subsets: a retention set and a holding set. With redundant sets Redundant token features constitute a redundant set: in, The number of redundant tokens is represented; the characteristics of the retained tokens constitute the retained set: in, This indicates the number of tokens to retain; pruning tokens reduces subsequent computational overhead, while retaining tokens with higher contributions as input to the frame neighborhood motion consistency module.

[0026] Step 3: The frame neighborhood motion consistency module takes the retained token sequence features and redundant token sequence features as input and outputs the dynamically reactivated token sequence features as output.

[0027] In this embodiment, step 3 specifically includes the following steps: Step 31: The tokens in the redundant set obtained in Step 29 may have been incorrectly pruned. Construct a motion trajectory predictor. It is used to predict the feature representation of the token in the current frame in the next frame; a multi-layer perceptron (MLP) with at least one hidden layer is used, and its structure is as follows: in, and Represents the learnable weight matrix; and Represents the bias vector; This represents a non-linear activation function, such as ReLU or GELU; This represents the spatial coordinates of the token in the original image; Step 32: After completing the Gaussian mixture model parameter iterative update in steps 21-24, obtain the latent semantic clusters of token features for each frame. Soft attribution probability and the corresponding semantic center vector Introducing a local metric matrix (Local Metric Matrix); for each semantic cluster Learn a shared trainable parameter vector ,in, For the token feature dimension, in the early stages of model training, the parameter vector Typically, the model is randomly initialized following a Gaussian or uniform distribution and used as model parameters in backpropagation for updates; the Softplus activation function is then used to... Mapped to positive definite weights: The local metric matrix corresponding to this semantic cluster is: ,in, vector Convert to a diagonal matrix; for any matrix belonging to the first... Tokens of semantic clusters That is, satisfying Its local metric matrix is ; Step 33: Based on the motion trajectory predictor in step 31 , obtained the Tokens in the Frame Redundancy Set In the Frame prediction feature representation: By obtaining the first [unit / item] through real spatial location relationships. In the frame and the first Frame number Token characteristics The most closely related real token features in spatial location ,in, This is a spatial location mapping function used to establish the correspondence between token indices at the same spatial location between adjacent frames; combined with the local metric matrix learned in step 32 for the semantic cluster to which the token belongs. Calculate predictive features With true characteristics Feature spatial distance between: in, This indicates the degree to which the actual motion deviates from the smooth trend. A value significantly greater than zero indicates that the token's movement is sudden or discontinuous, and may carry key action information; Step 34: To further capture drastic changes in motion direction, such as sudden stops, turns, and reversals, local curvature constraints are introduced within a three-frame window; the spatial correspondence is used to obtain the first... In the frame and the first Frame number Token characteristics Corresponding historical token characteristics Calculate the local curvature of the feature space of the token at the same spatial location within a three-frame window: when or If the time frame is not specified, the calculation for that frame is skipped to avoid exceeding the timing boundary.

[0028] Step 35: The result obtained by combining steps 33 and 34 and This determines whether to restore the pruned token, i.e., dynamic reactivation; for redundant sets... The token in the middle, if If so, then its mask is reset to 1, where, This is the curvature weighting coefficient; The threshold for dynamic reactivation; dynamic reactivation ensures that only tokens that deviate from a smooth trend and have undergone drastic directional changes will be restored; Step 36: After dynamic reactivation, obtain the dynamic reactivation token feature set. : in, The number of sparse tokens output in step 29. This represents the number of tokens recovered after dynamic reactivation. This set not only retains semantically important tokens within the frame but also supplements key tokens for cross-frame motion anomalies, serving as input for subsequent ViT operations and improving action recognition robustness while ensuring computational efficiency.

[0029] Step 4: Calculate the motion consistency loss and action classification loss, and optimize the model using the stochastic gradient descent algorithm until convergence.

[0030] In this embodiment, step 4 specifically includes the following steps: Step 41: Concatenate the dynamically reactivated token features from Step 36 along their spatial locations. ,in, For the first The actual number of tokens retained in the frame is calculated by appending a learnable classification token vector with added positional encoding to the token matrix. ,Right now Input to the ViT encoder, which is composed of It consists of alternating layers of multi-head self-attention (MSA) and feedforward network (MLP), with each layer containing LayerNorm pre-normalization and residual connections: in, and They represent the first Layer and first The layer encoder outputs the complete result; Indicates the first The layer encoder's MSA sublayer output; retrieve the classification token vector. Used for action classification prediction, where, This indicates that the first row of the matrix is ​​extracted. Step 42: Obtain the classification token vector using the method from step 41. Through linear classification head Mapped to the Logits space, where, Given the total number of action categories, the predicted probability distribution for each category is obtained by normalizing using the Softmax function: Where, vector This indicates the action classification result of the video action model for the current video segment; the index corresponding to the highest probability is the action category. .

[0031] Step 5: Input the new video sequence into the trained video action model to obtain the predicted action category probability distribution, and obtain the action category accordingly.

[0032] In this embodiment, step 5 specifically includes the following steps: Step 51: During the training process, simultaneously optimize action recognition accuracy and cross-frame motion consistency, and construct a composite loss function: in, Classification loss Using standard cross-entropy loss Supervision of action category prediction, among which, Ground Truth One-Hot Label; Motion Consistency Loss Local metric matrix used to guide step 3 With motion trajectory predictor To better fit the laws of motion in real space, specifically in the following forms: in, It is a redundant set; when or If the time limit is exceeded, skip the calculation for that frame to avoid exceeding the time boundary. Step 52: In the actual deployment phase, the video action model performs a gradient-free forward propagation process, sequentially performing steps 1-4 on the video samples to obtain dynamic reactivation token sequence features, which are then input into the ViT encoder to obtain the action category.

[0033] The video motion model pruning method based on semantic clustering local motion consistency of the present invention uses a semantic clustering pruning module to characterize the latent semantic structure of tokens within a frame and remove redundant tokens such as static backgrounds and uniform textures; then it uses a frame neighborhood motion consistency module to forcibly restore pruned tokens when the actual motion deviates from a smooth trend or the curvature changes drastically.

[0034] The embodiments and implementation process of the present invention have been described in detail above with reference to the accompanying drawings and tables, but the present invention is not limited to the described embodiments. For those skilled in the art, various changes, modifications, substitutions, and variations can be made to these embodiments, including components, without departing from the principles and spirit of the present invention, and these variations still fall within the protection scope of the present invention.

Claims

1. A video action model pruning method based on semantic clustering local motion consistency, characterized in that, Includes the following steps: Step 1: Perform interval sampling on the original video to obtain a set of video frames. Divide each frame into a Patch, and generate Token sequence features after linear projection and position encoding. Step 2: Construct a semantic clustering pruning module. The input is the token sequence features, and the output is the retained token sequence features and the redundant token sequence features. Step 3: Construct a frame neighborhood motion consistency module, with the input being the preserved token sequence features and the redundant token sequence features, and the output being the dynamically reactivated token sequence features; Step 4: Perform multi-head self-attention interaction and forward propagation on the dynamically reactivated token sequence features, calculate motion consistency loss and action classification loss, and optimize the video action model using the stochastic gradient descent algorithm until convergence. Step 5: For the new video sequence, input it into the trained video action model to obtain the predicted action category probability distribution, and obtain the action category accordingly.

2. The method of claim 1, wherein, Step 1 specifically includes the following steps: Step 11: Perform interval sampling on the input raw video to obtain a representative set of video frames. ,in, Represents a video frame; Represents the total number of frames, serving as a scalar parameter in the time dimension; These represent the pixel resolution of the video frame in the vertical and horizontal directions, respectively; the number of channels (3) corresponds to the RGB color space; Step 12: Convert the video frames... Divide into non-overlapping regions of fixed size, i.e., patches, with a size of [size missing]. , The side length of the patch is usually taken as... The video frames are divided into Each of the given patches is flattened into a one-dimensional vector and mapped to the feature dimension via a learnable linear projection matrix. , to obtain the original number Frame number Token characteristics: in, Represents a learnable linear projection matrix; Indicates the first Frame number One patch; Step 13: Add position encoding to the Patch obtained from Step 12 , obtaining the first frame the Token feature: Adding a learnable classification Token vector which after being processed by the encoder, its final output will be extracted for action classification; along spatial position concatenation to obtain the frame Token feature Each row of the matrix corresponds to the visual unit of a specific position of the frame, which constitutes the standard input form of ViT.

3. The method of claim 2, wherein the pruning the video motion model based on the semantic clustering of local motion consistency. Step 2 specifically includes the following steps: Step 21 : Assuming the first The frame Token obeys a Gaussian mixture model composed of semantic clusters, where a semantic cluster represents a set of latent distributions in the feature space composed of Tokens with similar semantic patterns, whose probability density function is defined as: wherein, represents the i-th semantic centroid vector, i.e. semantic cluster centroid, representing the mean position of the semantic pattern in the feature space; represents the i-th semantic centroid vector, i.e. semantic cluster centroid, representing the mean position of the semantic pattern in the feature space; represents the covariance matrix; and satisfies is a mixing coefficient; represents a Gaussian distribution; To ensure numerical stability, it is assumed that each semantic cluster has the same isotropic covariance, i.e. ,in, Indicates temperature parameter; Let represent the identity matrix; at this point, the Gaussian mixture model degenerates into a soft clustering model based on Euclidean distance, which is easier to solve using the expectation-maximization algorithm. Step 22: Since the parameters of the Gaussian mixture model cannot be solved analytically directly, the expectation-maximization algorithm is used for iterative optimization to approximate the maximum likelihood estimate; to avoid the expectation-maximization algorithm getting trapped in local optima, the maximum-minimum distance method is used. Initialize each semantic center vector; randomly initialize the first semantic center vector. ;for The following samples, in turn, are selected as the new semantic cluster centers: Ensure that the initial semantic cluster centers are as dispersed as possible in the feature space, thereby covering different semantic regions; Step 23: In the expectation step of the expectation-maximization algorithm, fix the current... semantic center vectors Calculate the first Frame number Token characteristics Belongs to the The soft affiliation probability of each semantic cluster: Wherein, the smaller the Euclidean distance between the Token feature and the semantic center vector, the higher the soft attribution probability; the denominator realizes normalization, ensuring obtain the first frame Token feature soft attribution probability of each semantic cluster; Step 24: In the maximization step of the expectation-maximization algorithm, the soft membership probabilities obtained in the expectation step are used Update each semantic centroid vector: Among them, the new semantic center vector It is the first The weighted average of frame token features, with each weight representing the soft-association probability of the semantic cluster; the semantic cluster center gradually moves to a data-dense region; the expectation step and maximization step are repeated until the change in the semantic cluster center is less than a preset threshold or the maximum number of iterations is reached, where the convergence threshold of the expectation-maximization algorithm is... The maximum number of iterations is The Gaussian mixture model parameters were iteratively optimized to obtain stable semantic cluster partitioning results.

4. The video action model pruning method based on semantic clustering local motion consistency according to claim 3, characterized in that, Step 2 also includes the following steps: Step 25: To assess the dominance of a semantic cluster in the current frame, define a semantic cluster importance index: in, From a global perspective, the importance of a semantic cluster in the current frame is characterized. For some tokens that are not prominent locally but important globally, even if the soft affiliation probability is not high, they can still get a high score because the overall activity of the semantic cluster is high. Step 26: To further distinguish between semantically clear regions and ambiguous boundaries, define the Token redundancy entropy: Step 27: Combine the semantic cluster importance obtained from Step 25 and Step 26 and Token Redundancy Entropy Construct a token importance scoring function: wherein, for introducing a global semantic context, avoiding isolated evaluation of individual Tokens; denotes a decay factor, suppressing high-entropy Token scores.

5. The method of claim 4, wherein, Step 2 also includes the following steps: Step 28: Set threshold , generate binary mask: wherein, is a redundant Token feature; is a reserved Token feature; Step 29: Based on the binary mask generated in step 28 , will the The frame token features are divided into two mutually exclusive subsets: a retention set and a holding set. With redundant sets Redundant token features constitute a redundant set: wherein, represents the number of tokens in the redundancy set; the reserved token feature constitutes the reserved set: wherein, denotes the reserved set Token number; the pruned Token reduces the subsequent calculation overhead, while the Token with higher contribution is reserved as the input of the frame neighborhood motion consistency module.

6. The method of claim 5, wherein, Step 3 includes the following steps: Step 31: The tokens in the redundant set obtained in Step 29 may have been incorrectly pruned. Construct a motion trajectory predictor. It is used to predict the feature representation of the token in the current frame in the next frame; a multi-layer perceptron (MLP) with at least one hidden layer is used, and its structure is as follows: wherein, and denote learnable weight matrices; and denote bias vectors; denotes a non-linear activation function, such as ReLU or GELU; denotes the spatial coordinates of the Token in the original image; Step 32: After completing the Gaussian mixture model parameter iterative update in steps 21-24, obtain the latent semantic clusters of token features for each frame. Soft attribution probability and the corresponding semantic center vector Introducing a local metric matrix (Local Metric Matrix); for each semantic cluster Learn a shared trainable parameter vector ,in, For the token feature dimension, in the early stages of model training, the parameter vector Typically, the model is randomly initialized following a Gaussian or uniform distribution and used as model parameters in backpropagation for updates; the Softplus activation function is then used to... Mapped to positive definite weights: The local metric matrix corresponding to this semantic cluster is: ,in, vector Convert to a diagonal matrix; for any matrix belonging to the first... Tokens of semantic clusters That is, satisfying Its local metric matrix is .

7. The video action model pruning method based on semantic clustering local motion consistency according to claim 6, characterized in that, Step 3 also includes the following steps: Step 33: Based on the motion trajectory predictor in step 31 , obtained the Tokens in the Frame Redundancy Set In the Frame prediction feature representation: By obtaining the first [unit / item] through real spatial location relationships. In the frame and the first Frame number Token characteristics The most closely related real token features in spatial location ,in, This is a spatial location mapping function used to establish the correspondence between token indices at the same spatial location between adjacent frames; combined with the local metric matrix learned in step 32 for the semantic cluster to which the token belongs. Calculate predictive features With true characteristics Feature spatial distance between: wherein, represents the degree to which the actual motion deviates from the smoothed trend, and if is significantly larger than zero, it indicates that the motion behavior of the Token has a burstiness or non-continuity, possibly carrying key action information; Step 34: Introduce local curvature constraints within a three-frame window; obtain the first frame through spatial correspondence. In the frame and the first Frame number Token characteristics Corresponding historical token characteristics Calculate the local curvature of the feature space of the token at the same spatial location within a three-frame window. ,when or If the time frame is not specified, the calculation for that frame is skipped to avoid exceeding the timing boundary.

8. The method of claim 7, wherein, Step 3 also includes the following steps: Step 35: The result obtained by combining steps 33 and 34 and This determines whether to restore the pruned token, i.e., dynamic reactivation; for redundant sets... The token in the middle, if If so, then its mask is reset to 1, where, This is the curvature weighting coefficient; The threshold for dynamic reactivation; dynamic reactivation ensures that only tokens that deviate from a smooth trend and have undergone drastic directional changes will be restored; Step 36: After dynamic reactivation, obtain the dynamic reactivation token feature set. : in, The number of sparse tokens output in step 29. This represents the number of tokens that have been restored after dynamic reactivation.

9. The method of claim 8, wherein, Step 4 specifically includes the following steps: Step 41: Concatenate the dynamically reactivated token features from Step 36 along their spatial locations. ,in, For the first The actual number of tokens retained in the frame is calculated by appending a learnable classification token vector with added positional encoding to the token matrix. ,Right now Input to the ViT encoder, which is composed of It consists of alternating layers of multi-head self-attention (MSA) and feedforward network (MLP), with each layer containing LayerNorm pre-normalization and residual connections: in, They represent the first and the The layer encoder outputs the complete result; Indicates the first The layer encoder's MSA sublayer output; retrieve the classification token vector. Used for action classification prediction, where, This represents the first row of the matrix; Step 42: Obtain the classification token vector using the method from step 41. Through linear classification head Mapped to the Logits space, where, Given the total number of action categories, the predicted probability distribution for each category is obtained by normalizing using the Softmax function: Where, vector This indicates the action classification result of the video action model for the current video segment; the index corresponding to the highest probability is the action category. .

10. The video action model pruning method based on semantic clustering local motion consistency according to claim 9, characterized in that, Step 5 specifically includes the following steps: Step 51: During the training process, simultaneously optimize action recognition accuracy and cross-frame motion consistency, and construct a composite loss function: in, Classification loss Using standard cross-entropy loss Supervision of action category prediction, among which, Ground Truth One-Hot Label; Motion Consistency Loss Local metric matrix used to guide step 3 With motion trajectory predictor To better fit the laws of motion in real space, specifically in the following forms: wherein, is a redundant set; when or the frame calculation is skipped to avoid timing boundary overflow. Step 52: In the actual deployment phase, the video action model performs a gradient-free forward propagation process, sequentially performing steps 1-4 on the video samples to obtain dynamic reactivation token sequence features, which are then input into the ViT encoder to obtain the action category.

Images