Video action detection method and device based on key frame filtering pixel block and medium

Abstract

A video action detection method and device based on key frame filtering pixel blocks and a medium are provided.The detector is constructed to perform action detection on an input video segment, the detector performs feature extraction on a video frame and token filtering centered on a key frame, then performs character positioning on the key frame based on a query, and finally performs relationship modeling based on an extended RoI and a compact space-time context to predict multiple actions that the character may perform.The application proposes an accelerated video action detector EVAD, a token selection module centered on a key frame is proposed on the network structure of vanilla ViT to gradually delete invalid tokens in the features of non-key frames, and character positioning and action classification are predicted in an end-to-end manner, which greatly improves the inference speed of the model and is friendly to real-time action detection.

Description

Technical Field

[0001] This invention belongs to the field of computer technology and relates to spatiotemporal motion detection technology. It is a video motion detection method, device and medium based on key frame filtering of pixel blocks. Background Technology

[0002] Transformers are widely used in various computer vision tasks, such as image classification, object detection, and video classification, achieving considerable results. Furthermore, some methods are exploring new Transformer architectures to improve model recognition performance. The quadratic complexity in the Transformer's self-attention module becomes a computational bottleneck when transferred to computer vision. This is especially true when applied to more data-intensive video tasks, where Transformers can significantly increase computational costs. Simultaneously, videos inherently possess high spatiotemporal redundancy, with adjacent frames sharing similar semantic information; therefore, retaining all visual pixel blocks (tokens) leads to computational waste.

[0003] Video action detection is a complex video understanding task, and common frame-level action detectors suffer from redundancy issues: an input video segment consists of keyframes and adjacent context frames, and the detector outputs the localization of people and the corresponding action classification results on the keyframes. Stacking more non-keyframes can bring effective contextual information, but inevitably introduces redundant information and additional computation. Furthermore, due to the movement of the actor or the camera, the actor's spatial position may change between adjacent frames. Therefore, using the predicted bounding boxes obtained from keyframes to extract actor features across the entire video segment will lose some actor information (beyond the predicted bounding box range). This invention finds that if redundant information in the video sequence can be removed, retaining only the information effective for action detection, the number of tokens involved in the computation can be reduced, improving model speed. On the other hand, since background information is removed, the region of the predicted bounding box can be appropriately expanded to extract complete human features without introducing interfering information. Summary of the Invention

[0004] The problem this invention aims to solve is that existing frame-level spatiotemporal action detectors take continuous video frames as input and output character localization and action classification results on keyframes. Stacking more non-keyframes can bring effective contextual information, but it also becomes a bottleneck limiting the performance and speed of spatiotemporal action detectors. In particular, when using a Transformer-like model as the base network, the secondary computational complexity of the joint spatiotemporal attention mechanism leads to a significant increase in computational cost.

[0005] The technical solution of this invention is as follows: An accelerated video action detector (EVAD) is constructed to detect action in input video segments. The implementation of the EVAD detector includes an input sample generation stage, a network configuration stage, a training stage, and a testing stage, as detailed below:

[0006] 1) Generate input samples: Taking the frame with label information as the center, i.e. the key frame, extract the temporal context of frame a forward and backward to form the input frame sequence of 2a frames, and then uniformly sample frame b as the input video sequence of the detector.

[0007] 2) Extracting video sequence features: Using video ViT as a feature extraction network, features are extracted from the video sequence generated in step 1) to obtain a token sequence f representing video features with the temporal resolution unchanged and the spatial resolution downsampled by 16 times;

[0008] 3) Keyframe-centric token selection: A keyframe-centric token selection mechanism is introduced in some MHSA and FFN layers of ViT to reduce the number of non-keyframe tokens involved in the calculation and reduce the spatiotemporal redundancy within the input. First, valid tokens are defined, including all tokens from keyframes and tokens with high importance scores from non-keyframes. The importance scores are characterized by an attention map. For each non-keyframe, a set proportion of important tokens are retained. Token selection is performed every 1 / 4 of the total number of layers in the ViT encoder, discarding redundant tokens and retaining valid tokens to be sent to the subsequent FFN at the current position. This process is repeated three times.

[0009] 4) Query-based keyframe person localization: The intermediate layer feature maps of the ViT encoder, which has undergone token filtering design in step 3), are taken out at intervals of 1 / 4 of the total number of layers, and up / downsampled to form a multi-scale feature map set for the keyframe. This set is then fed into the FPN for feature fusion. Next, the person localization branch uses a query-based method to predict N bounding box coordinates (bbox) and corresponding confidence scores (conf) on the multi-scale feature map set of the keyframe. conf represents the probability that the box contains a person.

[0010] 5) Human action classification based on extended RoI and compact spatiotemporal context: Action category prediction is performed by the action classification branch, and an empty feature map f is initialized. blank The M tokens retained from step 3) are reduced in dimensionality using a linear layer, and then placed in f according to their corresponding spatiotemporal positions. blankIn the middle, the remaining positions are filled with 0 to form a spatiotemporal feature map. Then, the bounding boxes predicted in step 4) are expanded to a certain extent. RoIAlign is performed on this spatiotemporal feature map to obtain the RoI features of N people. Subsequently, a context interaction decoder is constructed to model the scene between the person features and the context information from the ViT encoder. The RoI features of N people are concatenated with M spatiotemporal tokens and fed into a stacked 6-layer decoder network. Each layer consists of MHSA and FFN, consistent with the ViT encoder. The N RoI features in the output are extracted and processed by MLP for the final action classification to obtain the final action category prediction result.

[0011] 6) Training Phase: Initialize ViT from step 2) using pre-trained and fine-tuned weights provided by VideoMAE's Kinetics. Initialize the remaining new layers using Xavier. The character localization branch utilizes ensemble prediction loss to achieve optimal binary matching between predictions and ground truth values. The ensemble loss function L... set Includes: L1 loss of the bounding box L L1 GIoU loss of bounding box GIoU And confidence loss L conf The action classification branch is composed of the action classification loss L. act This means that only the predicted values ​​that successfully match the true values ​​during the calculation of the set prediction loss are calculated. The four loss functions are weighted according to the set ratio and optimized by the AdamW optimizer. This process is repeated until the number of iterations is reached.

[0012] 7) Testing phase: Given an input video segment, proceed through steps 1) to 5) to obtain the motion detection results on keyframes, and verify the motion detection performance of the constructed detector;

[0013] Step 1) corresponds to the input sample generation stage, steps 2) to 5) correspond to the network configuration stage, step 6) corresponds to the training stage, and step 7) corresponds to the testing stage.

[0014] This invention reveals that, compared to keyframes which retain the complete outline of the actor, non-keyframes have a relatively weaker correlation with action semantics. To improve the efficiency of action detection, this invention proposes a keyframe-centric token selection module on the vanilla ViT network structure to progressively remove invalid information from non-keyframe features. Based on this module, this invention designs a detection model for efficient action detection, called the Accelerated Video Action Detector (EVAD). In EVAD, only the tokens most relevant to action semantics in non-keyframes are retained to assist in the final action classification. For example, in non-keyframes, people's eyes and mouths are associated with the "talk to" action, and waving hands are associated with "point to," while the remaining tokens are redundant and should be discarded during the selection process. This invention introduces a token selection mechanism in the feature extraction stage of video images. In each selection, a set proportion of tokens are retained, consisting of two parts: 1) all tokens in keyframes, and 2) tokens with high importance scores in non-keyframes, with the score represented by the attention value after weighted averaging of keyframe queries. Therefore, it is called a "keyframe-centric" selection mechanism. Taking the ViT-B backbone network as an example, this invention performs token selection every three layers. Then, the expanded prediction bounding box is used to perform RoIAlign on the preserved feature map to capture the complete spatiotemporal information of the actor. Subsequently, a Transformer decoder is used to perform context modeling between the actor's region of interest (RoI) features and the compact context information extracted from the encoder, achieving better results than some previously meticulously designed action detection heads. Experiments on three action detection benchmark datasets—AVA, UCF101-24, and JHMDB—demonstrate the performance advantages of EVAD. Compared to vanilla ViT, EVAD reduces detection accuracy by 43% while improving real-time inference speed by 40%. Furthermore, even with comparable computational costs, EVAD can improve detection performance by 1% with higher resolution input.

[0015] The present invention has the following advantages compared with the prior art.

[0016] This invention takes into account the inherent high redundancy of videos and the semantic similarity between adjacent frames, and proposes a keyframe-centered token selection algorithm to eliminate spatiotemporal redundant information while maintaining the accuracy of action detection, thereby saving computational costs.

[0017] Based on the token selection module, this invention designs an end-to-end spatiotemporal action detector that can rival the detection performance of the best current two-stage models.

[0018] This invention is the first end-to-end action detection model using a single, non-hierarchical Transformer network structure. Combined with a low-retention token selection strategy, it significantly improves the model's inference speed and is highly compatible with real-time action detection. Furthermore, EVAD exhibits strong scalability and portability, serving as an efficient and concise action detection baseline applicable to a wider range of action detection models. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention.

[0020] Figure 2 This is a diagram of the detection framework used in this invention.

[0021] Figure 3 This is a network structure diagram of the EVAD feature extraction network proposed in this invention.

[0022] Figure 4 This is a schematic diagram of the keyframe-centered token selection module proposed in this invention.

[0023] Figure 5 This is a diagram illustrating token retention after applying a token selection strategy.

[0024] Figure 6 This is a schematic diagram of the query-based location branch proposed in this invention.

[0025] Figure 7 This is a schematic diagram of the action classification branches proposed in this invention.

[0026] Figure 8 This is a schematic diagram of expanding the RoI range to cover the complete trajectory of a person. Detailed Implementation

[0027] This invention proposes an accelerated video action detector, EVAD. Leveraging the inherent high spatiotemporal redundancy and semantic similarity between adjacent frames in videos, it introduces a keyframe-centric token selection module on the vanilla ViT network structure to progressively remove invalid tokens from non-keyframe features, and predicts person localization and action classification in an end-to-end manner. Figure 1 As shown, the detector performs feature extraction on video frames and keyframe-centric token filtering, then locates characters in keyframes based on queries, and finally models relationships based on extended RoIs and compact spatiotemporal context to predict multiple actions that characters may perform.

[0028] The detector of this invention consists of three parts: First, a sequence of video frames sampled at a fixed step size is input into the EVAD feature extraction network for feature extraction. During the feature extraction stage, our designed token selection module is inserted at equal intervals to retain the non-keyframe tokens most relevant to the action detection task. Next, two detection head branches are executed sequentially: (1) Person localization branch: predicts the position of persons on keyframes in a set of video frames; (2) Action classification branch: performs multi-label action classification of persons on keyframes based on contextual information. The combined outputs of the person localization branch and the action classification branch are used as the final spatiotemporal action detection result.

[0029] The method of the present invention includes the following steps:

[0030] (1) An efficient token selection mechanism for action detection characteristics was designed. During the feature extraction stage, non-key frame tokens participating in the calculation were gradually eliminated, which reduced redundancy in the video input and significantly improved the running speed of the model.

[0031] (2) In the character localization branch, the intermediate layer feature maps from the feature extraction stage are upsampled / downsampled to form multi-scale feature maps for the keyframes, and then fed into the feature pyramid network (FPN) for multi-scale fusion. Next, N candidate boxes on the keyframes are predicted using a query-based method.

[0032] (3) In the action classification branch, the N candidate boxes obtained from the localization branch are appropriately expanded, and then the RoIAlign operation is performed on the reconstructed spatiotemporal feature map to obtain RoI features containing complete information about the person. Subsequently, contextual information modeling is performed between the RoI features and the compact context features output by the encoder. The updated RoI features are then processed by MLP for multi-label action prediction. Finally, the outputs of the two branches are directly combined without additional post-processing to obtain the final action detection result on the keyframe.

[0033] The implementation of the Accelerated Video Action Detector (EVAD) of the present invention includes an input sample generation stage, a network configuration stage, a training stage, and a testing stage. The implementation of the present invention is described in detail below, with step 1) corresponding to the input sample generation stage, steps 2) to 5) corresponding to the network configuration stage, step 6) corresponding to the training stage, and step 7) corresponding to the testing stage.

[0034] 1) Generating Input Samples: For both training and testing videos, the frame containing the label information is used as the center (keyframe). 32 frames of temporal context are extracted forward and backward to form a 64-frame input frame sequence. Then, 16 frames are uniformly sampled at a step size of 4 as input to the detector model. During training, for each RGB image frame, random scaling is performed, setting the short side of the image to 256–320 pixels and the long side to no more than 1333 pixels. Next, data augmentation such as random horizontal flipping and color dithering is applied to the image frames. During testing, for each image frame, the short side is scaled to 256 pixels without additional data augmentation. We will refer to the training and testing input sequences collectively as... T represents the frame number, and H and W represent the width and height of the graphic.

[0035] The specific process is as follows:

[0036] 1.1) The original video sequence V obtained by extracting frames from the input video is as follows:

[0037] V={Img -32 ,…,Img -2 1mg -1 ,Img0,Img1,Img2,…,Img 31}

[0038] Where Img0 represents the keyframe, sequence numbers -32 to -1 represent the frame sequence to the left of the keyframe, and sequence numbers 1 to 31 represent the frame sequence to the right of the keyframe. The original video has an FPS of 30, and V contains approximately 2 seconds of context information.

[0039] 1.2) The video sequence I after sampling V with a fixed step size is as follows:

[0040] I={Img -32 ,…,Img -8 1mg -4 ,Img0,Img4,Img8,…,Img 28}

[0041] Where Img0 still represents a keyframe, Img i The relative sequence number of the keyframe remains unchanged, and sampling is performed with a fixed step size of 4. After a series of data processing and data augmentation, I is used as the input of the model.

[0042] 2) Extracting video sequence features: Using Video ViT as the basic network structure, feature extraction is performed on the input sequence I generated in 1), extracting a token sequence representing video features. Specifically, the video ViT first performs pixel tokenization, dividing I into non-overlapping blocks. There are 2×16×16 cubes. Each cube is then mapped to a 3D token using cube embedding. Next, positional encoding information is added to all tokens, which are then fed into a stacked L-layer encoder network. Each layer consists of a multi-head self-attention layer (MHSA) and a feed-forward network (FFN). The encoder output f serves as the spatiotemporal feature map of the video for subsequent detection processes. In our system, when using ViT-Base as the base network, the channel dimension D = 768 and the number of encoder layers L = 12; when using ViT-Large, D = 1024 and L = 24.

[0043] Step 2) is specifically implemented as follows:

[0044] 2.1) Perform pixel block tokenization on the input sequence I, dividing I into non-overlapping cubes:

[0045]

[0046] Where T, H, and W represent the temporal length of the video, the height of the video frame, and the width of the video frame, respectively. Each cube is 2×16×16 in size, and the total number of cubes is...

[0047] 2.2) Use a cube embedding layer to map each cube into a 3D token, forming the video sequence x:

[0048]

[0049] The CubeEmbedding functionality can be implemented using a 3D convolution operation with a kernel and stride of (2,16,16), and the number of output channels of the convolution is D.

[0050] 2.3) Add 3D sine and cosine position codes pos to the video sequence x:

[0051]

[0052] x = x + pos

[0053] in This represents the location encoding at the default resolution. Since the aspect ratio of the images in the dataset is not fixed, bicubic interpolation is required to perform online interpolation on the spatial location encoding of pos0.

[0054] 2.4) Use an L-layer Transformer encoder to extract features from the video sequence x, with x as the input to each encoder layer. l-1 The output is x l :

[0055] x′ l =MHSA(LN(x l-1 ))+x l-1

[0056] x l =FFN(LN(x′) l ))+x′ l

[0057] Where MHSA represents a multi-head self-attention layer, FFN represents a feedforward network, and LN represents a LayerNorm. The output x of the Lth layer... L This serves as a spatiotemporal feature of the video for use in subsequent detection processes.

[0058] 3) Keyframe-centric token selection: An attention-based token selection mechanism is introduced within the basic network structure of step 2). Similar to EViT, a token selection step is added between MHSA and FFN in some layers to ensure that a small number of high-quality tokens are passed down for action localization and classification, thereby reducing spatiotemporal redundancy among tokens. To retain low-redundancy and more effective tokens during selection, we first need to define which tokens are valid. In the token selection method of this invention, valid tokens consist of two parts: 1) tokens from keyframes, and 2) tokens with high attention / importance scores in non-keyframes. In spatiotemporal action detection tasks, the bounding box position of a person and the category of an action both depend on keyframe information; other frames only play a supporting role in action classification. Therefore, this invention defines all tokens in keyframes as valid tokens.

[0059] For measuring the importance of tokens in non-keyframes, this invention uses a pre-computed attention map to characterize the importance of each token without introducing additional learnable parameters or large computational overhead. First, the attention map's num_heads dimension is averaged to obtain an RxR matrix representing the attentiveness between tokens (ignoring batch size). For example, attn(i,j) represents the importance that query i considers pixel block j to have. Based on the previous analysis, keyframes are more important for human localization and action categories; tokens belonging to keyframes should play a greater role in importance calculation. Therefore, keyframe queries are given larger weights to better retain tokens more relevant to keyframes. In other words, this invention filters out some tokens that only have high responses to non-keyframes, as these tokens are redundant for the current sample. Next, a weighted average is performed on the query dimension to calculate the importance score of each token. Then, the top N tokens from the N2 non-keyframe tokens are selected in descending order of importance score. t ×ρ-N1 tokens, where N t N1 and N2 represent the total number of tokens, keyframe tokens, and non-keyframe tokens in the current sample, respectively. ρ represents the token retention rate; the model achieves the best performance-efficiency balance when ρ = 70%, so it is set as the system default. After token selection, the retained tokens are fed into the subsequent FFN of the encoder layer. This invention performs token selection three times. The first token selection is performed at 1 / 3 of the encoder layer to ensure the model has a high level of semantic representation capability. That is, token selection is performed every 1 / 4 of the total layers, discarding redundant tokens and retaining valid tokens. Through multiple selections, the number of tokens is significantly reduced, reducing unnecessary computation and accelerating the training and prediction process. The original input sequence I passes through the video ViT network with a token selection mechanism, and the video features are updated as follows:

[0060] An example of the keyframe-centric token selection mechanism described above is as follows:

[0061] Taking ViT-Base (12 layers) as an example, the insertion position of the token selection module is as follows:

[0062] Encoder={L0, L1, L2, T3, L4, L5, T6, L7, L8, T9, L 10 L 11}

[0063] T3, T6, T9, etc. represent encoder layers with keyframe-centered token selection modules, while L0, L1, L2, etc. represent ordinary encoder layers in step 2).

[0064] For layers T3, T6, and T9, a token selection process is added between the MHSA and FFN modules. The selection process is as follows:

[0065] 3.1) Use a pre-computed self-attention matrix to measure the importance of each token:

[0066]

[0067] Where attn is an R×R matrix, attn(i,j) represents the importance that query i considers to pixel block j, R is the number of input tokens, D represents the channel dimension of the query, and Head represents the number of heads.

[0068] 3.2) Apply greater weight to keyframe queries to better retain tokens that are more relevant to the keyframes, such as the importance of pixel block j (Imp). j It can be represented as:

[0069]

[0070] Where N t N1 and N2 represent the total number of tokens, keyframe tokens, and non-keyframe tokens, respectively. The first N1 tokens belong to keyframes. The weight w is a hyperparameter, which is preferably set to 4 in this invention to increase the weight of keyframe tokens.

[0071] 3.3) Select tokens based on importance score Imp:

[0072] selected_tokens=topK(tokens,Imp,N t ×ρ-N1)

[0073] Where tokens represent the input tokens, selected_tokens represent the tokens retained after selection, and ρ represents the token retention rate (keep rate), which is preferably set to 70% in this invention. topK selects the top N tokens from the N2 non-keyframe tokens in descending order of Imp. t ×ρ-N1 tokens are returned along with N1 keyframe tokens as input to the subsequent network.

[0074] Step 3) A keyframe-centric token selection mechanism is introduced within the video ViT network. Leveraging the inherent high redundancy of videos and the semantic similarity of adjacent frames, redundant information in non-keyframes is gradually eliminated during feature extraction, reducing unnecessary computation and significantly accelerating model execution. Unlike conventional methods that directly extract features from the input video using Transformers while ignoring the computational waste caused by information redundancy from stacked temporal context, this invention addresses the high redundancy in the input to improve action detection efficiency. During feature extraction, three token selection modules are evenly inserted into the basic network structure without introducing new model parameters. The keyframe-centric token selection mechanism uses a pre-computed self-attention matrix to calculate an importance score for each non-keyframe token, assigning greater weight to queries belonging to keyframes to highlight their crucial role in action detection. Then, the top 70% of tokens are retained in descending order of importance score, resulting in a more compact spatiotemporal representation of the video for subsequent detection.

[0075] 4) Keyframe-based character localization based on queries: The intermediate layer feature maps of the base network from step 3), i.e., the ViT encoder with token filtering design, are extracted at intervals equal to 1 / 4 of the total number of layers. These are then upsampled / downsampled from shallow to deep to form keyframe feature map sets at four scales using nearest neighbor interpolation. The resulting spatial resolutions are as follows: Next, the obtained hierarchical features are fed into the FPN for dimensionality reduction and feature fusion, allowing shallow features to retain detailed information while possessing deep semantics. The person localization branch uses a query-based approach to predict N bounding box coordinates on this keyframe feature set. and the corresponding confidence score The bounding box represents the probability that a person is contained within it. Following the approach of Sparse R-CNN, this invention sets up N learnable candidate bounding boxes and corresponding candidate features. The person localization branch consists of 6 layers, each layer comprising a self-attention layer that interacts between candidate features, a dynamic instance interaction layer that interacts between candidate features and corresponding RoI features, and an FFN layer. The output of each layer then passes through a regression layer and a classification layer to obtain a revised candidate bounding box and a corresponding confidence score. The final layer updates the candidate bounding boxes and corresponding confidence scores as the final result of human detection.

[0076] Step 4) The character localization branch generates the character bounding box and confidence score on the keyframes as follows:

[0077] 4.1) Extract the intermediate layer features of the keyframes f = {f2, f5, f8, f...} 11}, and uses nearest neighbor interpolation to obtain multi-scale features:

[0078]

[0079]

[0080]

[0081]

[0082] Where f2, f5, f8, f 11 Let f represent the video features output by the corresponding layer in the network structure of step 3). Interpolate(f, g) means that the spatial resolution of the video feature f is interpolated to g times using the nearest neighbor method.

[0083] 4.2) Feed multi-scale features into FPN for dimensionality reduction and feature fusion:

[0084] {p2, p5, p8, p 11}=FPN({f′2,f′5,f′8,f′ 11})

[0085] Where f′2, f′5, f′8, f′ 11 The channel dimensions are D, p2, p5, p8, p 11 The channel dimension is d. In this embodiment, D = 768 / 1024 and d = 256.

[0086] 4.3) Set N learnable candidate boxes and corresponding candidate features Update candidate bounding boxes and candidate features using a 6-layer localization branch. For each layer:

[0087] First, select a feature map of appropriate scale based on the size of prop_bbox to extract the RoI feature roi_feat. l :

[0088]

[0089] Where S H ×S W This represents the spatial resolution of the RoI feature, which is set to 7×7 in our system.

[0090] Next, using the prop_feat from the previous layer l-1 and the newly generated roi_feat l Update candidate features:

[0091] prop_feat′ l =LN(Dropout(MHSA(prop_feat)l-1 ))+prop_feat l-1 )

[0092]

[0093] prop_feat l =LN(Dropout(FFN(prop_feat″ l ))+prop_feat″ l )

[0094] Among them prop_feat′ l ,prop_feat″ l The intermediate result is represented by MHSA, FFN, and LN, which are the same network structures of the encoder in step 2). inst_interact represents a dynamic instance interaction layer that performs 1×1 dynamic convolution on the corresponding RoI features using the convolution parameters generated by prop_feat.

[0095] Finally, bounding box and confidence regression are performed separately to obtain the candidate bounding box prop_bbox for each layer. l and confidence level conf l As intermediate layer supervision signals during the training process:

[0096]

[0097]

[0098] In step 4) of this invention, keyframe character localization generates a hierarchical multi-scale feature map from the non-hierarchical output in step 3), and uses a query-based method to predict the character's position and confidence level in the keyframe on this feature map set. This invention is the first method to effectively combine the Transformer base network with a query-based detection method, and also the first method to achieve character localization using a non-hierarchical Transformer model, thus realizing an end-to-end approach.

[0099] 5) Character Action Classification Based on Extended RoI and Compact Spatiotemporal Context: Unlike the output obtained in the conventional feature extraction stage, step 3) yields M discrete spatiotemporal tokens. This invention requires the spatiotemporal structure of the feature map to be recovered before subsequent position-related operations such as RoIAlign can be performed. An empty feature map is initialized. The retained tokens are reduced in dimensionality using a linear layer, and then placed in f according to their corresponding spatiotemporal positions. blank In the middle, the remaining positions are filled with 0, which serves as the spatiotemporal feature map for subsequent operations.

[0100] The next step is to obtain the features of N people for action prediction. The conventional approach is to use the bounding boxes predicted by the localization branch in step 4) to extract the RoI features of the person from the aforementioned spatiotemporal feature map. However, because the person is in motion or the camera is moving horizontally, the person's spatial position changes in different frames. When using the predicted bounding boxes of keyframes to extract features, it may not be possible to obtain some features of the person that deviate from the predicted bounding box range. The bounding box range could be directly expanded to cover the complete features of the person, but this would introduce background or other interference information, thus affecting the feature representation of the person itself. However, after the feature extraction stage in step 3) of this invention, the interference information in the feature map has been removed. At this point, we slightly expand the bounding box range before performing the RoIAlign operation to introduce the human body features that have deviated due to motion.

[0101] After obtaining the RoI features of a person, the final action prediction can be performed directly through the classification layer. However, with the emergence of multi-person action detection datasets in complex scenes (such as AVA, Multisports, etc.), actions may arise from interactions with other people or objects in the scene, such as "talk to someone" or "work on a computer." Many methods no longer focus solely on the actor's features but instead explore various relationship modeling techniques to capture interactions, thereby obtaining better feature representations. To address this, this invention also designs a Context Interaction Decoder to model the scene between the person's features and the compact contextual information from the encoder. This invention concatenates the RoI features of N people with M spatiotemporal tokens and feeds them into a stacked 6-layer decoder network, with each layer consisting of MHSA and FFN, consistent with the encoder. The N RoI features are extracted from the output and processed by an MLP for final action classification. The EVAD decoder is simple to implement and, thanks to the compact contextual representation obtained during the feature extraction stage, can achieve better detection results than some carefully designed complex action branches.

[0102] When predicting the action category of a person, step 5) restores the spatiotemporal location of the compact video features output in step 3), and uses an expanded version of the bounding box predicted in step 4) to extract the RoI features of the person. Furthermore, a simple decoder is used to model the relationship between the RoI features and the compact context, and the updated RoI features are used for the final action classification.

[0103] A specific implementation of step 5) is as follows.

[0104] 5.1) Recovering the spatiotemporal structure of the feature map:

[0105]

[0106]

[0107] in This represents the discrete sequence that the network finally outputs in step 3). This indicates that the remaining positions are filled with 0. The tokens retained in step 3) are then subjected to channel dimensionality reduction to obtain the discrete feature map x′. L Then, according to their corresponding spatiotemporal positions, they are placed into an empty feature map to obtain a continuous spatiotemporal feature map X;

[0108] 5.2) Expand the bounding box to cover the complete feature trace of the person:

[0109] prop_bbox = Extend(prop_bbox) L (extend_scale)

[0110]

[0111] Where prop_bbox is the expanded bounding box, roi_feat represents the RoI feature of the person, and prop_bbox L For step 4), the bounding box of the branch prediction is located. Extend_scale represents the expansion coefficient. In this invention, extend_scale = (0.4, 0.2) is preferred. At this time, the model performance is optimal, which means that the width dimension is expanded outward by 0.4 times and the height dimension is expanded outward by 0.2 times.

[0112] 5.3) A 6-layer decoder is used to perform relationship modeling between the extended RoI and the compact context, outputting the predicted action:

[0113]

[0114] Where `num_classes` represents the number of action categories in the dataset. Here, the dataset is a predefined dataset of action categories; the dataset on which the test is performed indicates the number of categories included in that dataset. For each layer of the decoder, let the input be `y`. l-1 The output is y l Then we have:

[0115] y′ l =MHSA(LN(y l-1 ))+y l-1

[0116] y l =FFN(LN(y′) l ))+y′ l

[0117] The network structure steps are the same as those of the encoder in step 2), where y l =[roi_feat l ;x l ], for the updated roi_feat l Perform action classification prediction to obtain the action score for each layer. l As intermediate layer supervision signals during the training process:

[0118]

[0119] Step 5) The human action classification branch appropriately expands the bounding boxes predicted in step 4) to include the complete spatiotemporal features of the human figure. Unlike conventional methods that directly extract RoI features using bounding boxes, this invention removes invalid information such as background in step 3), thus allowing for appropriate expansion of the bounding boxes to incorporate human features that have deviated due to motion without introducing interfering information. Similarly, thanks to the compact contextual information extracted in step 3), this invention uses a simple decoder structure with fewer parameters for contextual modeling, achieving better detection results than some more complex action classification models.

[0120] 6) Training Phase: For model initialization, the base network in step 2) is initialized using the Kinetics pre-trained and fine-tuned weights provided by VideoMAE. The token selection module added in step 3) has no new parameters. The new layers in the character localization branch in step 4) and the action classification branch in step 5) are initialized using Xavier. Following the training method in the original WOO paper, for the character localization branch, the ensemble prediction loss is used to achieve the optimal binary matching between the prediction and the true value. The ensemble prediction loss L in step 4) set Includes: L1 loss of the bounding box L L1 GIoU loss of bounding box GIoU And confidence loss function L conf Supervision is performed using cross-entropy loss. For the action classification branch, only the predicted values ​​that successfully match the true values ​​in the ensemble prediction are calculated. The action classification loss L in step 5) is... act Supervision is performed using binary cross-entropy loss, where each character may have multiple actions. The intermediate layer output is supervised for both branches. The weights of each loss function are λ. L1 =5,λ GIoU =2,λ conf =2,λ act =4. The overall loss is optimized using the AdamW optimizer, and the network parameters are updated through backpropagation. This process is repeated until the required number of iterations is reached. The specific calculation process of the training loss function is as follows:

[0121] Lset =λ L1 L L1 +λ GIoU L GIoU +λ conf L conf

[0122] L = L set +λ act L act

[0123] 7) Testing phase: Given an input video segment, directly predict N candidate boxes and their corresponding person detection scores and action classification scores. In the example, N=100. No additional post-processing is required. For example, if it is not maximum suppression, retain the detection confidence scores that are greater than 0.7 as the final results.

[0124] This invention achieves high accuracy on three action detection benchmark datasets (AVA, UCF101-24, JHMDB), and is implemented using the Python3 programming language and the PyTorch 1.7.1 deep learning framework. Figure 2 The following is a schematic diagram of a specific implementation system framework of the present invention, and the specific implementation is as follows.

[0125] 1) Generating Input Examples: The AVA dataset is a sparse label dataset, labeled at 1 FPS. For AVA, we take the frame with label information as the center, extract the temporal context of 64 frames before and after it, and then uniformly sample 16 frames at a stride of 4 as the model input. UCF101-24 and JHMDB are densely labeled datasets, labeled at 30 FPS. On the training set, we use every frame with an action instance as an example; on the test set, all frames are used as input examples, with temporal sampling the same as AVA. During training, for each RGB image frame, random scaling is performed, i.e., the short side of the image is set to 256–320 pixels, and the long side does not exceed 1333 pixels. Then, random horizontal flipping and color jitter data augmentation are performed on the image frames. The resulting image sequence is normalized by subtracting the mean of the three channels of the ImageNet dataset and dividing by the standard deviation of the three channels. Finally, it is converted into a Tensor, processed in batches, and the data loading order is shuffled. During the testing phase, for each frame of the image, the shorter side was scaled to 256 pixels without any additional data augmentation.

[0126] 2) In the configuration phase of the feature extraction network, the Vision Transformer is used as the basic network structure. The network parameters are initialized using the pre-trained and fine-tuned weights provided by VideoMAE's Kinetics, such as... Figure 3As shown, when keeprateρ is 1, it corresponds to the original Transformer network structure. Feature extraction is performed on the input sequence generated in step 1). The network input size is T*H*W*3. After cube embedding and an L-layer encoder network, the output feature map is (T / 2*H / 16*W / 16)*D. For the ViT-Base network, L=12, D=768; for the ViT-Large network, L=24, D=1024. The keyframe feature maps output from stages 1-4 are fed into the character localization branch, and the spatiotemporal feature map output from stage 4 is fed into the action classification branch to obtain the spatiotemporal action detection results.

[0127] 3) Keyframe-centric token filtering progressively eliminates spatiotemporally redundant tokens from non-keyframes during the feature extraction stage, such as... Figure 4 As shown, a token selection strategy is implemented when the keep rate is less than 1, with an encoder layer containing a token selection module following each stage 1-3. When extracting features from the input sequence generated in step 1), the number of tokens involved in the computation is gradually reduced. The network input size is T*H*W*3, and the output feature map is (T / 2*H / 16*W / 16*p^3)*D. When the keep rate is 0.7, the number of tokens is significantly reduced, and the total number of GFLOPs of the model also decreases accordingly.

[0128] The specific token selection algorithm is as follows: Figure 4 As shown, the input token sequence (containing N) t The tokens are divided into keyframe sequences (containing N1 tokens) and non-keyframe sequences, and all keyframe tokens are retained. For non-keyframe tokens, a pre-computed self-attention matrix is ​​used to represent the importance of each token, and a larger weight w is applied to all keyframe queries to obtain a keyframe-enhanced attention matrix, which better retains those tokens that are more relevant to the keyframes. Next, the importance score of the tokens is calculated, and the top N tokens are retained in descending order based on this score. t ×p-N1 tokens. Finally, the retained non-keyframe tokens are concatenated with the keyframe tokens to form the retained tokens, which are then passed on. For example... Figure 5 As shown, the tokens retained during each selection process are visualized, demonstrating that the model of this invention can effectively retain important information such as people and chairs.

[0129] 4) Based on the keyframe person localization branch of the query, predict the bounding box of the person and the corresponding confidence score, such as... Figure 6As shown, firstly, the intermediate layer feature maps of the keyframes in step 3) are extracted, and upsampling / downsampling is used to obtain a multi-scale feature map set, which is then fed into the feature pyramid network for dimensionality reduction and feature fusion. Next, a learnable candidate feature is initialized, and a query-based method is used to predict the bounding boxes of people on the keyframes and their corresponding confidence scores, combining the multi-scale feature maps. Specifically, the query-based method consists of 6 identical modules, each of which sequentially executes a multi-head self-attention layer, a dynamic instance interaction layer, a feedforward network, and two regression layers.

[0130] 5) Based on the action classification branch of extended RoI and compact context, predict multiple actions that a character may perform on a keyframe, such as... Figure 7 As shown, firstly, the M discrete spatiotemporal features output in step 3) are extracted, and their spatial structure is restored to obtain a continuous feature map of the form T / 2*H / 16*W / 16*D / 2. Then, using the expanded bounding box generated by the localization branch in step 4), RoIAlign is performed on this feature map to obtain N person RoI features. For example... Figure 8 As shown, due to the large range of motion, the bounding box (solid line) from the keyframe may not be able to cover the person swimming. This invention covers the complete motion trajectory of the person by appropriately expanding the bounding box (dashed line). Meanwhile, step 3) removes redundant tokens without introducing additional interference information. Next, the discrete features are concatenated with the RoI features and fed into a 6-layer action classification branch for relationship modeling, obtaining the person's action category score. Each layer consists of a multi-head self-attention layer, a feedforward network layer, and an action classification layer. Introducing additional relationship modeling can improve the model's AP by 4.1, demonstrating the necessity of further modeling the context.

[0131] 6) During the training phase, ensemble prediction loss is used as the loss function for the person localization branch. This includes: using L1 loss and GIoU loss to supervise the bounding boxes of people, and using cross-entropy loss to supervise the confidence of the bounding boxes, indicating whether a person is included. Binary cross-entropy loss is used as the loss function for the action classification branch. Each person may perform multiple actions, and only the predicted values ​​that successfully match the ground truth in the ensemble prediction are calculated. During training, ground truth labels are used to supervise both branches. The four loss functions are weighted and summed in a 5:2:2:4 ratio. The overall loss is optimized using the AdamW optimizer with a weight decay of 1e-4. The initial learning rate is 2.5e-5, and the total training epochs are 12. The learning rate is reduced by a decay factor of 0.1 in epochs 5 and 8. Training is completed on 8 A100 GPUs, with each mini-batch consisting of 16 video clips and two samples per card.

[0132] 7) In the testing phase, given an input video segment, directly predict 100 candidate boxes and their corresponding person detection scores and action classification scores, without requiring additional post-processing operations (such as non-maximum suppression). Detection confidence scores above 0.7 are retained as the final results. In the evaluation phase, the model's performance is evaluated using the frame-mAP@IoU0.5 (mAP) metric. Throughput is measured using a single A100 GPU with a batch size of 8, and GFLOPs are calculated using fvcore. On the AVA dataset, mAP reached 32.2 and 39.1 with ViT-Base and ViT-Large network structures, respectively. Taking ViT-Base as an example, after using the token selection algorithm, the model's performance remained unchanged, but the total GFLOPs decreased by 43%, and real-time throughput increased by 40%. Furthermore, with comparable computational cost, using tokens from higher resolutions improved detection performance by 1%. On the UCF101-24 and JHMDB datasets, EVAD with a token retention rate of 60% achieved mAP of 85.1 and 90.2, respectively. The EVAD proposed in this invention achieves state-of-the-art detection performance on all three datasets mentioned above.

Claims

1. A method for video motion detection based on key frame filtering pixel blocks, characterized in that An accelerated video action detector, EVAD, is constructed to detect actions on input video segments. The implementation of EVAD includes an input sample generation stage, a network configuration stage, a training stage, and a testing stage, as detailed below: 1) Generate input samples: Taking the frame with label information as the center, i.e. the key frame, extract the temporal context of frame a forward and backward to form the input frame sequence of 2a frames, and then uniformly sample frame b as the input video sequence of the detector. 2) Extracting video sequence features: using video ViT as a feature extraction network, the video sequence generated in step 1) is subjected to feature extraction, and a token sequence representing video features is obtained, with the time resolution remaining unchanged and the spatial resolution being down-sampled by 16 times ; 3) Keyframe-centric token selection: A keyframe-centric token selection mechanism is introduced in some MHSA and FFN layers of ViT to reduce the number of non-keyframe tokens involved in the computation and reduce spatiotemporal redundancy within the input. First, valid tokens are defined, including all tokens from keyframes and tokens with high importance scores from non-keyframes. The importance score is characterized using an attention map. A set proportion of important tokens are retained for each non-keyframe selection. Token selection is performed every quarter of the total number of layers in the ViT encoder, discarding redundant tokens and retaining valid tokens to be sent to the subsequent FFN at the current position. This process is repeated three times. The specific token selection process is as follows: 3.1) Use a pre-computed self-attention matrix to measure the importance of each token: wherein is a matrix, represents the importance degree that query i thinks pixel block j has, R is the number of input tokens, D represents the channel dimension of the query, and Head represents the number of multiple heads. 3.2) Weighted average of the query over the importance scores of each token, the importance of pixel block j is represented as: wherein respectively denote the number of all tokens, keyframe tokens and non-keyframe tokens, assuming that the first tokens belong to the keyframe, and the weight is a hyperparameter; 3.3) Based on the importance score , the token selection is made: in This represents the input tokens. This indicates the tokens that are retained after selection. Indicates the token retention rate. according to descending order from Select the previous one from the non-keyframe tokens. tokens, and All keyframe tokens are returned together as input for subsequent network operations; 4) Query-based Keyframe Person Localization: The intermediate layer feature maps of the ViT encoder (which underwent token filtering in step 3) are extracted at intervals equal to 1 / 4 of the total number of layers. These are then upsampled / downsampled to form a multi-scale feature map set for the keyframes. This set is fed into the FPN for feature fusion. Next, the person localization branch uses a query-based method to predict N bounding box coordinates on the multi-scale feature map set of the keyframes. and the corresponding confidence score , Indicates the probability that the box contains a person; 5) Human action classification based on extended RoI and compact spatiotemporal context: Action category prediction is performed by the action classification branch, and an empty feature map is initialized. The M tokens retained from step 3) are then subjected to a linear layer for dimensionality reduction, and then placed according to their corresponding spatiotemporal positions. In the middle, the remaining positions are filled with 0 to form a spatiotemporal feature map. Then, the bounding boxes predicted in step 4) are expanded to a certain extent, and RoIAlign is performed on this spatiotemporal feature map to obtain the RoI features of N people. Subsequently, a context interaction decoder is constructed to model the scene between the person features and the context information from the ViT encoder. The RoI features of the N people are concatenated with M spatiotemporal tokens and fed into a stacked 6-layer decoder network, each layer consisting of MHSA and FFN, consistent with the ViT encoder. The N RoI features in the output are extracted and processed by MLP for final action classification. The final action category prediction result is obtained; the action classification result of the candidate objects generated by the action classification branch is as follows: 5.1) Recover the spatio-temporal structure of the feature map, put the tokens reserved in step 3) into an empty feature map according to the corresponding spatio-temporal position after channel dimension reduction, and obtain a continuous spatio-temporal feature map ;​ 5.2) Expand the bounding box to cover the complete feature trace of the person: in To locate the bounding box of the branch prediction in step 4), Indicates the expansion coefficient, set =(0.4, 0.2), indicating that the W dimension expands outward by a factor of 0.4 and the H dimension expands outward by a factor of 0.2; For the extended bounding box, Represents the RoI characteristics of a person. This represents the space max pooling operation. Indicates the RoIAlign operation; 5.3) Using a 6-layer decoder network in Perform relationship modeling between compact context and compact action, output predicted action: wherein represents the number of action classes of the dataset, for each layer of the decoder, let the input be , the output be then we have: wherein , is the output of each layer of the ViT encoder, and perform action classification prediction to obtain action scores for each layer as an intermediate layer supervision signal for the training process: ； 6) Training Phase: Initialize ViT from step 2) using pre-trained and fine-tuned weights provided by VideoMAE's Kinetics. Initialize the remaining new layers using Xavier. The character localization branch utilizes ensemble prediction loss to achieve optimal binary matching between predictions and ground truth. The ensemble loss function... Includes: L1 loss of bounding box GIoU loss of bounding box and confidence loss The action classification branch is composed of action classification loss. This means that only the predicted values ​​that successfully match the true values ​​during the calculation of the set prediction loss are calculated. The four loss functions are weighted according to the set ratio and optimized by the AdamW optimizer. This process is repeated until the number of iterations is reached. 7) Testing phase: Given an input video segment, proceed through steps 1) to 5) to obtain the motion detection results on keyframes, and verify the motion detection performance of the constructed detector; Step 1) corresponds to the input sample generation stage, steps 2) to 5) correspond to the network configuration stage, step 6) corresponds to the training stage, and step 7) corresponds to the testing stage.

2. The method of claim 1, wherein the method further comprises: For the input sample in step 1, during the training phase, for each frame of the input image, random scaling is performed, setting the range of the short side of the image to 256-320 pixels and the long side not exceeding 1333 pixels, and data augmentation is performed on the image frame, including random horizontal flipping and color jitter; during the testing phase, for each frame of the input image, the short side is scaled to 256 pixels, and no additional data augmentation is performed.

3. The method of claim 1, wherein the method further comprises: Step 4) The character localization branch generates the character bounding box and confidence score on the keyframes as follows: 4.1) Extract the intermediate layer features of keyframes from the ViT encoder designed with token filtering in step 3), and obtain multi-scale features using nearest neighbor interpolation: 4.2) Input multi-scale features into FPN for dimensionality reduction and feature fusion; 4.3) Set up N learnable candidate boxes and corresponding candidate features. The human localization branch has a total of 6 layers. Each layer consists of a self-attention layer that interacts between candidate features, a dynamic instance interaction layer that interacts between candidate features and corresponding RoI features, and an FFN layer. The output of each layer is then passed through a regression layer and a classification layer to obtain the corrected candidate boxes and corresponding confidence scores. The candidate boxes and corresponding confidence scores obtained by the last layer are used as the final result of human detection.

4. An electronic device, characterized by The device includes a storage medium and a processor. The storage medium is used to store a computer program, and the processor is used to execute the computer program. When the computer program is executed, it implements the video motion detection method based on keyframe filtering of pixel blocks as described in any one of claims 1-3, and obtains an accelerated video motion detector (EVAD) for motion detection of input video segments.

5. A computer readable storage medium, characterized by The computer-readable storage medium stores a computer program, which, when executed, implements the video motion detection method based on keyframe-based pixel block selection as described in any one of claims 1-3, to obtain the accelerated video motion detector EVAD.

Images