A real-time video action detection space-time optimization method for embedded platforms

Abstract

The application relates to a real-time video action detection space-time optimization method for an embedded platform, which comprises the following steps: step 1, obtaining a video segment containing only a high-information-density action subject; step 2, obtaining an action feature with fine-grained dynamic perception capability; step 3, obtaining a final action feature with high category distinguishability and strong spatial consistency; and step 4, obtaining a final video action detection result. The application has the beneficial effect that a multi-level collaborative efficient space-time modeling framework for real-time video action detection of an embedded platform is constructed, aiming at solving the essential conflict between limited computing power on the edge side and high redundancy of video data, and realizing the balance between detection accuracy and reasoning speed.

Description

Technical Field

[0001] This invention relates to the fields of deep learning, artificial intelligence, and action detection technology, and specifically to a spatiotemporal optimization method for real-time video action detection for embedded platforms. Background Technology

[0002] With the deep integration of next-generation artificial intelligence technology with 5G mobile communication technology and the Internet of Things (IoT) industry, video, as the data modality with the largest information carrying capacity and the most intuitive transmission, has become the main carrier for interaction between the physical and digital worlds. Faced with massive amounts of video data, traditional processing methods relying on manual monitoring or simple motion detection can no longer meet real-world needs. Computer vision technology is undergoing a paradigm shift from centralized cloud processing to real-time intelligent analysis at the edge. Against this backdrop, video action detection, as a cornerstone task in the field of video understanding, aims not only to identify behavior categories but also to accurately locate the spatiotemporal boundaries of actions occurring in unrestricted video streams. Thanks to breakthroughs in feature extraction using deep neural networks, deep learning-based action detection methods have completely replaced traditional manual feature extraction methods, demonstrating stronger generalization capabilities and showing significant application value in fields such as intelligent security, autonomous driving, smart healthcare, and human-computer interaction.

[0003] Based on different methods of modeling spatiotemporal information, existing research on action feature extraction can be broadly categorized into three types: convolutional neural networks based on a two-stream architecture, 3D convolutional neural networks, and self-attention models based on Transformers. The early two-stream architecture proposed by Simonyan et al. (2014) decoupled appearance and motion information representation through independent 2D CNN branches. Subsequently, the Temporal Segmentation Network (TSN) proposed by Wang et al. (2016) introduced a sparse sampling mechanism to expand the long temporal receptive field. The Temporal Shift Module (TSM) proposed by Lin et al. (2019) endowed 2D CNNs with the ability to process temporal information without increasing computational power through channel shifting operations, becoming an important foundation for efficient action detection. To achieve end-to-end feature extraction, 3D convolutional networks such as C3D proposed by Tran et al. (2015) and I3D proposed by Carreira et al. (2017) directly learn spatiotemporal features from the original video through dilated convolutional kernels. Although these algorithms improve modeling accuracy, the large number of parameters and computational burden make them difficult to run on edge devices. To address this issue, decomposition convolution strategies such as P3D and R(2+1)D alleviate computational demands by splitting the 3D kernel into spatial and temporal convolutions. Meanwhile, the SlowFast network, inspired by biological vision, further optimizes spatiotemporal representation efficiency through complementary dual-path rates. In recent years, inspired by the Vision Transformer, architectures such as TimeSformer and ViViT have achieved powerful global modeling capabilities using self-attention mechanisms. However, they often lack the inductive bias of convolutions, and their computational complexity during inference places extremely high demands on hardware computing power.

[0004] Despite the record-breaking performance of these solutions on public datasets, they still face significant challenges in practical engineering applications, especially when deploying such algorithms on embedded devices with limited computing power or on domestic AI chips like Ascend. First, action detection models generally suffer from a contradiction between high computational load and severe spatiotemporal redundancy. To capture complex dynamic changes, modern networks typically employ deep stacked structures, but video data naturally exhibits significant redundancy in the spatiotemporal dimension. Background areas often occupy most of the frame and change slowly, while key actions exist only in sparse spatiotemporal segments. Most existing algorithms lack efficient dynamic allocation strategies for spatiotemporal resources, causing models to waste valuable edge computing power when processing large amounts of irrelevant background or still frames, making it difficult to meet real-time requirements. Second, there is a significant imbalance between the stringent resource constraints of embedded platforms and model complexity. Unlike cloud servers, edge AI chips have only a fraction of the memory bandwidth, on-chip cache, and peak computing power of desktop GPUs. Large models often lead to problems such as memory overflow, high inference latency, and excessive power consumption. In addition, existing lightweight solutions such as the MobileNet series, neural architecture search, and channel pruning model compression techniques, while reducing computational load to some extent, often come at the cost of sacrificing the accuracy of fine-grained action recognition. Furthermore, many strategies fail to fully integrate with the underlying hardware characteristics for deep collaborative optimization, resulting in the theoretical reduction in computational load not being effectively translated into an increase in actual inference speed.

[0005] Video action detection in real-world scenarios typically exhibits significant spatiotemporal unevenness. Current mainstream networks, which employ indiscriminate, dense feature extraction, over-consume computational resources in low-information-density background regions. This is particularly problematic when dealing with complex poses or subtle movements, where lightweight networks often struggle to provide sufficiently consistent discriminative information. Therefore, designing a collaborative optimization architecture that adaptively eliminates spatiotemporal redundancy, enhances local dynamic perception, and is suitable for hardware acceleration, while maintaining detection accuracy, has become a critical challenge for both academia and industry. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a spatiotemporal optimization method for real-time video action detection for embedded platforms, which aims to overcome the shortcomings of existing action detection models on embedded platforms, such as high computing power consumption, poor real-time performance, and insufficient accuracy in fine-grained action recognition.

[0007] The technical solution of this invention to solve the above-mentioned technical problems is as follows: A spatiotemporal optimization method for real-time video motion detection for embedded platforms, comprising the following steps:

[0008] Step 1: Through the differentiable processing mechanism of the temporal and spatial policy networks, the original video stream is processed to filter key frames in the temporal dimension and crop key regions in the spatial dimension, resulting in video segments containing only high-information-density action subjects.

[0009] Step 2: Based on the high information density action subject video segment obtained in Step 1, it is fed into the backbone network of the integrated local dynamic enhancement module. Deep feature extraction is completed through joint modeling of amplitude and phase in the complex domain to obtain action features with fine-grained dynamic perception capabilities.

[0010] Step 3: Based on the deep action features obtained in Step 2, a representation enhancement mechanism of prototype-guided contrastive learning and spatial consistency structure constraint of human key points is introduced in the model training stage to complete the supervised optimization of features and obtain the final action features with high class discrimination and strong spatial consistency.

[0011] Step 4: Based on the action features with high class discrimination and strong spatial consistency obtained in Step 3, the action features are sent to the detection head to complete the recognition of action categories and the localization of spatiotemporal boundaries, and the final video action detection results are obtained.

[0012] Based on the above technical solution, the present invention can be further improved as follows:

[0013] Furthermore, step 1 specifically involves:

[0014] Step 1.1, using lightweight global features as input Constructing a learnable importance distribution from frame video sequences The Monte Carlo sampling mechanism is used to select the set of keyframes with the strongest action discrimination based on the joint probability distribution. The formula for the joint probability distribution is: To address the non-differentiability problem of discrete sampling, a Monte Carlo approximation loss is introduced during the training iteration. ,in It is the result of weighted normalized aggregation of the category scores of each frame within the sampling set, where u is a uniform distribution;

[0015] Step 1.2: Keyframe feature maps selected based on the temporal strategy network. Predict the center coordinates of potential action regions using a spatial policy network. With scale parameters Construct a differentiable affine matrix :

[0016]

[0017] Furthermore, step 2 specifically involves:

[0018] Step 2.1: Project the input key region features into complementary subspaces to represent the magnitude terms of the context intensity. Phase terms that reflect dynamic trends ;

[0019] Step 2.2: Extend the features to the complex domain using Euler's formula to achieve phase-differentiated correlation, and finally output the features. :

[0020]

[0021] Furthermore, step 3 specifically involves:

[0022] Step 3.1: Introduce class prototype vectors into the feature space. Construct a set of hard negatives consisting of k groups of similar error categories. Calculate dynamic geometric calibration terms By using contrastive loss to force the model to increase the discriminative distance between action categories, the objective formula for contrastive loss is:

[0023] ;

[0024] Where sim(·) is the feature similarity calculation function, For temperature coefficient, Batch size;

[0025] Step 3.2: By predicting the two-dimensional coordinates of human body key points, an object key point similarity metric is introduced, and a key point loss function is constructed:

[0026]

[0027] in, For the predicted first Coordinates of key points For the actual key point coordinates, The total number of key points. The scale factor for keypoints.

[0028] Furthermore, step 4 specifically involves:

[0029] Step 4.1: Input the motion features obtained in step 3 into the detection module to perform category recognition and spatiotemporal location prediction on the motion targets in the video clip, and obtain the initial motion detection results;

[0030] Step 4.2: Redundancy suppression processing is performed on the initial action detection results to remove overlapping detection results and obtain the final action detection results;

[0031] Step 4.3: Format the final motion detection results according to the embedded platform deployment interface to support real-time video motion detection applications in embedded systems.

[0032] The beneficial effects of this invention are as follows: This invention constructs a multi-level collaborative and efficient spatiotemporal modeling framework for real-time video action detection on embedded platforms, aiming to resolve the fundamental conflict between limited computing power at the edge and high redundancy of video data, and achieve a balance between detection accuracy and inference speed. Its innovation lies in proposing a full-link optimization strategy from data source to feature representation. First, the input video is reconstructed through a globally semantically guided spatiotemporal strategy network, using differentiable sampling and cropping mechanisms to adaptively remove redundant frames and irrelevant backgrounds, allowing limited computing resources to be precisely focused on the high-information-density action subject region. Second, a local dynamic enhancement module is designed in the feature extraction stage, using joint modeling of amplitude and phase in the complex domain to explicitly capture micro-motion patterns and fine-grained cross-frame correlations that are difficult to detect with lightweight convolution. Finally, prototype-guided contrastive learning and spatial structure constraints of human keypoints are introduced in the training stage, enhancing the class discriminativeness and spatial consistency of high-level features without incurring any additional inference overhead. Attached Figure Description

[0033] Figure 1 This is a diagram of the overall network structure of the present invention;

[0034] Figure 2 Performance comparison for embedded deployment. Detailed Implementation

[0035] The principles and features of the present invention are described below with reference to the accompanying drawings. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0036] To address the bottleneck issues of uneven computational resource allocation and severe spatiotemporal redundancy in real-time video motion detection on embedded platforms, this invention proposes a multi-level collaborative high-efficiency spatiotemporal optimization network for motion detection on embedded platforms. The overall network framework is as follows: Figure 1 As shown, this framework follows a hierarchical collaborative design concept of global perception guiding local fine-grained modeling, systematically optimizing the temporal and spatial modeling process from three dimensions: input, feature, and supervision. The overall processing flow adopts a serial collaborative logical paradigm. First, in the input stage, the original video stream is reconstructed through a spatiotemporal policy network to actively remove invalid spatiotemporal information. Then, the focused high-information-density segments are fed into a backbone network integrating a local dynamic enhancement module for deep feature extraction. Finally, in the training stage, the discriminative power of the features is enhanced by introducing contrastive learning and structural constraint mechanisms. This invention includes the following steps:

[0037] Step 1: Using the differentiable processing mechanism of the temporal and spatial policy networks, the original video stream is processed to filter keyframes in the temporal dimension and crop key regions in the spatial dimension, resulting in video segments containing only high-information-density action subjects. Specifically:

[0038] Step 1.1: In the spatiotemporal redundancy optimization stage at the input end, this invention constructs a perceptual processing mechanism based on global semantic awareness and differentiability. In the temporal dimension, the temporal policy network utilizes lightweight global features to construct a learnable importance distribution. Through the Monte Carlo sampling mechanism, from the input Selecting the set of keyframes with the most motion-determining capabilities from a video sequence The sampling process follows a joint probability distribution:

[0039]

[0040] To address the issue of non-differentiability of discrete sampling, this invention introduces a Monte Carlo approximation loss during training iterations, as shown below:

[0041]

[0042] in, This is the result of weighted normalized aggregation of the class scores of each frame within the sampling set. u is uniformly distributed, and the regularization term stabilizes the policy network's focus on key action segments at different sampling instances and iteration stages, suppressing sampling drift caused by local noise. During temporal policy training, the probability distribution w is adjusted to influence the occurrence probability of each sampling set, allowing the gradient to be propagated back to the policy network step-by-step along the differentiable link from the normalized mapping layer to the class score to the feature representation, enabling it to learn to assign greater weights to high-discriminative frames. In the inference phase, the policy network no longer uses random sampling but directly selects the Top-K frames with the highest probability from the distribution. These key frames will be further cropped into key regions in the spatial policy network.

[0043] Step 1.2: In the spatial dimension, the spatial policy network processes the selected keyframe feature maps. Prediction is performed to obtain the center coordinates of the potential action area. With scale parameters To achieve end-to-end positioning optimization, this invention employs a differentiable affine transformation to construct the affine matrix shown below. :

[0044] ;

[0045] This matrix maps the standardized sampling grid back to the input feature map coordinate system, and uses bilinear interpolation to achieve precise cropping of the main action region, so that the subsequent backbone network only needs to process local image regions with compact structure and concentrated semantics.

[0046] Step 2: Based on the high information density video clips of the main action subject obtained in Step 1, they are fed into the backbone network integrating the local dynamics enhancement module. Deep feature extraction is completed through joint modeling of amplitude and phase in the complex domain, resulting in action features with fine-grained dynamic perception capabilities. Specifically:

[0047] Step 2.1: In the feature extraction stage, to address the limitation of lightweight convolution in capturing fine-grained dynamic changes, this invention designs a local dynamic enhancement module in the backbone network. The core idea of ​​this module is to extend local neighborhood features from traditional static mapping to complex-domain dynamic dependency encoding. The module first projects the input features into complementary subspaces, representing the magnitude terms of the context intensity. Phase terms that reflect dynamic trends .

[0048] Step 2.2: Based on this, the features are extended to the complex domain using Euler's formula to achieve phase-differentiated correlation, and the final features are output. Represented as:

[0049] ;

[0050] In the above computational paradigm, and By constructing a dynamic attention map for local neighborhoods, the convolution kernel weights can be adaptively adjusted according to the phase changes of the input signal, thereby accurately capturing key region features with specific motion trends.

[0051] Step 3: Based on the deep action features obtained in Step 2, a representation enhancement mechanism combining prototype-guided contrastive learning and spatial consistency structural constraints of human keypoints is introduced during the model training phase. This completes supervised optimization of the features, resulting in final action features with high class discriminative power and strong spatial consistency. Specifically:

[0052] Step 3.1: To further enhance the discriminative power of high-level features without increasing inference costs, this invention introduces a representation enhancement mechanism at the supervision end. First, a prototype-guided contrastive learning strategy is used to introduce class prototype vectors into the feature space. To address the common problem of similar category confusion in action detection, this invention proposes a novel strategy combining hard negative sample mining and dynamic geometric calibration. This strategy involves constructing a hard negative set consisting of k groups of similar error categories. And calculate dynamic geometric calibration terms. The model is forced to widen the discriminative distance between action categories during the optimization process, and its final comparative loss objective is shown below:

[0053] ;

[0054] Where sim(·) is the feature similarity calculation function, For temperature coefficient, This refers to the batch size.

[0055] Step 3.2: Simultaneously, this invention further introduces a spatial consistency structural constraint based on key points. By predicting the two-dimensional coordinates of human body key points and introducing a similarity metric for object key points, the constructed loss function is as follows:

[0056]

[0057] in, For the predicted first Coordinates of key points For the actual key point coordinates, The total number of key points. The scale factor for keypoints.

[0058] This constraint uses the geometric prior of human pose as an auxiliary supervision signal to guide the network to focus on the relative layout relationships of human joints, so that the model can maintain stable representation output through semantic anchors when facing occlusion or complex backgrounds.

[0059] The aforementioned multi-level collaborative mechanisms together constitute a complete optimization chain from data screening and feature enhancement to semantic constraints, ensuring that the model balances real-time performance and detection accuracy under embedded and resource-constrained conditions.

[0060] Step 4: Based on the action features with high class discriminative power and strong spatial consistency obtained in Step 3, these features are fed into the detection head to complete action class recognition and spatiotemporal boundary localization, resulting in the final video action detection result. Details are as follows:

[0061] Step 4.1: Input the motion features obtained in step 3 into the detection module to perform category recognition and spatiotemporal location prediction on the motion targets in the video clip, and obtain the initial motion detection results;

[0062] Step 4.2: Redundancy suppression processing is performed on the initial action detection results to remove overlapping detection results and obtain the final action detection results;

[0063] Step 4.3: Format the final motion detection results according to the embedded platform deployment interface to support real-time video motion detection applications in embedded systems.

[0064] Experimental Analysis

[0065] The network model proposed in this invention is trained and tested primarily using the publicly available action detection datasets UCF24 and JHMDB, as well as a private behavior dataset constructed for real-world scenarios. The UCF24 dataset contains 24 everyday behavior categories, covering various behavior types such as sports, hand movements, and posture changes. It features complex backgrounds and large action ranges, serving as a benchmark for evaluating the model's spatiotemporal modeling capabilities. The private dataset, derived from actual monitoring scenarios, contains four typical behavior categories and exhibits stronger real-world noise and viewpoint changes, placing higher demands on the robustness of the lightweight model. In the experiments, this invention uniformly sets the input frame count to 16 frames, using frame-level average precision (Frame-mAP) and floating-point operations (GFLOPs) as the core evaluation metrics.

[0066] To verify the effectiveness of the proposed multi-level collaborative framework in improving detection accuracy, a detailed comparative experiment was first conducted on the UCF24 dataset. The experimental results are shown in Table 1. The experimental data shows that the proposed method achieves 85.22% F-mAP, achieving the best performance among all compared methods. Under the premise of comparable computational cost, compared to the most representative high-efficiency frameworks YOWOv2-T and YOWOv3-TSM, the accuracy of the proposed method is improved by 5.13% and 5.67%, respectively. Notably, even when using the lightweight ShuffleNetv2 as the backbone network, the performance of the proposed method surpasses that of the YOWO* model using the ResNext-101 backbone and the computationally intensive 3D network TubeR. This strongly demonstrates that through a globally semantically guided spatiotemporal filtering mechanism, the model can concentrate limited computational power on the most informative regions, thereby achieving deeper action understanding with a smaller model size.

[0067] Table 1. Comparison Experiment Results of UCF24 Dataset

[0068]

[0069] To address the real-time requirements of embedded platforms, this invention conducted ablation experiments to break down the time consumption of each stage in the inference process, as shown in Table 2. Experimental results show that after introducing the spatiotemporal redundancy optimization module, the average preprocessing time per frame decreased significantly from 35ms to 12ms. This significant performance gain stems from a dual optimization mechanism: the temporal policy network effectively filters out redundant frames with low information content, while the spatial policy network precisely trims background regions irrelevant to the action. The synergistic effect of these two mechanisms greatly compresses the data input to the backbone network in the spatiotemporal dimension, thereby significantly reducing the load on subsequent computationally complex modules. This demonstrates that the perception-pruning-recognition paradigm proposed in this invention reduces the memory bandwidth and computing power bottlenecks of edge devices from the data source.

[0070] Table 2 Embedded Inference Performance Ablation Experiment

[0071]

[0072] Furthermore, to verify the contribution of each core component of the framework to the accuracy, progressive ablation experiments were conducted on a private dataset and UCF24, as shown in Tables 3 and 4. On the private dataset, after introducing spatiotemporal redundancy optimization, the mAP improved from 93.35% to 94.72%. Further introduction of a local dynamic enhancement module further improved the accuracy by 0.63%.

[0073] Table 3. Ablation Experiments of the Overall Model (Private Dataset)

[0074]

[0075] Table 4 Results of the overall model ablation experiment (UCF24)

[0076]

[0077] Finally, regarding the actual deployment experiment of the Huawei Ascend embedded platform, such as... Figure 2 As shown, this further verifies the engineering value of the present invention. The baseline YOWO model only achieves 4.8 FPS on the Ascend platform, and its size is as high as 315.8MB. In contrast, the optimized architecture proposed in this invention improves the inference speed to 10.6 FPS without relying on additional pruning and quantization, and significantly compresses the model size to 37.8MB. After adding INT8 quantization technology, the frame rate is further improved to 13.3 FPS, and the size is reduced to 8.6MB. This comparative data proves that the spatiotemporal awareness strategy proposed in this invention is the fundamental reason for solving the performance bottleneck on the edge side, and it achieves performance improvement through structured design at the algorithm level.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A spatiotemporal optimization method for real-time video motion detection for embedded platforms, characterized in that, The steps include the following: Step 1: Through the differentiable processing mechanism of the temporal and spatial policy networks, the original video stream is processed to filter key frames in the temporal dimension and crop key regions in the spatial dimension, resulting in video segments containing only high-information-density action subjects. Step 2: Based on the high information density action subject video segment obtained in Step 1, it is fed into the backbone network of the integrated local dynamic enhancement module. Deep feature extraction is completed through joint modeling of amplitude and phase in the complex domain to obtain action features with fine-grained dynamic perception capabilities. Step 3: Based on the deep action features obtained in Step 2, a representation enhancement mechanism of prototype-guided contrastive learning and spatial consistency structure constraint of human key points is introduced in the model training stage to complete the supervised optimization of features and obtain the final action features with high class discrimination and strong spatial consistency. Step 4: Based on the action features with high class discrimination and strong spatial consistency obtained in Step 3, the features are sent to the detection head to complete the recognition of action categories and the localization of spatiotemporal boundaries, and the final video action detection results are obtained.

2. The spatiotemporal optimization method for real-time video motion detection for embedded platforms according to claim 1, characterized in that, Step 1 is as follows: Step 1.1, using lightweight global features as input Constructing a learnable importance distribution from frame video sequences The Monte Carlo sampling mechanism is used to select the set of keyframes with the strongest action discrimination based on the joint probability distribution. The formula for the joint probability distribution is: To address the non-differentiability problem of discrete sampling, a Monte Carlo approximation loss is introduced during the training iteration. ,in It is the result of weighted normalized aggregation of the category scores of each frame within the sampling set, where u is a uniform distribution; Step 1.2: Keyframe feature maps selected based on the temporal strategy network. Predict the center coordinates of potential action regions using a spatial policy network. With scale parameters Construct a differentiable affine matrix :

3. The spatiotemporal optimization method for real-time video motion detection for embedded platforms according to claim 1, characterized in that, Step 2 is as follows: Step 2.1: Project the input key region features into complementary subspaces to represent the magnitude terms of the context intensity. Phase terms that reflect dynamic trends ; Step 2.2: Extend the features to the complex domain using Euler's formula to achieve phase-differentiated correlation, and finally output the features. :

4. The spatiotemporal optimization method for real-time video motion detection for embedded platforms according to claim 1, characterized in that, Step 3 specifically involves: Step 3.1: Introduce class prototype vectors into the feature space. Construct a set of hardnegative errors consisting of k groups of similar error categories. Calculate dynamic geometric calibration terms By using contrastive loss to force the model to increase the discriminative distance between action categories, the objective formula for contrastive loss is: ； Where sim(·) is the feature similarity calculation function, For temperature coefficient, Batch size; Step 3.2: By predicting the two-dimensional coordinates of human body key points, an object key point similarity metric is introduced, and a key point loss function is constructed: in, For the predicted first Coordinates of key points For the actual key point coordinates, The total number of key points. The scale factor for keypoints.

5. The spatiotemporal optimization method for real-time video motion detection for embedded platforms according to claim 1, characterized in that, Step 4 is as follows: Step 4.1: Input the motion features obtained in step 3 into the detection module to perform category recognition and spatiotemporal location prediction on the motion targets in the video clip, and obtain the initial motion detection results; Step 4.2: Redundancy suppression processing is performed on the initial action detection results to remove overlapping detection results and obtain the final action detection results; Step 4.3: Format the final motion detection results according to the embedded platform deployment interface to support real-time video motion detection applications in embedded systems.

Images