Real-time online action detection method based on mamba architecture

Abstract

The application provides a real-time online action detection technology based on a Mamba architecture, comprising: S1, training data acquisition, acquiring video streams of multiple perspectives in a target scene; S2, data preprocessing, extracting video features from the video streams based on a pre-trained two-dimensional convolutional neural network; S3, introducing a Mamba OAD framework, which is composed of an action detection module and a future prediction module; S4, constructing the action detection module, which is mapped to a feature space through a linear projection layer, and the linear projection layer converts input features to D dimensions through a learnable weight matrix; S5, constructing the future prediction module, which predicts future latent features based on short-term latent features output by the action detection module; and S6, model training and reasoning, in which related modules are jointly optimized in the training process. The mAP and mcAP indicators are used to measure the online action detection accuracy, and the FPS and GFLOPs indicators are used to measure the complexity of model inference. The application improves the accuracy and robustness of action detection.

Description

Technical Field

[0001] This invention relates to the field of deep integration motion video understanding technology, specifically to a real-time online motion detection method based on the Mamba architecture. Background Technology

[0002] With the development of computer vision and deep learning technologies, video content understanding and analysis are becoming increasingly important. In fields such as surveillance, human-computer interaction, and autonomous driving, real-time detection and recognition of actions in videos have become key technological requirements. Online action detection (OAD), as an important task in video understanding, aims to identify ongoing actions in real time using only the video's history and the current frame. Compared to offline detection, OAD faces the challenge of lacking future information, requiring models to effectively capture temporal dependencies to accurately predict action categories with limited observations.

[0003] Currently, OAD models are mainly divided into RNN-based and Transformer-based models. RNN-based models excel at capturing sequence dependencies, are suitable for handling temporal transitions in video streams, and offer high computational efficiency and low resource consumption, showing significant advantages in real-time applications. However, they suffer from the vanishing gradient problem and difficulties in modeling long-term temporal dependencies, impacting long-term context processing performance. Transformer-based models, on the other hand, utilize self-attention mechanisms to capture long-distance dependencies, effectively modeling global temporal attributes and achieving state-of-the-art performance on various benchmark datasets. However, their high computational cost limits their application in low-latency, high-efficiency scenarios. Therefore, OAD model design requires a balance between efficiency and performance.

[0004] Recently, the Mamba model has broken the deadlock of Selective Structured State Space (SSMs), demonstrating excellent performance in balancing efficiency and performance, making it an ideal choice for foundational models. Mamba shares a similar recursive mechanism with RNNs, possessing efficient temporal modeling capabilities. Furthermore, by introducing a selective state space mechanism, it can dynamically filter and propagate information based on the input context, effectively capturing long-term dependencies, avoiding the high computational overhead of Transformers, and overcoming the gradient vanishing problem of RNNs. Its hardware-aware architecture optimizes memory usage and inference speed, making it faster than Transformer models and even surpassing traditional RNNs in some scenarios. Based on these advantages, Mamba provides a powerful solution for OAD tasks, enabling the processing of video streaming data with high accuracy and low latency. Summary of the Invention

[0005] Technical issues:

[0006] Long-term dependency modeling: Existing methods are insufficient in effectively modeling long-term dependencies, and it is difficult to accurately capture action features and dependencies over long periods of time in videos.

[0007] Computational Resources and Real-Time Performance: Achieving real-time inference with limited computing resources is a challenge. It requires improving the computational efficiency of the model while ensuring detection accuracy in order to meet the real-time requirements of practical applications.

[0008] Action boundary recognition in complex scenes: The ability to recognize blurred action boundaries in complex scenes needs to be improved. The model needs to more accurately distinguish the start and end of different actions to avoid misjudgment.

[0009] To address the aforementioned technical problems, this invention proposes an online action detection method based on Mamba. Through a unique architecture design and training strategy, it balances computational efficiency and modeling accuracy, thereby improving the performance of action detection.

[0010] Technical solution:

[0011] The real-time online motion detection method based on the Mamba architecture disclosed in this invention belongs to the field of motion video understanding. The method includes the following steps:

[0012] S1, Training data acquisition: Acquire video streams from multiple perspectives in the target scene;

[0013] S2, data preprocessing, based on a pre-trained two-dimensional convolutional neural network (2D CNN), extracts video features (RGB and optical flow features) from the video stream;

[0014] S3 introduces the MambaOAD framework, which consists of two closely related and complementary modules: the Action Detection Module (ADM) and the Future Prediction Module (FAM).

[0015] S4. Construct the action detection module. ADM is the core component of MambaOAD for achieving real-time online action detection. ADM maps to the feature space through a linear projection layer. The linear projection layer transforms the input video features to D dimensions through a learnable weight matrix.

[0016] S5, the future prediction module, greatly expands the functionality of the MambaOAD framework. It mainly predicts future latent features based on the short-term latent features output by ADM.

[0017] S6, Model Training and Inference: During the training process, the MambaOAD framework jointly optimizes ADM and FAM and evaluates performance based on actual action labels.

[0018] Further, step S1 includes the following:

[0019] (1) Check the perspective of the video dataset. It is necessary to monitor or film the same action of the same sample from at least 3 different perspectives.

[0020] (2) Check the length of the video dataset. The video must be in an unedited state and contain background and several actions that last for a certain period of time.

[0021] (3) Check the labeling of the video dataset. The videos must have fine-grained action labels, and the videos must have the same label that can be used to form a comparison between different perspectives for training the deep model.

[0022] Further, step S2 includes the following:

[0023] Real-time video streams are used as input data. Every N frames of video are processed as a group, and a pre-trained convolutional neural network is used to extract the video features of the frames in this group, namely RGB image features and optical flow features. The convolutional neural network is a two-dimensional convolutional neural network (2D CNN), which is used to capture the spatial and temporal information of video frames at the same time, so as to achieve efficient recognition of dynamic changes in video content.

[0024] Further, step S3 includes the following:

[0025] ADM fully leverages the high efficiency of Mamba, capturing spatiotemporal dependencies through State Space Models (SSMs) and convolutional mechanisms to achieve real-time online action detection. State Space Models can model dynamic changes, effectively capturing the evolution of actions over time in video action detection. Convolutional mechanisms excel at extracting spatial features from images; the combination of the two comprehensively processes spatiotemporal information in videos. FAM (Focus-Based Action Model) only plays a role during the training phase, its primary responsibility being to predict future actions, providing forward-looking information to strongly support the spatiotemporal representation learning of ADM. During the inference phase, to ensure computational efficiency, only ADM is retained. This design guarantees detection accuracy while significantly improving inference speed, meeting the demands of real-time applications.

[0026] Further, step S4 includes the following:

[0027] At time step T, ADM receives the input video features. Then, a linear projection layer is used to precisely map each frame to a D-dimensional feature space, resulting in the embedded sequence. Where x t Let represent the original video features of frame t. Let R represent the embedding sequence of frame t. T×DThis indicates that the sequence contains T time steps, each with a feature dimension of D. The linear projection layer transforms the input features from the original dimension to D dimensions using a set of learnable weight matrices. This process can be viewed as a re-encoding of the features to make them more suitable for processing by subsequent modules.

[0028] Embedded sequence Deep processing will be performed using the MambaOAD encoder (MOE), which is based on an improved Mamba block design. The MOE's structure is ingeniously designed, consisting of alternating Mamba blocks and normalization layers, and cleverly constructs residual connections to effectively capture long-term temporal dependencies. The Mamba blocks introduce a selective mechanism that dynamically adjusts the propagation and forgetting process of hidden states based on the input. Specifically, the selective mechanism in the Mamba blocks uses a gating structure to filter input information, deciding which information needs to be retained and propagated to the next time step, and which information can be forgotten. This dynamic adjustment capability allows the model to accurately retain key information and effectively suppress irrelevant or redundant data, providing more valuable feature representations for action detection. The normalization layers standardize the features, helping to accelerate model convergence and improve training stability. The residual connections skip some intermediate layers, directly adding the outputs of previous layers to the outputs of subsequent layers. This avoids the gradient vanishing or exploding problems that occur in deep networks, allowing the model to better learn long-term temporal dependencies.

[0029] Finally, the features extracted after MOE processing Where f t The enhanced features of frame t, after processing by the MOE module, will be fed into the classifier for the final action detection decision. The classifier typically uses a fully connected neural network (FCN), which calculates the probability of each action category based on the extracted feature vector and selects the category with the highest probability as the prediction result.

[0030] Further, step S5 includes the following:

[0031] Specifically, the F output of ADM D It will be meticulously categorized into long-term potential characteristics and short-term potential characteristics Where T S This represents the length of the short-term window. This division helps the model process and analyze features at different time scales separately.

[0032] FAM uses the short-term latent characteristics of ADM output F S As input, the MOE module is used to predict future latent features. The T here AThe FAM (Future Action Model) represents the range of future predictions. It employs the same MOE (Moment of Effect) module as the ADM (Active Model for Determining Moment of Time) and reuses the selective SSM (Selective Streaming Model) mechanism, enabling it to efficiently capture temporal dependencies. By providing the ADM with a future-oriented perspective, the FAM helps it learn richer and more comprehensive temporal representations, further enhancing its ability to model the dynamic evolution patterns of actions in videos, thereby improving the overall model's action detection performance in complex scenes. During training, the difference between the future latent features predicted by the FAM and the actual future features can serve as a supervisory signal, helping the ADM better learn temporal features.

[0033] Further, step S6 includes the following:

[0034] In terms of specific operation, during the training phase, the output F of ADM is... D With the output F of FAM A A concatenation operation is performed, which concatenates the two feature sequences sequentially to obtain a longer feature sequence. Normalization is then applied to ensure the concatenated features have a uniform scale, facilitating subsequent classifier learning. Finally, the classifier predicts action labels.

[0035] In this process, it is necessary to calculate the classification loss for the current frame separately. And auxiliary loss for future frame prediction Classification loss of the current frame This measures the accuracy of the model's prediction of the action at the current time step, and is typically calculated using the cross-entropy loss function. The auxiliary loss for future frame prediction... This reflects the difference between the future potential characteristics predicted by FAM and the actual future characteristics, and can also be calculated using cross-entropy loss function or mean squared error loss function. The total loss function is defined as the weighted sum of the two, i.e. Here, λ1 and λ2 are hyperparameters. In this method, after extensive experimental verification, λ1 is set to 1.0 and λ2 is set to 0.1. This setting aims to emphasize the primary importance of the current frame loss, while fully incorporating the future frame loss, in order to enhance the model's learning effect on temporal representation.

[0036] During the inference phase, the model only uses Action Detection Model (ADM) for action detection. ADM processes the input video features through a linear projection layer, the MOE module, and a classifier, directly outputting action prediction results. This ensures both detection accuracy and maximizes inference efficiency, meeting the stringent real-time requirements of practical applications. mAP and mcAP metrics are used to measure online action detection accuracy, while FPS and GFLOPs metrics are used to measure the complexity of model inference.

[0037] Beneficial effects:

[0038] This invention presents an online action detection technique based on the Mamba architecture, effectively improving the accuracy and robustness of action detection. The method utilizes Mamba's selective state-space mechanism and MOE module design to effectively capture long-term dependencies, thereby enhancing action detection accuracy. The method's linear computational complexity and hardware-aware optimization enable fast inference while maintaining low computational overhead, meeting the demands of real-time applications. It performs exceptionally well in action boundary recognition in complex scenes, enhancing the model's ability to identify ambiguous action boundaries and reducing false positives. Experiments on multiple benchmark datasets demonstrate that the proposed method outperforms traditional online action detection methods across various evaluation metrics, exhibiting higher detection accuracy and stronger generalization ability. Attached Figure Description

[0039] Figure 1 This is a schematic diagram of online motion detection in this invention;

[0040] Figure 2 This is a framework diagram of the Mamba-based online action detection model of the present invention;

[0041] Figure 3 This is a framework diagram of the motion detection module (ADM) in this invention;

[0042] Figure 4 This is a framework diagram of the future prediction module (FAM) in this invention. Detailed Implementation

[0043] To more clearly illustrate the technical solutions of this invention, the invention is described below in conjunction with the accompanying drawings. The examples given are for illustrative purposes only and are not intended to limit the scope of the invention.

[0044] refer to Figure 2 A real-time online action detection method based on the Mamba architecture includes the following steps:

[0045] S1, Acquiring Training Data: Obtaining videos that meet the criteria and creating a supervised dataset to support experiments with online action detection technology based on the Mamba architecture, such as... Figure 1 As shown. Further, S1 includes the following:

[0046] (1) Check the perspective of the video dataset. It is necessary to monitor or film the same action of the same sample from at least 3 different perspectives to ensure that the model can learn the ability to model space from different perspectives from the training data.

[0047] (2) Check the length of the video dataset. In the actual application of online action detection, it is necessary to perform real-time action classification on the online video sequence. Therefore, the videos in the dataset must be in an unedited state, with background and several actions lasting a certain period of time in the same video to simulate the real situation.

[0048] (3) Check the labeling of the video dataset. The videos must have fine-grained action labels, and the videos must have the same label that can be used to form a comparison between different perspectives for training the deep model.

[0049] S2, Data Preprocessing. In the multi-view online motion detection method of this invention, real-time video streams are processed as input data. To effectively capture motion features in the video, consecutive N frames of video are divided into a group, and the group currently being processed is labeled T. Furthermore, the concepts of short-term memory and long-term memory are introduced to enhance the model's understanding and prediction capabilities regarding dynamic changes in video sequences.

[0050] Specifically, the previous S groups of video frames, including the current group T, are defined as short-term memory I. S The first L groups of short-term memory are defined as long-term memory I. L RGB and optical flow information are extracted from each memory group to obtain rich spatiotemporal features. For short-term memory I... S and long-term memory I L RGB and optical flow features are extracted from each group respectively. The RGB features are extracted by selecting the RGB image of the [N / 2]th frame in the group, while the optical flow features are extracted by calculating the cumulative optical flow field of the N-1 frames of optical flow images generated from the N frames of RGB images.

[0051] Furthermore, a pre-trained two-dimensional convolutional neural network (2D CNN) is used to extract RGB and optical flow features for each group in depth. This 2D CNN is specifically designed to simultaneously capture spatial and temporal information from video frames, thereby achieving efficient recognition of dynamic changes in video content. In each group, RGB information is extracted by selecting the RGB image of the median frame within the group, ensuring the representativeness of the features; while optical flow information is extracted by superimposing N frames of RGB images to obtain N-1 frames of optical flow images, in the form of an accumulated optical flow field, which helps to capture motion information in the video.

[0052] The extracted RGB and optical flow features are concatenated along the channel dimension to form a set of comprehensive features. These features not only contain rich color and motion information, but also enhance the model's utilization of historical information in video sequences by introducing short-term and long-term memory, thereby improving the accuracy and robustness of action detection.

[0053] Through this design, the present invention enables a more comprehensive understanding and analysis of video content, demonstrating superior performance, especially in motion detection tasks involving multiple perspectives and complex scenes. Furthermore, the present invention can be applied to various fields such as video surveillance, human-computer interaction, and autonomous driving, providing an efficient and reliable solution for real-time video analysis.

[0054] S3, based on the action detection module and the future prediction module, builds an online action detection network with a Mamba architecture, specifically through the following steps:

[0055] The MambaOAD framework is introduced, which consists of two closely related and complementary modules: the Action Detection Module (ADM) and the Future Prediction Module (FAM).

[0056] ADM fully leverages the high efficiency of Mamba, capturing spatiotemporal dependencies through State Space Models (SSMs) and convolutional mechanisms to achieve real-time online action detection. State Space Models can model dynamic changes, effectively capturing the evolution of actions over time in video action detection. Convolutional mechanisms excel at extracting spatial features from images; the combination of the two comprehensively processes spatiotemporal information in videos. FAM (Focus-Based Action Model) only plays a role during the training phase, its primary responsibility being to predict future actions, providing forward-looking information to strongly support the spatiotemporal representation learning of ADM. During the inference phase, to ensure computational efficiency, only ADM is retained. This design guarantees detection accuracy while significantly improving inference speed, meeting the demands of real-time applications.

[0057] S4, build Figure 3 The motion detection module shown in the figure consists of the following steps:

[0058] At time step T, ADM receives the input video features. Then, a linear projection layer is used to precisely map each frame to a D-dimensional feature space, resulting in the embedded sequence. Where x t Represents the original video features of frame t. Let R represent the embedding sequence of frame t. T×D This indicates that the sequence contains T time steps, each with a feature dimension of D. The linear projection layer transforms the input features from the original dimension to D dimensions using a set of learnable weight matrices. This process can be viewed as a re-encoding of the features to make them more suitable for processing by subsequent modules.

[0059] Embedded sequence Deep processing will be performed using the MambaOAD encoder (MOE), which is based on an improved Mamba block design. The MOE's structure is ingeniously designed, consisting of alternating Mamba blocks and normalization layers, and cleverly constructs residual connections to effectively capture long-term temporal dependencies. The Mamba blocks introduce a selective mechanism that dynamically adjusts the propagation and forgetting process of hidden states based on the input. Specifically, the selective mechanism in the Mamba blocks uses a gating structure to filter input information, deciding which information needs to be retained and propagated to the next time step, and which information can be forgotten. This dynamic adjustment capability allows the model to accurately retain key information and effectively suppress irrelevant or redundant data, providing more valuable feature representations for action detection. The normalization layers standardize the features, helping to accelerate model convergence and improve training stability. The residual connections skip some intermediate layers, directly adding the outputs of previous layers to the outputs of subsequent layers. This avoids the gradient vanishing or exploding problems that occur in deep networks, allowing the model to better learn long-term temporal dependencies.

[0060] Finally, the features extracted after MOE processing Where f t The enhanced features of frame t, after processing by the MOE module, will be fed into the classifier for the final action detection decision. The classifier typically uses a fully connected neural network (FCN), which calculates the probability of each action category based on the extracted feature vector and selects the category with the highest probability as the prediction result.

[0061] S5, build Figure 4 The future prediction module shown below involves the following steps:

[0062] Specifically, the output of ADM will be meticulously divided into long-term latent features. and short-term potential characteristics Where T S This represents the length of the short-term window. This division helps the model process and analyze features at different time scales separately.

[0063] FAM uses the short-term latent characteristics of ADM output F S As input, the MOE module is used to predict future latent features. The T here AThe FAM (Future Action Module) represents the range of future predictions. It employs the same MOE (Moment of Effect) module as the ADM (Active Motion Module) and reuses the selective SSM (Selective SSM) mechanism, enabling it to efficiently capture temporal dependencies. By providing the ADM with a future-oriented perspective, the FAM helps it learn richer and more comprehensive temporal representations, further enhancing its ability to model the dynamic evolution patterns of actions in videos, thereby improving the overall model's action detection performance in complex scenes. During training, the difference between the future latent features predicted by the FAM and the actual future features can serve as a supervisory signal, helping the ADM better learn temporal features.

[0064] S6, Model Training and Optimization, specifically includes the following steps:

[0065] The MambaOAD framework performs joint optimization of ADM and FAM during training. Specifically, during the training phase, the output F of ADM is... C With the output F of FAM A A concatenation operation is performed, which concatenates the two feature sequences sequentially to obtain a longer feature sequence. Normalization is then applied to ensure the concatenated features have a uniform scale, facilitating subsequent classifier learning. Finally, the classifier predicts action labels.

[0066] In this process, it is necessary to calculate the classification loss for the current frame separately. And auxiliary loss for future frame prediction In this process, it is necessary to calculate the classification loss for the current frame separately. And auxiliary loss for future frame prediction Classification loss of the current frame This measures the accuracy of the model's prediction of the action at the current time step, and is typically calculated using the cross-entropy loss function. The auxiliary loss for future frame prediction... This reflects the difference between the future potential characteristics predicted by FAM and the actual future characteristics, and can also be calculated using cross-entropy loss function or mean squared error loss function. The total loss function is defined as the weighted sum of the two, i.e. Here, λ1 and λ2 are hyperparameters. In this method, after extensive experimental verification, λ1 is set to 1.0 and λ2 is set to 0.1. This setting aims to emphasize the primary importance of the current frame loss, while fully incorporating the future frame loss, in order to enhance the model's learning effect on temporal representation.

[0067] During the inference phase, the model only uses Action Detection Model (ADM) for action detection. ADM processes the input video features through a linear projection layer, the MOE module, and a classifier, directly outputting action prediction results. This ensures both detection accuracy and maximizes inference efficiency, meeting the stringent real-time requirements of practical applications. mAP and mcAP metrics are used to measure online action detection accuracy, while FPS and GFLOPs metrics are used to measure the complexity of model inference.

[0068] Experiments on publicly available datasets demonstrate the performance of the proposed method across different datasets and modalities. In this embodiment, the THUMOS14, FineAction, 50Salads, and IKEA ASM datasets were selected for experiments. Experimental results show that the method of this invention has high effectiveness and generalization ability in handling long-term sequence action detection tasks. To verify the detection performance and operational efficiency of this invention, a comparison was made with traditional online action detection methods. Specific performance metrics for the THUMOS14 dataset are shown in Table 1, metrics for other datasets are shown in Table 2, and efficiency metrics are shown in Table 3.

[0069] Table 1 Comparison of different methods on the THUMOS'14 dataset

[0070]

[0071] Table 2 compares the different methods on the FineAction dataset, the 50Salads dataset, and the IKEA ASM dataset.

[0072]

[0073] As shown in Tables 1 and 2, the method of this invention outperforms traditional online action detection methods in both mAP and mcAP evaluation metrics, thus demonstrating higher detection accuracy. This invention not only solves the problems of insufficient background suppression and inadequate capture of temporal relationships but also possesses good generalization ability. Furthermore, as shown in Table 3, this invention effectively addresses the problem of limited computational resources, significantly reducing the model size and achieving sufficient running speed to meet practical application requirements. Its performance on complex datasets surpasses that of traditional online action detection models, proving its effectiveness.

[0074] Table 3 Comparison of online action detection efficiency metrics for different methods on the THUMOS'14 dataset.

[0075]

[0076] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any other changes or substitutions that do not involve substantial innovation should be covered within the scope of protection of the present invention. Therefore, the specific scope of protection of the present invention should be determined by the scope of protection of the claims.

Claims

1. A real-time online action detection method based on Mamba architecture, characterized in that, Includes the following steps: S1, Training data acquisition: Acquire video streams from multiple perspectives in the target scene; S2, data preprocessing, extracts video features from the video stream based on a pre-trained two-dimensional convolutional neural network; S3 introduces the MambaOAD framework, which consists of the Action Detection Module (ADM) and the Future Prediction Module (FAM). S4. Construct the Action Detection Module (ADM). The ADM is used to realize real-time online action detection. It maps to the feature space through a linear projection layer. The linear projection layer transforms the input video features to D dimensions through a learnable weight matrix to obtain the embedded sequence. The embedded sequence will be deeply processed by the MambaOAD encoder (MOE module) based on the improved Mamba block design. S5, Construct the future prediction module FAM. FAM is used to extend the functionality of the MambaOAD framework. It predicts future potential features based on the short-term potential features output by ADM. The FAM uses the MOE module to predict future potential features. S6, Model Training and Inference: During the training process, the MambaOAD framework jointly optimizes ADM and FAM and evaluates performance based on actual action labels. Step S3 includes the following: Action Detection Model (ADM) captures spatiotemporal dependencies through State-Space Models (SSMs) and convolutional mechanisms to achieve real-time online action detection. The State-Space Model models dynamically change and effectively capture the evolution of actions over time in video action detection. Convolutional mechanisms extract spatial features from images; the combination of these two methods comprehensively processes spatiotemporal information in videos. The Functional Aspect Model (FAM) only plays a role during the training phase, predicting future actions and providing forward-looking information to strongly support the spatiotemporal representation learning of ADM. During the inference phase, to ensure computational efficiency, only ADM is retained, guaranteeing both detection accuracy and significantly improving inference speed to meet the demands of real-time applications.

2. The real-time online action detection method based on Mamba architecture according to claim 1, characterized in that, Step S2 includes the following: Real-time video streams are used as input data. Every N frames of video are processed as a group, and a pre-trained convolutional neural network is used to extract the video features of the frames in this group, namely RGB image features and optical flow features. The convolutional neural network is a two-dimensional convolutional neural network, which is used to capture the spatial and temporal information of video frames at the same time, so as to achieve efficient recognition of dynamic changes in video content.

3. The real-time online action detection method based on Mamba architecture according to claim 1, characterized in that, Step S4 includes the following: At time step T, the ADM receives the input video features. Then, through a linear projection layer, each frame is precisely mapped to a D-dimensional feature space to obtain the embedded sequence. ,in Indicates the first The original video features of the frame, Indicates the first Frame embedding sequence, This indicates that the sequence contains T time steps, and each step has a feature dimension of D; The linear projection layer transforms the input video features from the original dimension to D dimension through a set of learnable weight matrices, which is used to re-encode the features and make them more suitable for processing by subsequent modules. Embedded sequence The deep processing will be performed by the MambaOAD encoder, or MOE module, based on an improved Mamba block design. The MOE module consists of alternating Mamba blocks and normalization layers, and effectively captures long-term temporal dependencies by constructing residual connections. The Mamba blocks introduce a selective mechanism that dynamically adjusts the propagation and forgetting process of hidden states based on the input. Specifically, the selective mechanism in the Mamba blocks uses a gating structure to filter input information, deciding which information needs to be retained and propagated to the next time step, and which information can be forgotten. This dynamic adjustment capability allows the model to accurately retain key information and effectively suppress irrelevant or redundant data, providing more valuable feature representations for action detection. The normalization layers standardize the features to accelerate model convergence and improve training stability. The residual connections skip intermediate layers, directly adding the outputs of previous and subsequent layers, avoiding the gradient vanishing or exploding problems in deep networks, enabling the model to better learn long-term temporal dependencies. Finally, the features extracted after processing by the MOE module ,in Indicates the first The enhanced features of the frame after processing by the MOE module will be fed into the classifier for the final action detection decision. The classifier uses a fully connected neural network, which calculates the probability of each action category based on the extracted feature vector and selects the category with the highest probability as the prediction result.

4. The real-time online action detection method based on Mamba architecture according to claim 3, characterized in that, Step S5 includes the following: The ADM output is will be divided into long-term latent features and short-term latent features where represent the length of the short-term window, for the model to process and analyze features of different time scales respectively; FAM as a short-term latent feature of ADM output As input, the MOE module is used to predict future latent features. ,in FAM represents the range of future predictions; it uses the same MOE module as ADM and reuses the selective SSM mechanism, enabling it to efficiently capture temporal dependencies; by providing ADM with a future-oriented perspective, FAM helps ADM learn richer and more comprehensive temporal representations, further enhancing its ability to model the dynamic evolution patterns of actions in videos, thereby improving the overall model's action detection performance in complex scenes; during training, the difference between the future potential features predicted by FAM and the actual future features serves as a supervisory signal, helping ADM to better learn temporal features.

5. The real-time online action detection method based on the Mamba architecture according to claim 4, characterized in that, Step S6 includes the following: During the training phase, the output of ADM will be... With FAM output The concatenation operation is performed, which concatenates the two feature sequences in sequence to obtain a longer feature sequence; then, normalization is performed to ensure that the concatenated features have a uniform scale, which facilitates the learning of the subsequent classifier; finally, the action label is predicted by the classifier. During this process, the classification loss for the current frame is calculated respectively. And auxiliary loss for future frame prediction Classification loss for the current frame The measurement focuses on the accuracy of the model's prediction of the action at the current time step, calculated using the cross-entropy loss function; the auxiliary loss is used for future frame prediction. This reflects the difference between the future potential features predicted by FAM and the actual future features, calculated using the cross-entropy loss function or the mean squared error loss function; the total loss function is defined as the weighted sum of the two, i.e. ,in and For hyperparameters; and Set to 1.0, Setting it to 0.1 emphasizes the primary importance of the current frame loss while fully incorporating the future frame loss to enhance the model's learning of temporal representations. During the inference phase, the model only uses ADM for action detection. Based on the input video features, ADM processes the data through a linear projection layer, MOE module, and classifier, and directly outputs the action prediction results. This ensures both the accuracy of detection and maximizes inference efficiency, meeting the stringent real-time requirements of practical applications. The online action detection accuracy is measured using mAP and mcAP metrics, while the inference complexity is measured using FPS and GFLOPs metrics.

Images