Online action detection method based on multi-stage memory mapping

Abstract

The application relates to an online action detection method based on a multi-stage memory mapping, and the specific steps comprise the following steps of S1, feature extraction: extracting frame-level features from a video stream by using a pre-trained model; S2, memory division: dividing the extracted features into long-term memory and short-term memory according to time; S3, multi-stage mapping memory coding: adopting a multi-stage mapping encoder for the long-term memory, including a segmented compression branch and a global compression branch, compressing and extracting key information, and suppressing irrelevant backgrounds; S4, cyclic decoding update: adopting a cyclic decoder to dynamically update the short-term memory, generating prediction information of future actions; S5, model training and optimization: improving a loss function, optimizing the fusion effect of local features and global dependence in the model, and improving the overall detection performance; S6, online action detection: based on the memory features updated by the cyclic decoding, real-time detection of actions is carried out, and the performance is evaluated according to actual action labels. The application improves the accuracy and robustness of action detection.

Description

Technical Field

[0001] This invention relates to an online motion detection technology based on multi-stage memory mapping, belonging to the field of motion video understanding technology. Background Technology

[0002] With the continuous advancement of computer vision and deep learning technologies, real-time video understanding has become a core requirement in fields such as intelligent surveillance, autonomous driving, and human-computer interaction. Online action detection technology requires models to identify ongoing actions in real time relying solely on current and historical frame information, without utilizing future frame data. This presents the challenge of efficiently utilizing historical memory while simultaneously suppressing irrelevant background noise.

[0003] Traditional online action detection methods are mainly divided into two categories: RNN-based and Transformer-based. The former performs well in capturing local temporal dependencies but is insufficient in maintaining long-term memory; while the latter, although able to model global dependencies through cross-attention mechanisms, is easily affected by noisy backgrounds when processing a large number of historical frames, leading to performance degradation. These problems limit the application effectiveness of traditional methods in complex video scenes.

[0004] To address the aforementioned issues, an online action detection technique based on multi-stage memory mapping is proposed. This framework refines historical memory through a dual-branch memory encoder. On one hand, it utilizes Selective State Memory (SSM) combined with one-dimensional convolution and a Transformer decoder to extract local continuous event features. On the other hand, it employs a lightweight grouped cascaded attention mechanism to capture global temporal dependencies. These two mechanisms are fused through a multi-stage reference process, resulting in a high-quality historical memory representation that is sensitive to local dynamics while also considering global information. Summary of the Invention

[0005] Technical issues:

[0006] The challenge of balancing long-term memory and local event features: Online action detection relies solely on past frame information. How to effectively compress long-term memory and extract key information from local continuous events is a significant challenge facing current methods.

[0007] Noise interference suppression problem: Historical frames often contain a lot of irrelevant background information. Traditional models are easily distracted when dealing with this noise, thus reducing detection performance.

[0008] Real-time performance and computational efficiency issues: Under limited computing resources, how to design an efficient memory encoding and update mechanism to meet the needs of real-time detection is a major challenge in practical applications.

[0009] To address the aforementioned technical problems, this invention proposes an online action detection method based on multi-stage memory mapping. This method, by constructing a dual-branch memory encoder and a cyclic decoding and update mechanism, can simultaneously ensure the real-time performance and accuracy of the detection process.

[0010] Technical solution:

[0011] This invention relates to an online action detection method based on multi-stage memory mapping, belonging to the field of action video understanding. The method includes the following steps:

[0012] S1, Feature Extraction: Frame-level features (such as RGB and optical flow) are extracted from the video stream using a pre-trained model to provide basic data for subsequent processing;

[0013] S2, Memory Division: The extracted features are divided into long-term memory (distant historical information) and short-term memory (recent key information) according to time, so as to handle global dependencies and local dynamics respectively;

[0014] S3, Multi-level Mapping Memory Encoding: A multi-stage mapping encoder is used for long-term memory, which is divided into a segmented compression branch (focusing on local events through local 1D convolution and Transformer decoder) and a global compression branch (capturing global temporal dependencies based on grouped cascaded attention) to compress and extract key information while suppressing irrelevant background.

[0015] S4, Loop Decoding Update: A loop (recursive) decoder is used to dynamically update the short-term memory and generate prediction information for future actions, ensuring semantic continuity between the current detection and future prediction;

[0016] S5, Model Training and Optimization: By designing an improved loss function (integrating background suppression loss and cross-entropy loss), the fusion effect of local features and global dependencies in the model is optimized, thereby improving the overall detection performance;

[0017] S6, an online action detection system, performs real-time action detection based on memory features updated through loop decoding and evaluates performance based on actual action labels.

[0018] Further, step S1 includes the following:

[0019] The input video stream is processed at fixed intervals, with each group consisting of N frames, to ensure a stable temporal structure. A pre-trained two-dimensional convolutional neural network (2D CNN) is used to process the frames within each group, extracting RGB image features and optical flow features. This process converts each frame into a feature representation. Here, C represents the feature dimension. This process simultaneously captures the spatial and motion information of video frames, ensuring efficient identification of dynamic changes in video content. The extracted RGB and optical flow features are input into the model for encoding and then concatenated along the channel dimension to form a complete video frame feature representation, providing unified foundational data for subsequent memory segmentation and action detection.

[0020] Further, step S2 includes the following:

[0021] The extracted features are further continuously divided into subsets containing different temporal fine-grained sets: long-term memory and short-term memory. Long-term memory stores features far removed from the current time point and is defined as follows: Short-term memory is used to store a small number of recently occurring features, defined as Where t represents a single time frame, m s Let T represent the length of short-term memory and T represent the total time of the input model. The length of long-term memory is much longer than that of short-term memory. According to the streaming structure of video, as time goes by, frames in short-term memory will gradually enter long-term memory, and short-term memory will also be dynamically updated over time.

[0022] Further, step S3 includes the following:

[0023] (1) Construction of segmented compressed branches

[0024] Long-term memory M L Divide into k non-overlapping subsequences Each subsequence s i The length is For each s i First, one-dimensional convolution (1D Conv) is used to extract local continuous event features along the temporal direction; then, a set of learnable query vectors is introduced. (T S (where D is the subsequence length and D is the feature embedding dimension), and a Transformer decoder with shared weights is used to process s. i Self-attention encoding is performed to focus on key local information. Finally, average pooling is applied to the encoding results to obtain an abstract representation of each subsequence. The concatenated representations of all subsequences are then input into the Mamba module to achieve selective information transfer and noise suppression, thereby generating segmented compressed memory.

[0025] (2) Construction of global compression branch

[0026] To capture the entire long-term memory M L The global temporal dependencies are determined using a Transformer decoder based on grouped concatenated attention. The specific steps are as follows:

[0027] a. M L The block is divided into multiple sub-blocks, and self-attention is calculated for each attention head j (j = 1, 2, ..., h). The calculation formula is as follows:

[0028]

[0029] Where X ij M represents L Sub-features within the j-th group Here, Attn represents the corresponding projection matrix, and Concat represents the concatenation along the feature dimension.

[0030] b. Employ a cascaded fusion strategy to gradually add the outputs of each attention head:

[0031]

[0032] c. In cross-attention computation, an equalization factor α is introduced to modify the query vector to mitigate the impact of low-quality accumulated information, thereby obtaining globally compressed memory.

[0033]

[0034] (3) Dual-branch feature fusion and final memory representation generation

[0035] Compress and memorize segments containing key local information. Use this query to access the globally compressed memory that contains global dependencies in the upper branch. This process produces As a supplement to local information, M is obtained by weighted summation followed by LayerNorm normalization. L M L The input is fed into two consecutive Transformer encoder blocks to obtain the long-term memory for the final compression. It is expressed as follows:

[0036]

[0037] This final representation incorporates both sensitive features of local continuous events and global temporal dependency information, providing a high-quality historical information reference for subsequent short-term memory enhancement and online action detection.

[0038] Further, step S4 includes the following:

[0039] The Transformer decoder module is used to inject compressed long-term memory into short-term memory. Short-term memory is used as a query to extract local event-sensitive information and stable long-term time dependencies from historical memory, thereby enhancing short-term memory.

[0040] (1) Initialization:

[0041] Compressed long-term memory M L With short-term memory M S As input, initialize and generate the query vector Q. g and future query vector Q F At the same time, short-term memory sets and future prediction set Initialize to empty.

[0042] (2) Generate future predictions:

[0043] First, long-term and short-term memories are cascaded in chronological order to form a memory matrix.

[0044] M A =[M L M S ]

[0045] Then, short-term memory M S Selective feature extraction is performed to obtain an enhanced short-term memory representation.

[0046] M S′ =Mamba(M S )

[0047] and combine it with the initial query Q g Cascade to form future query vectors

[0048]

[0049] Next, the cross-attention mechanism is used to calculate the future prediction representation.

[0050] F A =CrossAttn(Q F′ M A M A )

[0051] (3) Cyclic Update:

[0052] For subsequent loop decoding iterations, the following update strategy is adopted:

[0053] Update short-term memory: Utilize the current memory matrix M A And the predicted result F A Short-term memory is updated through a cross-attention mechanism, calculated using the following formula:

[0054] M S =CrossAttn(M S ,F E ,F E )

[0055] Feature extraction and query update: For the updated M S′ The shared Mamba module is applied again to generate a new enhanced representation M. S′ And concatenate with the original query to update the future query vector Q F′ =[M S′ Q F ], and then recalculate the future prediction F A The formula is as follows:

[0056] F E =[M L M S ,F A ]

[0057] F A =CrossAttn(Q F′ ,F E ,F E )

[0058] Iterative output: After each iteration, the updated short-term memory M will be output. S And future prediction F A Add to set respectively and In, until the predetermined N is completed. t The loop iterates once.

[0059] Output result:

[0060] Finally, after N t After each iteration, the recurrent decoder outputs an updated short-term memory representation and future prediction information. These two outputs not only capture the dynamic features of the current frame but also semantically maintain the continuity between the current detection and future action prediction, thus providing a reliable contextual reference for subsequent online action detection.

[0061] Further, step S5 includes the following:

[0062] (1) A customized loss function is used to train the model globally. Specifically, for each video frame t, a loss function is defined. as follows:

[0063]

[0064] in, This indicates that the current frame is classified as background. γ represents the model's predicted probability for class i in video frame t. a With γ b This design incorporates hyperparameters to control the penalty level for hard samples. It enables the model to impose higher penalties on low-confidence samples, effectively suppressing background noise and mitigating class imbalance caused by the significantly larger number of background samples compared to action samples.

[0065] (2) Calculate the loss for each of the two main branches in the model—the classification branch and the probability branch. In the initial stage, using the enhanced short-term memory and the output generated by future predictions, the action probabilities are obtained through a weighted classification head, thus yielding the short-term memory classification loss. and future predicted losses

[0066]

[0067] Where m s and T F These represent the length of short-term memory and the length of time used to generate future predictions, respectively.

[0068] Subsequently, for each iteration of the loop (1≤i≤N) t Define the corresponding memory loss. and anticipated losses The calculation method is the same as that in the initial stage.

[0069] (3) The overall training loss is expressed as a weighted summation, defined as follows:

[0070]

[0071] Where, λ s and λ f These are the balance coefficients for short-term memory loss and future prediction loss, respectively, used to ensure that the model can optimize the fusion of local features and global temporal dependencies while taking into account the accuracy of current detection and future prediction.

[0072] (4) By minimizing the joint loss function mentioned above, the model not only improves the action recognition ability of the classification branch during training, but also optimizes the fusion effect of the probability branch on local features and global dependency information, thereby significantly improving the overall detection performance. In particular, since the number of background samples far exceeds that of action samples, different hyperparameter settings can better suppress background noise, prompting the model to pay more attention to high-confidence prediction of action classes.

[0073] Further, step S6 includes the following:

[0074] (1) Construct a detection head for online action detection and prediction tasks.

[0075] (2) Use the output M generated by the last interaction S and F A These are respectively used as the results of online action detection and prediction. Specifically, for the online action detection task, M is extracted. S The last token is detected. For the action prediction task, based on the interval time τ, from F... A Extract the corresponding token from the data for prediction.

[0076] (3) Use mAP and mcAP metrics to measure the accuracy of online action detection, and use FPS and GFLOPs metrics to measure the complexity of model inference.

[0077] Beneficial effects:

[0078] The online action detection method based on multi-stage memory mapping in this invention effectively improves the accuracy and robustness of action detection by introducing a multi-level compressed memory mechanism. This method effectively addresses the key challenges of capturing long-term dependencies and local event sensitivity, enhancing the model's ability to compress high-quality historical information. Simultaneously, the optimized network design and training strategy ensure the real-time performance and computational efficiency of the detection process. Furthermore, test results on multiple datasets verify its good generalization ability and application potential in various monitoring and analysis tasks. It is easy to implement and integrate, providing an efficient and reliable technical solution for real-time action detection. Attached Figure Description

[0079] Figure 1 This is a schematic diagram of an example video stream for online motion detection in this invention;

[0080] Figure 2 This is a framework diagram of the online action detection model based on multi-stage memory mapping of the present invention;

[0081] Figure 3 This is a diagram of the segmented compression branch structure framework of the present invention;

[0082] Figure 4 This is a framework diagram of the cascaded attention module of the present invention;

[0083] Figure 5 This is a framework diagram of the Transformer decoder module in this invention. Detailed Implementation

[0084] To more clearly illustrate the technical solutions of this invention, the invention is described below in conjunction with the accompanying drawings. The examples given are for illustrative purposes only and are not intended to limit the scope of the invention.

[0085] refer to Figure 2This invention provides an online action detection method based on multi-stage memory mapping, comprising the following steps:

[0086] S1, Feature Extraction: This step utilizes a pre-trained model to extract frame-level features (such as RGB and optical flow) from the video stream, providing foundational data for subsequent processing. Figure 1 As shown, the system utilizes a pre-trained two-dimensional convolutional neural network (2D CNN) to deeply extract RGB and optical flow features from each group. This 2D CNN is specifically designed to simultaneously capture spatial and temporal information from video frames, thereby achieving efficient recognition of dynamic changes in video content. In each group, RGB information is extracted by selecting the RGB image of the median frame within the group, ensuring the representativeness of the features; while optical flow information is extracted by superimposing N frames of RGB images to obtain N-1 frames of optical flow images, in the form of an accumulated optical flow field, which helps to capture motion information in the video.

[0087] In this embodiment, S1 includes the following:

[0088] (1) Divide the input video stream into fixed time intervals, forming a group every N frames to ensure that the input data has a stable and clear temporal structure;

[0089] (2) For each frame in the group, a pre-trained two-dimensional convolutional neural network (2D CNN) is used to extract features, and RGB image features (capturing spatial information) and optical flow features (capturing motion information) are extracted at the same time, so as to efficiently represent the static and dynamic features in the video frame.

[0090] (3) The extracted RGB features and optical flow features are encoded by the model and then spliced ​​along the channel dimension to form a complete and unified video frame feature representation, ensuring that the data has rich spatial and motion information.

[0091] S2, Memory Division. In the online motion detection system of this invention, real-time video streams are processed as input data. To effectively capture motion features in the video, the system divides N consecutive video frames into a group, labeling the currently processed group as T. Furthermore, the system introduces the concepts of short-term memory and long-term memory to enhance the model's understanding and prediction capabilities regarding dynamic changes in video sequences.

[0092] In this embodiment, S2 includes the following:

[0093] Specifically, long-term memory is used to store features that are far removed from the current moment, and is defined as follows: Short-term memory is used to store a small number of recently occurring features, defined as Long-term memory is much longer than short-term memory. According to the streaming structure of video, as time goes by, frames in short-term memory will gradually enter long-term memory, and short-term memory will also be dynamically updated over time to obtain rich spatiotemporal features.

[0094] Through this design, the system of this invention can more comprehensively understand and analyze video content, demonstrating superior performance, especially in motion detection tasks in complex scenes. Furthermore, the system can be applied to various fields such as video surveillance, human-computer interaction, and autonomous driving, providing an efficient and reliable solution for real-time video analysis.

[0095] S3, an online action detection network based on multi-level mapping memory encoding, specifically comprises the following steps:

[0096] (1) Construction of segmented compressed branches: Refer to Figure 3 , will long-term memory M L Divide into k non-overlapping subsequences Each subsequence s i The length is For each s i First, one-dimensional convolution (1D Conv) is used to extract local continuous event features along the temporal direction; then, a set of learnable query vectors is introduced. (T S (where is the subsequence length), and uses a Transformer decoder with shared weights to process s. i Self-attention encoding is performed to focus on key local information. Finally, average pooling is applied to the encoding results to obtain an abstract representation of each subsequence. The concatenated representations of all subsequences are then input into the Mamba module to achieve selective information transfer and noise suppression, thereby generating segmented compressed memory.

[0097] (2) Construction of the global compression branch: to capture the entire long-term memory M L For global time-series dependencies, refer to Figure 4 A Transformer decoder based on grouped cascaded attention is used. The specific steps are as follows:

[0098] a. M L The block is divided into multiple sub-blocks, and self-attention is calculated for each attention head j (j = 1, 2, ..., h). The calculation formula is as follows:

[0099]

[0100] Where X ij M represents L Sub-features within the j-th group For the corresponding projection matrix;

[0101] b. Employ a cascaded fusion strategy to gradually add the outputs of each attention head:

[0102]

[0103] c. In cross-attention computation, an equalization factor α is introduced to modify the query vector to mitigate the impact of low-quality accumulated information, thereby obtaining globally compressed memory.

[0104]

[0105] (3) Dual-branch feature fusion and final memory representation generation: Reference Figure 5 Compressed memory containing key local information Use the query to access historical memories in the upper branch that contain global dependencies. This process produces As a supplement to local information, M is obtained by weighted summation followed by LayerNorm normalization. L M L The input is fed into two consecutive Transformer encoder blocks to obtain the final compressed long-term memory. It is expressed as follows:

[0106]

[0107] This final representation incorporates both sensitive features of local continuous events and global temporal dependency information, providing a high-quality historical information reference for subsequent short-term memory enhancement and online action detection.

[0108] S4, Construct the loop decoding and update module, specifically through the following steps:

[0109] The Transformer decoder module is used to inject compressed long-term memory into short-term memory. Short-term memory is used as a query to extract local event-sensitive information and stable long-term time dependencies from historical memory, thereby enhancing short-term memory.

[0110] (1) Initialization:

[0111] Compressed long-term memory M L With short-term memory M S As input, initialize and generate the query vector Q. g and future query vector Q F At the same time, short-term memory sets and future prediction set Initialize to empty.

[0112] (2) Generate future predictions:

[0113] First, long-term and short-term memories are cascaded in chronological order to form a memory matrix.

[0114] M A =[M L M S ]

[0115] Then, short-term memory M S Selective feature extraction is performed to obtain an enhanced short-term memory representation.

[0116] M S′ =Mamba(M S )

[0117] and combine it with the initial query Q g Cascade to form future query vectors

[0118]

[0119] Next, the cross-attention mechanism is used to calculate the future prediction representation.

[0120] F A =CrossAttn(Q F′ M A M A )

[0121] (3) Cyclic Update:

[0122] For subsequent loop decoding iterations, the following update strategy is adopted:

[0123] Update short-term memory: Utilize the current memory matrix M A And the predicted result F A Short-term memory is updated through a cross-attention mechanism, calculated using the following formula:

[0124] M S =CrossAttn(M S ,F E ,F E )

[0125] Feature extraction and query update: For the updated M S′ The shared Mamba module is applied again to generate a new enhanced representation M. S′ And concatenate with the original query to update the future query vector Q F′ =[M S′ Q F ], and then recalculate the future prediction F A The formula is as follows:

[0126] F E =[ML M S ,F A ]

[0127] F A =CrossAttn(Q F′ ,F E ,F E )

[0128] Iterative output: After each iteration, the updated short-term memory M will be output. S And future prediction F A Add to set respectively and In, until the predetermined N is completed. t The loop iterates once.

[0129] Output result:

[0130] Finally, after N t After each iteration, the recurrent decoder outputs an updated short-term memory representation and future prediction information. These two outputs not only capture the dynamic features of the current frame but also semantically maintain the continuity between the current detection and future action prediction, thus providing a reliable contextual reference for subsequent online action detection.

[0131] S5, Model Training and Optimization, specifically involves the following steps:

[0132] This invention employs a customized loss function for global model training. Specifically, for each video frame t, a loss function is defined. as follows:

[0133]

[0134] in, This indicates that the current frame is classified as background. γ represents the predicted probability of the i-th class at time t. a With γ b This design incorporates hyperparameters to control the penalty level for hard samples. It enables the model to impose higher penalties on low-confidence samples, effectively suppressing background noise and mitigating class imbalance caused by the significantly larger number of background samples compared to action samples.

[0135] For the two main branches of the model—the classification branch and the probability branch—the losses are calculated separately. In the initial stage, using the enhanced short-term memory and the output generated by future predictions, the action probabilities are obtained through a weighted classification head, resulting in the short-term memory classification loss. and future predicted losses

[0136]

[0137] Subsequently, for each iteration of the loop (1≤i≤N) t Define the corresponding memory loss. and anticipated losses Its calculation method is similar to that of the initial stage.

[0138] The overall training loss is expressed as a weighted sum and is defined as follows:

[0139]

[0140] Where, λ s and λ f These are the balance coefficients for short-term memory loss and future prediction loss, respectively, used to ensure that the model can optimize the fusion of local features and global temporal dependencies while taking into account the accuracy of current detection and future prediction.

[0141] (4) By minimizing the joint loss function mentioned above, the model not only improves the action recognition ability of the classification branch during training, but also optimizes the fusion effect of the probability branch on local features and global dependency information, thereby significantly improving the overall detection performance. In particular, since the number of background samples far exceeds that of action samples, different hyperparameter settings can better suppress background noise, prompting the model to pay more attention to high-confidence prediction of action classes.

[0142] S6, online motion detection, specifically involves the following steps:

[0143] (1) Construct a classification detection MLP layer for online action detection and action prediction tasks. The input of the detection head is the output generated by the last interaction of the loop decoder, and the output dimension is the number of actions (including background). The aim is to use MLP to compress the feature encoding in the complex temporal space into a low-dimensional action space.

[0144] (2) The detection performance of the model is calculated using mean accuracy (mAP) and calibrated mean accuracy (mcAP) as performance evaluation metrics. At the same time, the efficiency performance of the model is calculated using the number of parameters, GFLOPs (1 billion floating-point operations per second), and inference speed (Frames Per Second, FPS) as efficiency evaluation metrics.

[0145] Experiments on publicly available datasets demonstrate the performance of the proposed method across different datasets and modalities. In this embodiment, the THUMOS14, TVSeries, and HDD datasets were selected for experiments. Experimental results show that the method of this invention has high effectiveness and generalization ability in handling long-term sequence action detection tasks. To verify the detection performance and operational efficiency of this invention, a comparison with traditional online action detection methods is provided. Specific performance indicators are shown in Table 1, and efficiency indicators are shown in Table 2.

[0146] Table 1. Comparison of online motion detection performance between traditional online motion detection methods and the proposed method.

[0147]

[0148]

[0149] As shown in Table 1, the method of this invention outperforms traditional online action detection methods in both mAP and mcAP evaluation metrics, thus demonstrating higher detection accuracy. This invention not only solves the problems of insufficient background suppression and inadequate capture of temporal relationships but also endows the system with good generalization ability. Furthermore, Table 2 shows the model's effective ability to capture early action cues, proving the stability of the method in prediction tasks and enabling it to adapt to other real-world online applications.

[0150] Table 2 shows the action prediction results of this method on the THUMOS14 dataset using ActivityNet feature pre-extraction.

[0151]

[0152] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any other changes or substitutions that do not involve substantial innovation should be covered within the scope of protection of the present invention. Therefore, the specific scope of protection of the present invention should be determined by the scope of protection of the claims.

Claims

1. An online action detection method based on multi-stage memory mapping, characterized in that, Includes the following steps: S1, Feature Extraction: Frame-level features are extracted from the video stream using a pre-trained model to provide basic data for subsequent processing; S2, Memory partitioning: The extracted frame-level features are divided into long-term memory and short-term memory according to time, so as to handle global dependencies and local dynamics respectively; S3, Multi-level Mapping Memory Coding: A multi-stage mapping encoder is used for long-term memory, which is divided into segmented compression branches and global compression branches to compress and extract key information while suppressing irrelevant background. S4, Loop Decoding Update: A loop decoder is used to dynamically update the short-term memory and generate prediction information for future actions, ensuring semantic continuity between current detection and future prediction; S5, Model Training and Optimization: By designing an improved loss function, the fusion effect of local features and global dependencies in the model is optimized to improve the overall detection performance; S6, online action detection, performs real-time action detection based on memory features updated by loop decoding, and evaluates performance based on actual action labels; Step S3 specifically includes the following: (1) Construction of segmented compressed branches Long-term memory Divide into k non-overlapping subsequences Each subsequence The length is For each First, one-dimensional convolution is used to extract local continuous event features along the temporal direction; Subsequently, a set of learnable query vectors is introduced. ,in Where D is the subsequence length and D is the feature embedding dimension; and a Transformer decoder with shared weights is used to... Self-attention encoding is performed to focus on key local information. Finally, average pooling is applied to the encoding results to obtain an abstract representation of each subsequence. The concatenated representations of all subsequences are then input into the Mamba module to achieve selective information transfer and noise suppression, thereby generating segmented compressed memory. ; (2) Construction of global compression branch To capture the entire long-term memory The global temporal dependencies in the data are determined using a Transformer decoder based on grouped concatenated attention; the specific steps are: a. ... The block is divided into multiple sub-blocks, and self-attention is calculated for each attention head j, where j = 1, 2, ..., h. The calculation formula is as follows: in express Sub-features within the j-th group Here, Attn represents the corresponding projection matrix, and Concat represents the concatenation along the feature dimension. b. Employ a cascaded fusion strategy to progressively sum the outputs of each attention head: c. In cross-attention computation, an equalization factor α is introduced to modify the query vector to mitigate the impact of low-quality accumulated information, thereby achieving globally compressed memory. ; (3) Dual-branch feature fusion and final memory representation generation Compress and memorize segments containing key local information. Use this query to access the globally compressed memory that contains global dependencies in the upper branch. This process produces As a supplement to local information, the result is obtained by weighted summation followed by LayerNorm normalization. ,Will The input is fed into two consecutive Transformer encoder blocks to obtain the long-term memory for the final compression. , means as follows: This final representation incorporates both sensitive features of local continuous events and global temporal dependency information, providing a high-quality historical information reference for subsequent short-term memory enhancement and online action detection.

2. The online action detection method based on multi-stage memory mapping according to claim 1, characterized in that, Step S1 specifically includes the following: Real-time video streams are used as input data, and every N frames of video are processed as a group. A pre-trained two-dimensional convolutional neural network is used to extract the RGB image features and optical flow features of each frame in the group, converting each frame into a corresponding feature representation. ,in, The two-dimensional convolutional neural network represents the feature dimension; it is used to simultaneously capture the spatial and temporal information of video frames, enabling efficient recognition of dynamic changes in video content.

3. The online action detection method based on multi-stage memory mapping according to claim 2, characterized in that, Step S2 specifically includes the following: The extracted features are further continuously divided into subsets containing different temporal fine-grained sets: long-term memory and short-term memory. Long-term memory stores features far removed from the current time point and is defined as follows: Short-term memory is used to store a small number of recently occurring features, defined as... ,in Represents a single time frame. Indicates the length of short-term memory. This represents the total duration of the input model; the length of long-term memory is much greater than the length of short-term memory. According to the streaming structure of video, as time goes by, frames in short-term memory will gradually enter long-term memory, and short-term memory will also be dynamically updated over time.

4. The online action detection method based on multi-stage memory mapping according to claim 3, characterized in that, Step S4 specifically includes the following: The Transformer decoder module is used to inject compressed long-term memory into short-term memory; short-term memory is used as a query to extract local event-sensitive information and stable long-term time dependencies from historical memory, thereby enhancing short-term memory. (1) Initialization: Initialize the compressed long-term memory With short-term memory As input, initialize and generate the query vector. and future query vectors At the same time, short-term memory sets and future prediction set Initialize to empty; (2) Generating future predictions: First, long-term memory and short-term memory are cascaded in chronological order to form a memory matrix. Then short-term memory Selective feature extraction is performed to obtain an enhanced short-term memory representation. and combine it with the initial query Cascade to form future query vectors Next, the cross-attention mechanism is used to calculate the future prediction representation. (3) Loop Update: For subsequent loop decoding iterations, the following update strategy is adopted: Update short-term memory: Utilize the current memory matrix and prediction results Short-term memory is updated through a cross-attention mechanism, calculated using the following formula: Feature extraction and query update: For the updated... The shared Mamba module is applied again to generate a new enhanced representation. And concatenate with the original query to update the future query vector. Therefore, future predictions are recalculated. The formula is as follows: Iterative output: After each iteration, the updated short-term memory will be output. and future predictions Add to set respectively and In, until the scheduled completion. The loop iteration; Output: Finally, after... After each iteration, the recurrent decoder outputs an updated short-term memory representation and future prediction information. These two outputs not only capture the dynamic features of the current frame, but also maintain the semantic continuity between the current detection and the future action prediction, thus providing a reliable contextual reference for subsequent online action detection.

5. The online action detection method based on multi-stage memory mapping according to claim 4, characterized in that, Step S5 specifically includes the following: (1) A customized loss function is used to train the model globally; specifically, for each video frame t, a loss function is defined. as follows: in, This indicates that the current frame is classified as background. This represents the model's predicted probability for class i in video frame t. and The hyperparameters are designed to control the penalty intensity for hard samples, enabling the model to impose higher penalties on low-confidence samples, effectively suppressing background noise, and mitigating the class imbalance problem caused by the fact that there are far more background samples than action samples. (2) Calculate the loss for the classification branch and the probability branch in the model respectively; in the initial stage, use the enhanced short-term memory and the output generated by future prediction to obtain the action probability through the classification head with shared weights, and obtain the short-term memory classification loss respectively. and future predicted losses ; in, and These represent the length of short-term memory and the length of the generated future prediction, respectively. Subsequently, for each iteration Define the corresponding memory loss and anticipated losses The calculation method is the same as that in the initial stage; (3) The overall training loss is expressed as a weighted summation, defined as follows: in, and These are the balance coefficients for short-term memory loss and future prediction loss, respectively, used to ensure that the model can optimize the fusion of local features and global temporal dependencies while taking into account the accuracy of current detection and future prediction. (4) By minimizing the above joint loss function.

Images