Time sequence action detection method and system for frequency adaptive time sequence fusion
By combining low-frequency, mid-frequency, and high-frequency feature extraction modules with a gating fusion module, the problem of poor action recognition performance in existing technologies is solved, and adaptive modeling and efficient detection of multi-frequency components are achieved.
Patent Information
- Application Number
- CN202511610413.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-05
- Publication Date
- 2026-01-06
AI Technical Summary
Existing time series modeling methods struggle to balance long-range dependencies, regular rhythms, and boundary/transient changes, resulting in poor action recognition performance, especially in actions with complex multi-frequency components.
Low-frequency, medium-frequency, and high-frequency feature extraction modules are used to extract information at different time scales, and feature fusion is performed through a gating fusion module. Adaptive allocation is achieved using prototype routing units and differentiable fusion units to construct a frequency-adaptive temporal action detection model.
It enhances the modeling capabilities and interface compatibility of multi-band time-series signals, improves the accuracy and efficiency of action detection, and enhances training convergence and modular reusability.
Smart Images
Figure CN121280971A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of motion detection technology, specifically relating to a frequency-adaptive temporal fusion method and system for temporal motion detection. Background Technology
[0002] With the development of deep learning technology, action detection has become an important research direction in the field of video understanding. Traditional action recognition methods typically assume that actions are brief and complete in time, thus classifying them within fixed-length video clips. However, in real-world applications, actions often exhibit significant temporal extension and structural complexity; for example, actions like "long jump" and "waving" involve multiple stages: start, duration, and end. To accurately identify such actions, it is necessary to model the temporal dependencies between frame-level features in the video, that is, to establish an effective description of the evolution of actions over time. This learning of information associations in the temporal dimension is called "temporal modeling."
[0003] Existing time series modeling methods mostly employ a single operator backbone (such as fixed kernel convolution or single-path sequence operators) to extract information at different time scales. This type of structure often struggles to simultaneously address slow evolutionary trends, regular rhythms, and multi-frequency components such as boundary / transient states: (1) parameter and computational efficiency are limited under ultra-large receptive fields, hindering the capture of long-range dependencies; (2) single-scale or single-path operators are insufficient in responding to boundary and transient changes; (3) differences between different samples in the optimal processing frequency band cannot be adaptively matched using a unified operator. Therefore, it is necessary to design a structure capable of sample-level, adaptive allocation and fusion of low / medium / high-frequency information to achieve interpretable and efficient time series feature modeling. Summary of the Invention
[0004] The purpose of this invention is to provide a timing action detection method and system based on frequency adaptive timing fusion.
[0005] In a first aspect, the present invention provides a time-series action detection method based on frequency adaptive timing fusion, the method comprising:
[0006] Acquire videos containing different types of actions and construct a dataset based on the acquired videos; use a video feature extraction model to extract features from the videos in the dataset to obtain their corresponding sequence features;
[0007] A temporal action detection model is constructed. This model includes parallel low-frequency feature extraction modules, mid-frequency feature extraction modules, high-frequency feature extraction modules, a gating fusion module, and a detection head. In the temporal action detection model, sequence features are extracted using the low-frequency, mid-frequency, and high-frequency feature extraction modules, respectively. The extracted features are then input into the gating fusion module for fusion to obtain gating fused features. The detection head processes the gating fused features to obtain the detection result of the temporal action detection model.
[0008] The dataset was used to train the temporal action detection model, and the trained temporal action detection model was used to detect actions in the test video.
[0009] Preferably, the gated fusion module includes an intermediate frequency gated unit, a prototype routing unit, and a differentiable fusion unit; the intermediate frequency gated unit processes the sequence features using a linear layer, performs a global averaging operation on the input sequence features, and then inputs them into the linear layer to obtain intermediate frequency gated features;
[0010] In the prototype routing unit, the low-frequency sequence features and high-frequency sequence features are subjected to global averaging, and the processing results are compared with the first prototype vector and the second prototype vector to calculate cosine similarity, resulting in the first similarity vector and the second similarity vector. The first similarity vector and the second similarity vector are normalized using activation functions to obtain the first weight allocation coefficient and the second weight allocation coefficient. The first weight allocation coefficient and the second weight allocation coefficient are then weighted with the mid-frequency gated features to obtain the low-frequency gated features and the high-frequency gated features.
[0011] The differentiable fusion unit is used to fuse gating features and sequence features.
[0012] Preferably, in the differentiable fusion unit, the low-frequency gated features, mid-frequency gated features, and high-frequency gated features are multiplied by the low-frequency sequence features, mid-frequency sequence features, and high-frequency sequence features, respectively, to obtain the low-frequency gated sequence features, mid-frequency gated sequence features, and high-frequency gated sequence features; the low-frequency gated sequence features, mid-frequency gated sequence features, and high-frequency gated sequence features are added together to obtain the output of the differentiable fusion unit, which is then used as the output of the gated fusion module.
[0013] Preferably, the low-frequency feature extraction module includes a state-space model layer, a first linear layer, a layer normalization layer, and a second linear layer. In the low-frequency feature extraction module, the sequence features are processed sequentially through the state-space model layer and the first linear layer to obtain low-frequency intermediate features. The low-frequency intermediate features are then processed sequentially through the layer normalization layer and the second linear layer, and the processing result is added to the low-frequency intermediate features to obtain the low-frequency sequence features output by the low-frequency feature extraction module.
[0014] Preferably, the mid-frequency feature extraction module includes a series of layer normalization layers and a combined convolution module; the combined convolution module includes multiple convolutional layers in parallel; the output of the combined convolution module is the result of adding the output feature maps of the multiple convolutional layers.
[0015] In the mid-frequency feature extraction module, the sequence features are processed sequentially through a layer normalization layer and a combined convolution module, and the processing result is added to the sequence features to obtain the mid-frequency sequence features.
[0016] Preferably, the high-frequency feature extraction module uses a max pooling layer to process the sequence features to obtain high-frequency sequence features.
[0017] Preferably, the video feature extraction model uses the Inflated 3D ConvNet model.
[0018] Secondly, the present invention provides a frequency-adaptive temporal fusion temporal action detection system for performing the aforementioned temporal action detection method; the temporal action detection system includes a data acquisition module, a video feature extraction module, and a temporal action detection module; the data acquisition module is used to acquire videos containing different action types; the video feature extraction module is used to extract sequence features from the videos; and the temporal action detection module is used to detect the action type in the tested video based on the sequence features.
[0019] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the memory stores the computer program; and the processor executes the aforementioned timing action detection method.
[0020] Fourthly, the present invention provides a readable storage medium storing a computer program; when the computer program is executed by a processor, it is used to implement the above-described timing action detection method.
[0021] The beneficial effects of this invention are:
[0022] 1. This invention extracts sequence features through a low-frequency feature extraction module, a mid-frequency feature extraction module, and a high-frequency feature extraction module, focusing on low-frequency (long-range / gradually varying), mid-frequency (regular / multi-scale), and high-frequency (boundary / transient) information, respectively. Specifically, the low-frequency feature extraction module uses long-kernel convolutions parameterized by a structured state-space model to cover long-range dependencies across segments, taking into account both numerical stability and effective receptive field; the mid-frequency feature extraction module aggregates regular rhythms and multi-scale patterns through multi-scale convolution branches to form a robust representation of the main temporal structure; and the high-frequency feature extraction module strengthens boundary and transient changes through pooling, thereby improving the response to sudden signals at a fine-grained level.
[0023] 2. This invention fuses features extracted by the low-frequency, mid-frequency, and high-frequency feature extraction modules through a gated fusion module, achieving interpretable allocation and fusion of components from different frequency bands at the sample level. This enhances the modeling capability and interface compatibility for multi-frequency time-series signals while ensuring structural interpretability. Furthermore, this invention introduces a prototype routing unit into the gated fusion module. By adjusting the scaling factor through the routing module and supplementing it with orthogonalization constraints on prototype vectors, competition between feature extraction modules is reduced, training convergence is improved, and modular reuse is facilitated. Attached Figure Description
[0024] Figure 1 This is the timing action detection model in this invention.
[0025] Figure 2 This is a schematic diagram of the low-frequency feature extraction module in this invention.
[0026] Figure 3 This is a schematic diagram of the mid-frequency feature extraction module in this invention.
[0027] Figure 4 This is a schematic diagram of the high-frequency feature extraction module in this invention.
[0028] Figure 5 This is a schematic diagram of the gating fusion module in this invention. Detailed Implementation
[0029] The present invention will be further described below with reference to the accompanying drawings.
[0030] A frequency-adaptive temporal fusion-based temporal action detection method employs a temporal action detection system comprising a data acquisition module, a video feature extraction module, and a temporal action detection module. The data acquisition module is used to acquire videos containing different action types; the video feature extraction module is used to extract sequence features from the videos; and the temporal action detection module is used to detect the action types in the tested video based on the sequence features.
[0031] The timing action detection method includes the following steps:
[0032] Step 1: Building the dataset
[0033] The publicly available THUMOS14 dataset was selected as the benchmark data source for temporal action detection. This dataset consists of undisturbed online videos covering 20 target action categories. The validation set and test set contain approximately 200 and 213 videos with precise start and end time annotations, respectively. The original video frame rate is mainly 25 frames per second, and the duration of a single video segment ranges from tens of seconds to several minutes.
[0034] Step 2: Data Preprocessing
[0035] A video feature extraction model is used to extract features from videos containing different action types in the dataset. Specifically, 16 frames are used as a time slice, and a sliding window with an 8-frame step size is used to perform forward inference on the entire video window by window to obtain the final sequence features of size T×C, and the sequence features are added to the dataset; where T is the number of original video frames / 8; and C represents the feature channel dimension of each time slice, which has a value of 2048.
[0036] In this embodiment, the video feature extraction model adopts the Inflated 3D ConvNet (I3D) model.
[0037] Step 3: Construct a temporal action detection model
[0038] like Figure 1 As shown, the temporal action detection model includes parallel low-frequency feature extraction modules, mid-frequency feature extraction modules, high-frequency feature extraction modules, a gating fusion module, and a detection head. Feature extraction is performed on the sequence using the low-frequency, mid-frequency, and high-frequency feature extraction modules respectively. The extracted features are then input into the gating fusion module for fusion to obtain gated fused features. The detection head processes these gated fused features to obtain the detection result of the temporal action detection model.
[0039] like Figure 2 As shown, the low-frequency feature extraction module includes a state-space model layer, a first linear layer, a layer normalization layer, and a second linear layer. In the low-frequency feature extraction module, the sequence features are processed sequentially through the state-space model layer and the first linear layer to obtain low-frequency intermediate features; the low-frequency intermediate features are then processed sequentially through the layer normalization layer and the second linear layer, and the processing results are added to the low-frequency intermediate features to obtain the low-frequency sequence features output by the low-frequency feature extraction module.
[0040] like Figure 3 As shown, the mid-frequency feature extraction module includes a series of layer normalization layers and a combined convolution module. The combined convolution module consists of a first convolutional layer, a second convolutional layer, and a third convolutional layer connected in parallel. The output of the combined convolution module is the sum of the output feature maps of the three convolutional layers. The first convolutional layer has a kernel size of 1, a stride of 1, and padding of 0; the second convolutional layer has a kernel size of 3, a stride of 1, and padding of 1; and the third convolutional layer has a kernel size of 5, a stride of 1, and padding of 2. In the mid-frequency feature extraction module, the sequence features are processed sequentially through the layer normalization layer and the combined convolution module, and the processed result is added to the sequence features to obtain the mid-frequency sequence features.
[0041] like Figure 4As shown, the high-frequency feature extraction module uses a one-dimensional max pooling layer with a kernel size of 3, a stride of 1, and padding of 1 to process the sequence features and obtain high-frequency sequence features.
[0042] like Figure 5 As shown, the gated fusion module includes an intermediate frequency (IF) gate unit, a prototype routing unit, and a differentiable fusion unit. The IF gate unit processes the sequence features using a linear layer. The input sequence features are global averaged along the time dimension and then input into the linear layer to obtain the IF gated feature α.
[0043] In the prototype routing unit, the low-frequency sequence features and high-frequency sequence features are globally averaged along the time dimension, and the results are then used to calculate cosine similarity with the first prototype vector and the second prototype vector, respectively, to obtain the first similarity vector and the second similarity vector. The first similarity vector and the second similarity vector are then normalized using the softmax function to obtain the first weight allocation coefficient β and the second weight allocation coefficient γ. The first weight allocation coefficient β and the second weight allocation coefficient γ are then weighted with the mid-frequency gated feature α to obtain the low-frequency gated feature (1-a)·β and the high-frequency gated feature (1-a)·γ.
[0044] In the differentiable fusion unit, the low-frequency gated features, mid-frequency gated features, and high-frequency gated features are multiplied by the low-frequency sequence features, mid-frequency sequence features, and high-frequency sequence features, respectively, to obtain the low-frequency gated sequence features, mid-frequency gated sequence features, and high-frequency gated sequence features. The output of the differentiable fusion unit is obtained by adding the low-frequency gated sequence features, mid-frequency gated sequence features, and high-frequency gated sequence features, and this output is used as the output of the gated fusion module.
[0045] The detection head consists of two linear layers, which are used to output the confidence scores for each category and the time boundaries corresponding to the actions, respectively. The output results are the final generated detection results.
[0046] Step 4: Train the temporal action detection model using the dataset, and then use the trained temporal action detection model to detect actions in the test video.
[0047] Step 5: Model Evaluation
[0048] Action detection experiments were conducted on the THUMOS14 dataset using the present invention and the existing action detection model ActionFormer, respectively. The detection results are shown in Table 1.
[0049] Table 1. Comparison of detection results of different action detection models
[0050] Action detection model mAP (%) ActionFormer 67.9 This invention 69.8
[0051] As can be seen from Table 1, the present invention outperforms existing action detection methods on the THUMOS14 dataset, indicating that the present invention has excellent performance in temporal action detection.
Claims
1. A frequency adaptive temporal action detection method, characterized in that: The method comprises: acquiring videos containing different action types, and constructing a dataset based on the acquired videos; using a video feature extraction model to extract features of the videos in the dataset, and acquiring corresponding sequence features thereof; constructing a time sequence action detection model; the time sequence action detection model comprises parallel low-frequency feature extraction modules, medium-frequency feature extraction modules, high-frequency feature extraction modules, a gating fusion module, and a detection head; in the time sequence action detection model, the sequence features are extracted by the low-frequency feature extraction modules, the medium-frequency feature extraction modules, and the high-frequency feature extraction modules respectively, and the extracted features are input into the gating fusion module for fusion to obtain gating fusion features; the detection head is used to process the gating fusion features to obtain a detection result of the time sequence action detection model; the time sequence action detection model is trained using the dataset, and the trained time sequence action detection model is used to detect actions of a video to be detected.
2. The method of claim 1, wherein the method further comprises: The gating fusion module comprises a medium-frequency gating unit, a prototype routing unit, and a differentiable fusion unit; the medium-frequency gating unit adopts a linear layer to process the sequence features, and the sequence features after global average operation are input into the linear layer to obtain medium-frequency gating features; in the prototype routing unit, the low-frequency sequence features and the high-frequency sequence features are subjected to global average operation respectively, and the processing results are subjected to cosine similarity calculation with a first prototype vector and a second prototype vector respectively to obtain a first similarity vector and a second similarity vector; the first similarity vector and the second similarity vector are subjected to normalization processing by using an activation function respectively to obtain a first weight distribution coefficient and a second weight distribution coefficient; the first weight distribution coefficient and the second weight distribution coefficient are subjected to weight distribution with the medium-frequency gating features respectively to obtain low-frequency gating features and high-frequency gating features; the differentiable fusion unit is used to fuse the gating features and the sequence features.
3. The method of claim 2, wherein the method further comprises: In the differentiable fusion unit, the low-frequency gating features, the medium-frequency gating features, and the high-frequency gating features are multiplied with the low-frequency sequence features, the medium-frequency sequence features, and the high-frequency sequence features respectively to obtain low-frequency gating sequence features, medium-frequency gating sequence features, and high-frequency gating sequence features; the low-frequency gating sequence features, the medium-frequency gating sequence features, and the high-frequency gating sequence features are added to obtain an output of the differentiable fusion unit, and the output is taken as an output of the gating fusion module.
4. The method of claim 1, wherein the method further comprises: The low-frequency feature extraction module comprises a state space model layer, a first linear layer, a layer normalization layer, and a second linear layer; in the low-frequency feature extraction module, the sequence features are processed by the state space model layer and the first linear layer in sequence to obtain low-frequency intermediate features; the low-frequency intermediate features are processed by the layer normalization layer and the second linear layer in sequence, and the processing results are added to the low-frequency intermediate features to obtain low-frequency sequence features output by the low-frequency feature extraction module.
5. The method of claim 1, wherein: The medium-frequency feature extraction module comprises a layer normalization layer and a combined convolution module connected in series; the combined convolution module comprises multiple convolution layers connected in parallel; an output feature map of the multiple convolution layers is added to obtain an output of the combined convolution module; In the intermediate frequency feature extraction module, the sequence feature is processed through a layer normalization layer and a combined convolution module in sequence, and the processing result is added to the sequence feature to obtain an intermediate frequency sequence feature.
6. The method of claim 1, wherein: The high frequency feature extraction module adopts a max pooling layer to process the sequence feature to obtain a high frequency sequence feature.
7. The method of claim 1, wherein the method further comprises: The video feature extraction model adopts an Inflated 3D ConvNet model.
8. A frequency adaptive temporal fusion temporal action detection system, characterized by: The time action detection method for performing the frequency adaptive time sequence fusion of claim 1; the time action detection system comprises a data acquisition module, a video feature extraction module and a time action detection module; the data acquisition module is used for acquiring a video containing different action types; the video feature extraction module is used for extracting sequence features from the video; and the time action detection module is used for detecting the action type in the measured video according to the sequence features.
9. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that: The memory stores a computer program; and the processor executes the time action detection method in any one of claims 1-7.
10. A readable storage medium, storing a computer program; characterized in that: The computer program is executed by the processor to implement the time action detection method in any one of claims 1-7.