A time series action detection method based on a collaborative perception pyramid network

Abstract

This invention discloses a temporal action detection method based on a collaborative perceptual pyramid network, belonging to the field of temporal action detection. The method extracts the feature sequence of the video to be detected and inputs it into a collaborative perceptual pyramid network. This network has a multi-layer pyramid structure, with each layer sequentially using a temporal decomposition collaborative perceptual module and a dual-path collaborative perceptual module to process the features. These features are then aggregated by a dynamic feature aggregation module to form the final features. Based on the final features, the action category is simultaneously identified and the action temporal boundary is located. Specifically, the temporal decomposition collaborative perceptual module captures features through interval-level and instantaneous-level branches, and adaptively fuses the branch weights to output fused features. The dual-path collaborative perceptual module employs depthwise separable convolution and max-pooling dual-path downsampling, and adaptively fuses the path weights to output optimized features. This invention, by combining a collaborative perceptual pyramid network with an adaptive fusion mechanism, achieves a simultaneous improvement in the accuracy of action boundary localization and category recognition in undressed videos.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of time series action detection, and particularly relates to a time series action detection method based on a collaborative perception pyramid network. BACKGROUND

[0002] Time series action detection is an important research direction in the field of computer vision and video analysis, and its core goal is to locate the time boundary (including the starting and ending time) of the action and identify the corresponding action category in the unpruned video. The current mainstream method is to integrate proposal generation, classification and boundary regression into a unified algorithm architecture to realize efficient collaborative processing of time action positioning and action recognition, thereby improving the automation level of video understanding. However, the existing scheme still has limitations in boundary accuracy and generalization ability.

[0003] In the prior art scheme, the query-based framework, the anchor-based detector and the anchor-free paradigm are three types of mainstream methods. The query-based framework is inspired by Carion et al. in the field of target detection, which predicts action instances through instance-level queries, but the boundary positioning effect is not ideal. Zhu et al. proposed a double query framework, which integrates instance-level queries for action recognition and boundary-level queries for accurate time positioning. Subsequently, Kim et al. introduced a Transformer-based architecture that combines time-aligned coordinate representation to realize overall end-to-end detection.

[0004] The anchor-based detector uses pre-defined anchor boxes to generate action proposals. For example, Lin et al. applied a single multi-box detector framework to the time series action detection task, which directly predicts action categories and boundaries by deploying multi-scale anchors on the time axis. These manually designed anchor boxes lack flexibility in action instance positioning, hindering the diversity of action instances. This method itself has limitations and requires additional post-processing steps to discard redundant proposal boxes.

[0005] The anchorless paradigm abandons predefined anchors, directly predicting boundaries and categories. Ma et al. pioneered this paradigm, utilizing salient boundary pooling to aggregate boundary features and enforcing consistency constraints for fine-grained localization. Subsequently, driven by anchorless architectures and Transformer-based models, this paradigm became the mainstream research direction. For example, Zhang et al. combined a Transformer-based backbone network with an anchorless prediction head network, achieving optimal performance by directly regressing action temporal boundaries and categories. However, the computational cost of Transformers remains an issue. Tang et al. balanced efficiency and contextual modeling using a parameterless max-pooling module; Shi et al. modeled boundaries by estimating the relative probability distribution around them using a classifier-regressor; and Yang et al. enhanced multi-scale detection by dynamically adjusting convolutional kernel weights and receptive fields using an adaptive encoder. Liu et al. proposed a temporal information adapter to reduce training memory overhead while increasing input frame capacity, thus significantly improving detection performance. These methods have made progress in specific scenarios, but none have completely solved the boundary ambiguity problem, and they generally suffer from inaccurate boundary prediction, low computational efficiency, or insufficient generalization ability.

[0006] Therefore, there is an urgent need in this field for a novel temporal action detection method that can significantly improve action detection accuracy and temporal localization capabilities while enhancing the model's versatility and practicality, in order to overcome the core challenge posed by boundary ambiguity. Summary of the Invention

[0007] To address the aforementioned issues, this invention proposes a temporal action detection method based on a collaborative perceptual pyramid network. By combining a collaborative perceptual pyramid network with an adaptive fusion mechanism, the method achieves simultaneous improvement in the accuracy of action boundary localization and category recognition in untrimmed videos, exhibiting high computational efficiency and strong generalization ability.

[0008] The technical solution adopted in this invention is as follows:

[0009] In a first aspect, this invention proposes a temporal action detection method based on a cooperative perceptual pyramid network, comprising:

[0010] Extract the video feature sequence from the video to be detected;

[0011] The video feature sequence is processed using a collaborative perception pyramid network model. The collaborative perception pyramid network is a multi-layer pyramid structure. Each layer sequentially uses a temporal decomposition collaborative perception module and a dual-path collaborative perception module to process features, and the optimized features output by the previous layer are used as the input features of the next layer. The optimized features output by each pyramid layer are aggregated into the final features by a dynamic feature aggregation module. Based on the final features, the action category is identified and the action time boundary is located.

[0012] In each pyramid structure, the temporal decomposition collaborative perception module captures time-interval and instantaneous features from the input features through interval-level and instantaneous-level branches, respectively. It dynamically learns the fusion weights across branches using an adaptive parameter fusion mechanism and superimposes the original features from the corresponding layer to output fused features. The dual-path collaborative perception module adopts a dual-path downsampling strategy, processes the fused features through depthwise separable convolutional paths and max-pooling paths, respectively, and dynamically learns the fusion weights across paths using an adaptive parameter fusion mechanism to output optimized features.

[0013] Furthermore, the collaborative perception pyramid network model is trained using a predefined dataset, in which video clip samples are labeled with action time boundaries and action category labels. The action time boundaries include the start and end times of the action.

[0014] Furthermore, the total training loss includes action classification loss and temporal regression loss, wherein the temporal regression loss is a weighted value of the overlap between the predicted action temporal boundary and the actual action temporal boundary and the distance between the center points.

[0015] Furthermore, the action categories identified based on the final features are multiple predefined categories in the predefined dataset, including action categories for ball sports, track and field events, water / air sports, and combat sports. This can be used to provide professional athletes and coaching teams with automated analysis tools; training or competition videos serve as the videos to be detected, allowing for the rapid location, statistical analysis, and categorization of specific technical movements from these videos.

[0016] Furthermore, in the temporal decomposition collaborative perception module, the interval-level branch uses convolution operations and a fully connected network to capture time interval-level features, while the instantaneous-level branch uses pooling operations and a fully connected network to capture instantaneous-level features. The adaptive parameter fusion mechanism uses a trainable weight parameter to weight the output features of the interval-level branch and the output features of the instantaneous-level branch, and then adds the result to the original input features of the temporal decomposition collaborative perception module to output fused features.

[0017] Specifically, the interval-level branch calculation process is as follows: Three different convolution operations are performed on the input features to obtain the first convolutional feature, the second convolutional feature, and the third convolutional feature; the first convolutional feature and the second convolutional feature are multiplied by a dot product; the dot product result is input into the first fully connected layer and then activated by the first hyperbolic tangent activation function; the activation result is input into the second fully connected layer, and the output is scaled by dividing by the square root of the feature dimension and then activated by the second hyperbolic tangent activation function; the result of the second activation is multiplied by the third convolutional feature to generate the output feature of the interval-level branch.

[0018] The instantaneous branch calculation process is as follows: global average pooling is performed on the input features; the pooling result is input into the fourth fully connected layer and then activated by the hyperbolic tangent activation function; the input features are input into the third fully connected layer in parallel; the output of the activated fourth fully connected layer is multiplied by the output of the third fully connected layer to generate the output features of the instantaneous branch.

[0019] Furthermore, in the dual-path collaborative perception module, the depthwise separable convolutional path fuses features through convolution and downsampling, the max pooling path directly fuses features through pooling operations, and the adaptive parameter fusion mechanism uses a trainable weight parameter to weight the final output features of the depthwise separable convolutional path and the final output features of the max pooling path to output optimized features.

[0020] Specifically, the calculation process of the depthwise separable convolutional path is as follows: perform the fifth fully connected operation on the input fused features; perform a convolution operation on the features output by the fifth fully connected operation; activate the features output by the convolution operation using the GeLU activation function; and perform the sixth fully connected operation on the activated features to generate the output features of the depthwise separable convolutional path.

[0021] Furthermore, the aggregation process of the dynamic feature aggregation module includes:

[0022] Receives the optimized feature set output from each layer of the multi-layer pyramid structure;

[0023] For the set of the th Each layer optimizes its features by downsampling its adjacent features from the previous layer and upsampling its adjacent features from the next layer, so that the processed features of adjacent layers are similar to those of the first layer. The optimized features of each layer have the same dimension;

[0024] The downsampled optimized features of the previous layer are summed with the upsampled optimized features of the next layer. The sum is then compared with the first optimized feature. The optimized features of each layer are multiplied by their corresponding trainable weight parameters and then summed to output the final features.

[0025] Secondly, this invention proposes a temporal action detection system based on a cooperative sensing pyramid network, which is used to implement the aforementioned temporal action detection method based on a cooperative sensing pyramid network.

[0026] Thirdly, the present invention proposes a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the above-mentioned temporal action detection method based on a cooperative perception pyramid network.

[0027] Fourthly, the present invention proposes a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the above-described temporal action detection method based on a cooperative sensing pyramid network.

[0028] The beneficial effects of this invention are:

[0029] This invention proposes a novel collaborative perception pyramid network model for temporal action detection, capable of simultaneously and accurately locating the start and end boundaries of actions and identifying the corresponding action categories in unedited videos. The temporal decomposition collaborative perception module achieves multi-level feature capture through a dual-path structure of interval-level and instantaneous-level branches. The interval-level branches extract long-range temporal dependencies, while the instantaneous-level branches enhance local discriminativity. An adaptive parameter fusion mechanism dynamically learns cross-branch weights, effectively addressing the ambiguity of action boundaries. The dual-path collaborative perception module employs a dual downsampling strategy: a depthwise separable convolutional path preserves feature details, while a max-pooling path filters redundant information. Similarly, an adaptive fusion mechanism balances the outputs of the two paths, resulting in complementary feature optimization. This overcomes the limitations of traditional single-path perception and replaces static rules with adaptive fusion, enabling end-to-end optimization of the entire network. While maintaining computational efficiency, it significantly improves semantic discrimination and boundary localization capabilities, solving the problems of inaccurate action classification and localization in existing temporal action detection models. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the training process of the collaborative sensing pyramid network proposed in this invention.

[0031] Figure 2 This is a schematic diagram of the overall architecture of the collaborative sensing pyramid network provided by the present invention.

[0032] Figure 3 This invention relates to the visualization of data results applied to the THUMOS14 dataset.

[0033] Figure 4 This is a schematic diagram of the time-series action detection system based on a collaborative sensing pyramid network, as presented in this invention. Detailed Implementation

[0034] The present invention will be further described and illustrated below with reference to specific embodiments. The embodiments described are merely examples of the content of this disclosure and do not limit the scope of the invention. The technical features of each embodiment in the present invention can be combined accordingly, provided that there is no mutual conflict.

[0035] The accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. Some of the block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0036] The flowchart shown in the attached diagram is merely an illustrative example and does not necessarily include all steps. For example, some steps may be broken down, while others may be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0037] like Figure 1 As shown, this invention proposes a temporal action detection method based on a collaborative perception pyramid network, which mainly includes preprocessing and feature extraction of raw video data, construction of a collaborative perception pyramid network model, training the model using the backpropagation algorithm to learn model parameters, and temporal action detection.

[0038] The implementation process of this invention includes the following steps:

[0039] S1, Video Feature Extraction

[0040] First, a predefined dataset for model training is collected. The video clip samples in the predefined dataset are labeled with action time boundaries and action category labels. The action time boundaries include the start time and end time of the action.

[0041] Taking sports scenarios as an example, this method identifies the action categories and time boundaries of various sports. In this embodiment, the THUMOS14 dataset is selected as a predefined dataset, which contains 20 action categories, including: 1. Baseball Pitch; 2. Basketball Dunk; 3. Billiards; 4. Clean and Jerk; 5. Cliff Diving; 6. Cricket Bowling; 7. Cricket Shot; 8. Diving; 9. Frisbee Catch. 10. Golf Swing; 11. Hammer Throw; 12. High Jump; 13. Javelin Throw; 14. Long Jump; 15. Pole Vault; 16. Shotput; 17. Soccer Penalty; 18. Tennis Swing; 19. Throw Discus; 20. Volleyball Spiking. The training and test sets contain 200 and 213 video clips respectively, with 3,007 and 3,358 action instances respectively.

[0042] Given a video segment x, this invention extracts the feature sequence of the video segment. The dimension of the feature sequence is Where the sequence length T varies with the video length, and D is the feature dimension. The feature extractor can use a pre-trained InternVideo2 model. Additionally, preprocessing and data augmentation operations can be introduced into the original video to improve the generalization ability of the subsequent collaborative perceptual pyramid network and enhance its robustness to changes in action scale, position, and time in untrimmed videos. The preprocessing and data augmentation operations are existing technologies in this field and will not be described further in this invention.

[0043] S2, using a collaborative perceptual pyramid network model to process video feature sequences.

[0044] The collaborative perception pyramid network is a multi-layer pyramid structure. Each layer uses a temporal decomposition collaborative perception module and a dual-path collaborative perception module to process features in sequence, and the output of the previous layer serves as the input of the next layer.

[0045] like Figure 2As shown, the collaborative perception pyramid network model in this embodiment adopts a 7-layer pyramid structure (L=7). Each layer includes a temporal decomposition collaborative perception module, a dual-path collaborative perception machine module, and a dynamic feature aggregation module. The temporal decomposition collaborative perception module inputs the processed features to the dual-path collaborative perception machine module. The outputs of the dual-path collaborative perception machine modules at each layer are input to the dynamic feature aggregation module. The final aggregated features are used to simultaneously identify action categories and locate action time boundaries.

[0046] (2.1) Temporal Decomposition Collaborative Sensing Module

[0047] The aforementioned temporal decomposition collaborative perception module captures time-interval and instantaneous features from the input features through interval-level and instantaneous-level branches, respectively, and dynamically learns the cross-branch fusion weights using an adaptive parameter fusion mechanism to output fused features. The first... The input features of the layered pyramid structure are denoted as Then the first The computation process of the temporal decomposition collaborative sensing module of the layer includes the following sub-steps:

[0048] a. Interval-level branches utilize large convolutional kernels to capture temporal features, and dynamically focus on features at different times through auxiliary weighted branches; The process of implementing layer-level branching is described as follows:

[0049]

[0050] in, These are three independent convolutional layers with a kernel size of 7; It is a non-linear activation function. They are two independent, fully connected networks. This represents the dot product operation between corresponding elements of two matrices of the same size. For the first The layer's interval-level branch output features, which are related to Same dimensions.

[0051] b. The instantaneous branch enhances the discriminative power between action and non-action features by aggregating video-level average features. The instantaneous branching process of a layer is described as follows:

[0052]

[0053] in, It is a non-linear activation function. They are two independent, fully connected networks. For average pooling layer, This represents the dot product operation between corresponding elements of two matrices of the same size. For the first The instantaneous branch output characteristics of the layer, which are related to Same dimensions.

[0054] c. Maintaining the feature dimension unchanged in the adaptive fusion layer, where the output features of the interval-level branch and the instantaneous-level branch have the same feature dimension as the input features, this invention employs learnable weighted fusion to combine the first... The output features of the interval-level branches, the output features of the instantaneous branches, and the input features of the pyramid layer. The implementation process of the adaptive fusion layer is described as follows:

[0055]

[0056] Among them, the fusion weight It is the first The trainable parameters of each pyramid layer are optimized during training. It is the first The fused features output by the temporal decomposition and collaborative sensing module of the pyramid layer.

[0057] (2.2) Dual-path collaborative sensing module

[0058] The dual-path collaborative perception module adopts a dual-path downsampling strategy, which processes the fused features through depthwise separable convolutional paths and max pooling paths respectively, and uses an adaptive parameter fusion mechanism to dynamically learn the fusion weights across paths and output optimized features.

[0059] like Figure 2 As mentioned above, the first The input features of the dual-path collaborative sensing module of the layer are denoted as Then the first The computation process of the dual-path collaborative sensing module of the layer includes the following sub-steps:

[0060] a. Depthwise separable convolutional path: downsampling that preserves feature details.

[0061] This approach utilizes depthwise separable convolutions for downsampling, focusing on preserving detailed information of the input features. Depthwise separable convolutions significantly reduce the number of parameters by separating spatial and channel convolution operations, while maintaining the feature representation capability. The specific process is described below:

[0062]

[0063] in, It is a convolutional layer. It is a non-linear activation function. They are two independent, fully connected networks. For the first The depth of the layer pyramid allows for the separation of convolutional paths and output features. Input features (Output from the temporal decomposition and collaborative perception module) First, feature transformation is performed through a fully connected layer. Then, a convolutional layer is applied to extract local features in the temporal dimension. Next, a non-linear transformation is introduced through the GeLU activation function. Finally, a fully connected layer is used again to further restore the feature dimension, outputting the path features. This path effectively captures local temporal patterns through layered processing, while avoiding information loss caused by over-downsampling.

[0064] b. Max pooling path: Filters redundant information and retains discriminative features.

[0065] Another approach uses max pooling to filter out redundant information while retaining discriminative local features. Max pooling highlights significant features by selecting the maximum value within a time window; the specific process is described below:

[0066]

[0067] in, For maximum pooling layer, For the first The max-pooling path of the pyramid layer outputs features. Directly applying the input features... (Output from the temporal decomposition and collaborative perception module) Perform max pooling operation, downsample along the time dimension to reduce the feature dimension. This feature can focus on local key time segments, enhance action boundaries, and has high computational efficiency.

[0068] c. In the adaptive fusion layer, the feature dimension remains unchanged. The output features of the depthwise separable convolutional path and the max-pooling path have the same feature dimension. The features of the two paths are fused, and the fusion weights across the paths are dynamically learned to balance the trade-off between detail preservation and information filtering. The adaptive fusion mechanism uses trainable parameters... End-to-end optimization of weights is achieved, enabling the network to adaptively adjust path contributions based on input features. The specific process is described below:

[0069]

[0070] Among them, the fusion weight It is the first The trainable parameters of each pyramid layer are optimized during training. It is the first The optimized features after processing the pyramid layer, i.e., the first layer... Input features of a layered pyramid structure .

[0071] In one specific embodiment of the present invention, each layer receives the output features of the previous layer and processes them sequentially through a temporal decomposition collaborative sensing module and a dual-path collaborative sensing module. After iterative processing through a 7-layer pyramid structure, multi-scale optimized features are output.

[0072] (2.3) Dynamic Feature Aggregation Module

[0073] This module is used to aggregate multi-scale optimized features to obtain the final features, which are then used to identify action categories and locate action time boundaries based on the final features.

[0074] In this embodiment, the dynamic aggregation process is as follows:

[0075] Optimized feature sets from layers 1 to 7 of the pyramid The output features of layer 1 are derived from the output features of layer 1. The upsampled features of the output of the second layer are dynamically aggregated. The output features of the second layer are then dynamically aggregated with the upsampled features of the output of the third layer and the downsampled features of the output of the first layer, and so on.

[0076] The specific description is as follows:

[0077]

[0078] in, and For the first The trainable parameters of the layer, DOWN for downsampling and UP for upsampling, are used to make the input feature sequence equal to the downsampling parameters. Features with the same dimension are dynamically aggregated to obtain the final features, which are used for action category recognition and action time boundary localization.

[0079] In this embodiment, a classification head and a regression head are used to implement action category classification and action temporal boundary regression, respectively. The classification head can consist of a series of fully connected layers and a softmax activation function. It maps the feature vector of each time position to a C+1 dimensional probability distribution, where C is the predefined number of action categories, and +1 represents no action occurring. Each element in the probability distribution represents the confidence level that a certain time position belongs to a certain category. The regression head can also consist of fully connected layers, but it outputs the temporal boundary parameters of the action corresponding to each time position. Both the classification head and the regression head can be implemented using existing technologies.

[0080] S3, Training a collaborative perception pyramid network model based on a joint loss function.

[0081] The entire model is trained within an end-to-end supervised learning framework. Given a predefined dataset with full annotations, the training objective is to minimize the overall error between the model's predictions and the ground truth annotations.

[0082] In this embodiment, the model's total loss function is a multi-task joint loss, which simultaneously optimizes the accuracy of action classification and the localization precision of the temporal boundary. This loss function is defined as the weighted sum of the action classification loss and the temporal regression loss, and its mathematical expression is as follows:

[0083]

[0084] Where L is the total loss, It uses action classification loss, employing Focal Loss; It is a time regression loss, using DIoU Loss, which regresses the time boundary of the predicted action to make it coincide with the true boundary as much as possible. This is an improved IoU (Intersection over Union) loss, which can better handle the bounding box regression problem. These are preset hyperparameters used to balance the importance of classification and regression tasks; These are the predicted action category and the actual action category. These are the predicted time boundary and the actual time boundary, respectively.

[0085] Based on the loss result, the parameters of the deep network model are trained using the backpropagation algorithm until the entire network model converges.

[0086] S4 deploys the trained collaborative perception pyramid network model to new untrimmed videos for temporal action detection.

[0087] The complete application process is as follows:

[0088] First, the input video to be detected undergoes the same preprocessing as during the training phase, using the same pre-trained feature extractor to convert it into a video feature sequence.

[0089] Subsequently, this feature sequence is used as input for feature extraction in a pre-trained collaborative perception pyramid network. The network sequentially passes through a 7-layer pyramid structure, where the temporal decomposition collaborative perception module and the dual-path collaborative perception module in each layer process the features layer by layer. Finally, a dynamic feature aggregation module generates a unified final feature representation. Based on this final feature, the network's classification head and regression head output prediction results in parallel: the classification head outputs the probability distribution of each candidate time position belonging to each predefined action category; the regression head outputs the time boundary corresponding to each action.

[0090] Finally, post-processing steps such as confidence thresholding and non-maximum suppression can be introduced to filter out high-confidence prediction results, thereby obtaining all detected action instances in the video. Each instance is fully described by its precise start time, end time, and corresponding action category label. The entire process is efficient and automated, achieving end-to-end recognition and localization of actions in uncropped videos.

[0091] This invention was tested on a single NVIDIA GeForce RTX 3090 GPU. During training, the AdamW optimizer was used with an initial learning rate of 5e. -4 The learning rate is updated using a cosine annealing algorithm, and a linear warm-up strategy is introduced at the start of training to enhance the stability of model convergence. This invention rigorously evaluates model performance on the validation set. This invention reports the mean accuracy (mAP) at multiple intersection-over-union (IoU) thresholds to comprehensively evaluate localization accuracy. For the THUMOS14 dataset, this invention calculates the mean mAP over a continuous range of IoU thresholds from 0.3 to 0.7 (with a step size of 0.1), according to standard evaluation metrics for temporal action detection. This multi-threshold approach ensures a comprehensive analysis of the model's capabilities in both coarse-grained and fine-grained temporal localization tasks. As shown in Table 1, compared to other mainstream methods, this invention's method significantly improves action detection performance.

[0092] Table 1

[0093]

[0094] To further evaluate the performance of the method of this invention, the results were visualized on the THUMOS14 dataset. For example... Figure 3 As shown, the actual actions are marked by the first row of sliders, and the prediction results of the method of this invention are marked by the second row of sliders, with the corresponding prediction time boundaries marked below. These visualization results quantitatively confirm that the method of this invention is accurate and effective in action detection. Human behavior detection plays an important role in image or video-based analysis of human behavior. Accurate and efficient human behavior detection can promote various applications, and the proposed improvement method enhances the final detection performance of existing models.

[0095] This embodiment also provides a temporal action detection system based on a cooperative perceptual pyramid network for implementing the above-described temporal action detection method, such as... Figure 4 As shown, it includes:

[0096] The feature extraction module is used to extract the video feature sequence of the video to be detected;

[0097] A collaborative perception pyramid network module is connected to the feature extraction module. It is used to process the video feature sequence to obtain the final features, and then simultaneously identify the action category and locate the action time boundary based on the final features.

[0098] The collaborative sensing pyramid network is a multi-layer pyramid structure. Each layer uses a temporal decomposition collaborative sensing network and a dual-path collaborative sensing network to process features in sequence, and the optimized features output by the previous layer are used as the input features of the next layer. The optimized features output by each pyramid layer are aggregated into the final features through a dynamic feature aggregation module.

[0099] In each layer of the pyramid structure, the temporal decomposition collaborative perception network includes interval-level branch units and instantaneous-level branch units, which are used to capture time interval-level features and instantaneous-level features from the input features. An adaptive parameter fusion unit dynamically learns the fusion weights across branches and superimposes the original features of the corresponding layer input to output fused features. The dual-path collaborative perception network adopts a dual-path downsampling strategy, which processes the fused features through depthwise separable convolutional paths and max-pooling paths respectively, and uses an adaptive parameter fusion mechanism to dynamically learn the fusion weights across paths to output optimized features.

[0100] For the system embodiments, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments; the implementation methods of the modules will not be repeated here. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0101] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0102] It should also be noted that the temporal action detection method based on the cooperative perceptual pyramid network in the above embodiments can essentially be executed by a computer program. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer electronic device corresponding to the method provided in the above embodiments, which includes a memory and a processor;

[0103] The memory is used to store computer programs;

[0104] The processor is configured to implement the temporal action detection method based on a cooperative perceptual pyramid network in the above embodiments when executing the computer program.

[0105] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium.

[0106] Therefore, based on the same inventive concept, another preferred embodiment of the present invention also provides a computer-readable storage medium corresponding to the method provided in the above embodiments. The storage medium stores a computer program, which, when executed by a processor, can realize the temporal action detection method based on a cooperative sensing pyramid network in the above embodiments.

[0107] It is understood that the computer-readable storage medium can be an internal storage unit of any data processing device described in any of the foregoing embodiments, such as a hard disk or memory. The computer-readable storage medium can also be an external storage device of any data processing device, such as a plug-in hard disk, smart media card (SMC), SD card, flash card, etc., equipped on the device. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of any data processing device. The computer-readable storage medium is used to store the computer program and other programs and data required by the data processing device, and can also be used to temporarily store data that has been output or will be output.

[0108] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. A temporal action detection method based on a cooperative perceptual pyramid network, characterized in that, include: Extract the video feature sequence from the video to be detected; The video feature sequence is processed using a collaborative perception pyramid network model. The collaborative perception pyramid network is a multi-layer pyramid structure. Each layer sequentially uses a temporal decomposition collaborative perception module and a dual-path collaborative perception module to process features, and the optimized features output by the previous layer are used as the input features of the next layer. The optimized features output by each pyramid layer are aggregated into the final features by a dynamic feature aggregation module. Based on the final features, the action category is identified and the action time boundary is located. In each layer of the pyramid structure, the temporal decomposition collaborative perception module captures time interval features and instantaneous features from the input features through interval-level branches and instantaneous branches, respectively. It adopts an adaptive parameter fusion mechanism, using a trainable weight parameter to dynamically weight and fuse the output features of the interval-level branches and the instantaneous branches. The result is then added to the original input features of the temporal decomposition collaborative perception module to output the fused features. The dual-path collaborative perception module adopts a dual-path downsampling strategy, which processes the fused features through a depthwise separable convolutional path and a max-pooling path respectively. It also employs an adaptive parameter fusion mechanism, which uses a trainable weight parameter to dynamically weight and fuse the final output features of the depthwise separable convolutional path and the final output features of the max-pooling path to output optimized features.

2. The temporal action detection method based on a cooperative perceptual pyramid network according to claim 1, characterized in that, The collaborative perception pyramid network model is trained using a predefined dataset. The video clip samples in the predefined dataset are labeled with action time boundaries and action category labels. The action time boundaries include the start time and end time of the action.

3. The temporal action detection method based on a cooperative perceptual pyramid network according to claim 2, characterized in that, The total training loss includes action classification loss and temporal regression loss. The temporal regression loss is a weighted value of the overlap between the predicted action temporal boundary and the actual action temporal boundary and the distance between the center points.

4. The temporal action detection method based on a cooperative perceptual pyramid network according to claim 2, characterized in that, The action categories identified based on the final features are multiple predefined categories in the predefined dataset.

5. The temporal action detection method based on a cooperative perceptual pyramid network according to claim 1, characterized in that, In the time-series decomposition collaborative sensing module: The interval-level branch calculation process is as follows: Three different convolution operations are performed on the input features to obtain the first convolutional feature, the second convolutional feature, and the third convolutional feature; the first convolutional feature and the second convolutional feature are multiplied by a dot product; the dot product result is input into the first fully connected layer and then activated by the first hyperbolic tangent activation function; the activation result is input into the second fully connected layer, and the output is scaled by dividing by the square root of the feature dimension and then activated by the second hyperbolic tangent activation function; the result of the second activation is multiplied by the third convolutional feature to generate the output feature of the interval-level branch. The instantaneous branch calculation process is as follows: global average pooling is performed on the input features; the pooling result is input into the fourth fully connected layer and then activated by the hyperbolic tangent activation function; the input features are input into the third fully connected layer in parallel; the output of the activated fourth fully connected layer is multiplied by the output of the third fully connected layer to generate the output features of the instantaneous branch.

6. The temporal action detection method based on a cooperative perceptual pyramid network according to claim 1, characterized in that, In the dual-path collaborative sensing module: The calculation process of the depthwise separable convolutional path is as follows: the fifth fully connected operation is performed on the input fused features; the convolution operation is performed on the features output by the fifth fully connected operation; the GeLU activation function is applied to the features output by the convolution operation; the activated features are then subjected to the sixth fully connected operation to generate the output features of the depthwise separable convolutional path. Max pooling paths process fused features through pooling operations.

7. The temporal action detection method based on a cooperative perceptual pyramid network according to claim 1, characterized in that, The aggregation process of the dynamic feature aggregation module includes: It receives the set of optimized features from the output of each layer of a multi-layered pyramid structure; For the set of the th Each layer optimizes its features by downsampling its adjacent features from the previous layer and upsampling its adjacent features from the next layer, so that the processed features of adjacent layers are similar to those of the first layer. The optimized features of each layer have the same dimension; The downsampled optimized features of the previous layer are summed with the upsampled optimized features of the next layer. The sum is then compared with the first optimized feature. The optimized features of each layer are multiplied by their corresponding trainable weight parameters and then summed to output the final features.

8. A temporal action detection system based on a collaborative sensing pyramid network, characterized in that, include: The feature extraction module is used to extract the video feature sequence of the video to be detected; A collaborative perception pyramid network module is connected to the feature extraction module. It is used to process the video feature sequence to obtain the final features, and then simultaneously identify the action category and locate the action time boundary based on the final features. The collaborative sensing pyramid network is a multi-layer pyramid structure. Each layer uses a temporal decomposition collaborative sensing network and a dual-path collaborative sensing network to process features in sequence, and the optimized features output by the previous layer are used as the input features of the next layer. The optimized features output by each pyramid layer are aggregated into the final features through a dynamic feature aggregation module. In each layer of the pyramid structure, the temporal decomposition collaborative sensing network includes interval-level branch units and instantaneous-level branch units, which are used to capture time interval-level features and instantaneous-level features from the input features. An adaptive parameter fusion unit is used to dynamically weight and fuse the output features of the interval-level branch and the instantaneous-level branch using a trainable weight parameter. The result is then added to the original input features of the temporal decomposition collaborative sensing module to output the fused features. The dual-path collaborative perception network adopts a dual-path downsampling strategy, which processes the fused features through a depthwise separable convolutional path and a max-pooling path respectively. It also employs an adaptive parameter fusion mechanism, which uses a trainable weight parameter to dynamically weight and fuse the final output features of the depthwise separable convolutional path and the final output features of the max-pooling path to output optimized features.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the temporal action detection method based on the cooperative perception pyramid network as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the temporal action detection method based on the cooperative perception pyramid network as described in any one of claims 1 to 7.

Images