Time sequence action positioning method and device based on bidirectional interaction and dynamic feature enhancement

Abstract

The present application relates to the technical field of video analysis, and particularly relates to a time sequence action positioning method and device based on bidirectional interaction and dynamic feature enhancement. The method comprises: constructing an adaptive feature enhancement strategy, performing hierarchical network architecture integrated dynamic modeling and local and global time sequence interaction modeling through the adaptive feature enhancement strategy; the adaptive feature enhancement strategy comprises: a multi-scale dynamic time sequence modeling module and a global and local adaptive bidirectional interaction module; introducing the adaptive feature enhancement strategy into an encoder; performing feature enhancement on initial features through the encoder to generate enhanced time sequence features. The present application proposes an effective adaptive bidirectional interaction dynamic time sequence enhancement framework, which uses learnable local and global affine matrices to perform time sequence modeling in parallel. This parallel structure supports iterative bidirectional information transmission between features and adaptive balance of short-term and long-term dependencies.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video analysis, and particularly to a temporal action localization method and device based on bidirectional interaction and dynamic feature enhancement. BACKGROUND

[0002] Temporal action localization (TAL) is an important task in video analysis, aiming to localize and recognize the boundaries and categories of actions in untrimmed videos. With the rapid growth of video data on different platforms, the importance of TAL has significantly increased, as it enables efficient video understanding and retrieval, powering applications such as video surveillance, motion analysis, autonomous driving, and human-computer interaction. Existing TAL methods are generally divided into two-stage methods and one-stage methods, each with their own advantages and disadvantages. Two-stage methods first generate action proposals, and then further process and classify these proposals using a classifier, while one-stage methods integrate proposal generation and classification into an end-to-end process.

[0003] Two-stage methods typically include two steps: proposal generation and proposal classification. The purpose of proposal generation is to identify potential action periods from the video, usually using methods such as sliding windows, boundary prediction, or frame action degree evaluation. Sliding window-based methods generate temporal proposals by applying multi-scale sliding windows to the video, such as TURN TAP. Boundary-based methods locate the temporal boundaries of actions by predicting the boundary confidence at specific positions in the video, such as BMN. Frame action degree-based methods generate proposals by evaluating the action degree score of each frame, such as SSN. These methods usually rely on well-trained classifiers to classify each proposal and predict the action category.

[0004] One-stage methods simplify the process of action proposal generation and classification through end-to-end training, reducing the complexity of training and inference. Instead of relying on traditional time anchors or sliding windows, one-stage methods directly predict the temporal span and category of actions at the video frame level. For example, anchor-free TAL models directly predict the action category for each video frame while simultaneously estimating the corresponding action start and end times. In addition, transformer-based one-stage models such as ActionFormer enhance the ability to model actions of different scales through multi-scale self-attention mechanisms. PBBNet improves the accuracy of action localization by progressively refining predictions, while TriDet uses a granularity-aware layer to optimize temporal modeling.

[0005] Although existing two-stage and one-stage methods have made significant progress in temporal action localization tasks, they generally face the challenge of capturing complex temporal dependencies. Especially when dealing with actions with diverse time spans and complex temporal dependencies, how to effectively balance short-term and long-term dependencies remains a key to improving temporal action localization accuracy. Existing methods often have biases in modeling short-term and long-term dependencies, leading to difficulty in maintaining high localization accuracy under different temporal characteristics of actions. Short-term dependencies usually involve the details and local changes of actions, suitable for precise localization of the starting moment of actions; while long-term dependencies focus on the continuation and integrity of actions, especially important for the ending moment and duration of actions. Traditional methods either overemphasize short-term information, ignoring the overall evolution of actions; or focus on long-term information, leading to decreased sensitivity to local action changes. Therefore, how to adaptively adjust the balance between short-term and long-term dependencies in different scale actions is the key to improving the accuracy of temporal action localization.

[0006] Temporal action localization (TAL) is a challenging visual task that aims to accurately classify and localize all actions in untrimmed videos. Due to the multi-scale nature of action intervals, short-time actions rely on local features, while long-time actions rely on global information, making it a core problem in the TAL domain to alleviate complex temporal dependencies. In this context, the quality of features is crucial to improving the performance of temporal action localization. However, existing TAL methods usually rely on pre-trained features as input, and the expressive ability of these features is limited by the limitations of pre-trained models, making it difficult to fully capture the complex temporal dependencies of actions. Therefore, in the TAL task, how to enhance the representation ability of features through feature enhancement to more efficiently support complex temporal modeling has become a key problem that cannot be ignored. At the same time, to better model short-term and long-term dependencies, it is necessary to further explore effective mechanisms for temporal modeling, including convolution-based methods and Transformer-based methods.

[0007] (1) Convolution-based TAL methods: In the past few years, convolutional neural networks (CNNs) have been the key driving force for the development of computer vision. Since the advent of the landmark AlexNet, a series of highly influential CNN architectures have emerged, which have performed outstandingly in many image understanding tasks and have greatly improved performance indicators. In recent years, as video has become a key source of data in many real-world scenarios, researchers have actively explored ways to extend CNNs to the video domain due to their excellent temporal efficiency. In particular, in the TAL task, convolutional neural network (CNN)-based TAL methods have demonstrated significant advantages in reducing redundancy and accelerating the computing process due to their local connectivity and parameter sharing characteristics. Although CNNs perform well in handling short-term actions with clear boundaries, their limitations make it difficult to effectively model global temporal dependencies when dealing with complex action sequences spanning a long duration, resulting in insufficient performance in handling long-term temporal dependencies. To this end, later methods capture long-term information by expanding the convolution kernel field of view. They have alleviated the problem of temporal dependencies to some extent, but either lack sufficient dynamics and flexibility or fail to fully consider the balance between global and local features, leading to unstable performance of the model in action localization at different scales.

[0008] (2) Transformer-based TAL methods: With its powerful modeling capability, Transformer has achieved remarkable results in computer vision and natural language processing tasks in recent years. This advantage has also prompted the widespread application of Transformer in the field of temporal action localization. For the TAL task, Transformer significantly improves the performance of action localization and classification by capturing long-range dependencies between video frame features. On this basis, Transformer-based TAL methods continue to emerge and show diverse development trends. Some end-to-end modeling methods simplify the complexity of the traditional TAL process by constructing trainable frameworks. For example, TALLFormer combines a short-term Transformer encoder and a long-term memory mechanism to efficiently model video actions while effectively reducing GPU memory overhead; TadTR is based on a Transformer with a deformable attention mechanism, selectively focusing on key sparse context subsets in the video to improve efficiency and performance. In addition, some methods further optimize the use of global information by enhancing context awareness. For example, ActionFormer uses a multi-head self-attention mechanism to directly capture global temporal context, enabling efficient modeling of global temporal relationships; while SAFormer builds an efficient one-stage Transformer model for optimizing action localization and classification by introducing a global channel feature response self-attention mechanism and a classification refinement modification loss. Although these Transformer-based methods have made significant progress in the TAL task, there are still limitations in handling local redundant information and balancing long and short-term dependencies in complex video data. The key problem lies in the lack of an effective bidirectional interaction mechanism to simultaneously capture local and global features, thereby better adapting to complex action temporal relationships.

[0009] (3) The importance of feature enhancement in TAL: Although there has been significant progress in temporal action localization (TAL) methods in recent years, there is still considerable room for improvement. Existing TAL methods rely heavily on annotated, untrimmed video data during training. Unfortunately, the relatively small size of existing TAL datasets severely limits the training performance and generalization potential of models, making it difficult for them to perform accurately in a wider range of scenarios. Compared to image datasets, video data exhibits a more complex and diverse set of characteristics due to the additional temporal dimension, which undoubtedly adds many difficult challenges to the TAL task. Therefore, how to effectively expand existing data to help models achieve a breakthrough in performance has become a key point that needs to be addressed. Although data augmentation is a simple and effective strategy, video data is often long, and directly applying conventional data augmentation methods can result in high computational costs, making the entire process inefficient. In addition, most existing TAL methods rely on pre-trained features as input, making feature-level enhancement particularly important in the TAL task. SUMMARY

[0010] To solve the technical problem of how to adaptively adjust the balance between short-term and long-term dependencies in different scale actions in the prior art, an embodiment of the present application provides a temporal action localization method and device based on bidirectional interaction and dynamic feature enhancement. The technical solution is as follows:

[0011] In one aspect, a temporal action localization method based on bidirectional interaction and dynamic feature enhancement is provided, characterized in that the method comprises:

[0012] S1, obtaining an untrimmed video, processing the untrimmed video through a pre-training network to extract initial features;

[0013] S2, constructing an adaptive feature enhancement strategy, and performing hierarchical network architecture integration dynamic modeling and local and global temporal interaction modeling through the adaptive feature enhancement strategy; the adaptive feature enhancement strategy includes a multi-scale dynamic temporal modeling module and a global and local adaptive bidirectional interaction module;

[0014] S3, introducing the adaptive feature enhancement strategy into an encoder; performing feature enhancement on the initial features through the encoder to generate enhanced temporal features;

[0015] S4, passing the enhanced temporal features to a classification and regression head to predict the action category and time boundary.

[0016] Optionally, in S2, the multi-scale dynamic temporal modeling module includes;

[0017] The multi-scale dynamic temporal modeling module includes three dynamic local DynL affine transformations; through dynamic adjustment of the receptive field of the multi-scale dynamic temporal modeling module, time characteristics of multiple scales are captured.

[0018] Optionally, through dynamic adjustment of the receptive field of the multi-scale dynamic temporal modeling module, time characteristics of multiple scales are captured, including:

[0019] Through dynamic local affine transformation DynL affine transformation, the dynamic changes of actions at different time scales in the TAL task are alleviated;

[0020] Given the input features, the DynL affine transformation uses two depth convolution branches of different scales to process the input features to obtain multi-scale time information;

[0021] Through the hierarchical convolution method, the first branch performs deep convolution with a smaller kernel to capture local time information; the second branch first performs convolution with a kernel k1, and then performs convolution with a larger kernel k2 to gradually expand the receptive field to capture more extensive context;

[0022] The outputs of the two branches are fused to obtain multi-scale sequence feature representations;

[0023] A learnable mask generation module is constructed, and the multi-scale sequence feature representations are input into the mask generation module to generate a mask sequence, which assigns different weights to each position in the sequence, highlighting the features of key frames while suppressing background information;

[0024] Local patterns are extracted from the sequence using convolution layers, and then refined through batch normalization BN; an S-shaped activation function is used to compress the weights to the range [0, 1] to obtain the final mask sequence; the mask sequence is multiplied element-wise with the multi-scale sequence features to generate a weighted sequence feature.

[0025] Optionally, in S2, the global and local adaptive bidirectional interaction module includes:

[0026] The dynamic local affine block DynL affine and the global Global affine interact bidirectionally through the adaptive interactive feature fusion submodule AIFF to form the global and local adaptive bidirectional interaction module.

[0027] The global features are aggregated by the global affine module; the local features are aggregated by the dynamic local affine block DynL affine; and the global and local information is fused by the adaptive interactive feature fusion submodule AIFF.

[0028] Optionally, aggregating the global features by the global affine module includes:

[0029] The self-attention mechanism is incorporated into the global affine module to obtain attention-based global feature aggregation; the global affine module is used to aggregate global features and enhance the ability of the model to identify long-term relationships within a sequence, thereby obtaining robust action boundary positioning.

[0030] Optionally, the global and local information is fused by an adaptive interactive feature fusion sub-module AIFF, including:

[0031] The adaptive interactive feature fusion sub-module dynamically assigns weights to perform feature fusion, balances and fuses long-term and short-term dependencies in time series information.

[0032] Optionally, the adaptive interactive feature fusion sub-module dynamically assigns weights to perform feature fusion, balances and fuses long-term and short-term dependencies in time series information, including:

[0033] Given two input feature tensors;

[0034] The adaptive interactive feature fusion sub-module connects the two input feature tensors along the channel dimension to form a fused feature tensor, and the connected features are converted to generate dynamic weights. Time average pooling is used to reduce the time dimension;

[0035] The merged features are extracted through a convolution layer to extract key context information;

[0036] An S-shaped activation function is applied to generate adaptive weights of local and global features;

[0037] The obtained weights are refined by one-dimensional convolution to reduce the dimension to obtain the final weights; the obtained weights and the original features are weighted to obtain the final output.

[0038] On the other hand, a timing action positioning device based on bidirectional interaction and dynamic feature enhancement is provided, which is applied to a timing action positioning method based on bidirectional interaction and dynamic feature enhancement, and the device includes:

[0039] An initial feature extraction module is configured to obtain an uncropped video, process the uncropped video through a pre-trained network, and extract initial features;

[0040] A feature enhancement strategy construction module is configured to construct an adaptive feature enhancement strategy, perform hierarchical network architecture integrated dynamic modeling and local and global time sequence interaction modeling through the adaptive feature enhancement strategy; the adaptive feature enhancement strategy includes a multi-scale dynamic time sequence modeling module and a global and local adaptive bidirectional interaction module;

[0041] A timing feature enhancement module is configured to introduce the adaptive feature enhancement strategy into an encoder; the encoder is used to perform feature enhancement on the initial features to generate enhanced timing features.

[0042] a prediction module configured to deliver the enhanced temporal features to a classification and regression head to predict the action class and temporal boundary.

[0043] In another aspect, a device for temporal action localization based on bidirectional interaction and dynamic feature enhancement is provided, which comprises a processor and a memory having computer readable instructions stored thereon, the computer readable instructions being executed by the processor to implement any one of the above-mentioned methods for temporal action localization based on bidirectional interaction and dynamic feature enhancement.

[0044] In another aspect, a computer readable storage medium is provided, the storage medium having at least one instruction stored therein, the at least one instruction being loaded and executed by a processor to implement any one of the above-mentioned methods for temporal action localization based on bidirectional interaction and dynamic feature enhancement.

[0045] The technical solutions provided by the embodiments of the present application have at least the following beneficial effects:

[0046] In the embodiments of the present application, 1、The present application proposes a brand-new adaptive temporal enhancement framework, which aims to solve the complex temporal dependency problem in the temporal action localization (TAL) task through bidirectional dynamic interaction and balance of short-term and long-term features;

[0047] 2、The present application designs a multi-scale dynamic temporal modeling module (Multi-Scale Dynamic Temporal Modeling, MS-DyTM), which integrates multi-scale adaptive convolution kernel selection and a learnable mask mechanism, can dynamically adjust the receptive field, and can efficiently capture multi-scale temporal features, thereby significantly improving the distinguishability of the features and enhancing the accuracy and flexibility of the model in different action scenarios;

[0048] 3、The present application proposes an adaptive interaction feature fusion sub-module (Adaptive Interaction Feature Fusion, AIFF), which dynamically adjusts the feature weight through an attention allocation mechanism, realizes efficient balance and fusion of short-term and long-term temporal dependencies, and significantly improves the overall performance of the temporal action localization task;

[0049] 4、The extensive experiments on multiple TAL benchmark datasets show that the proposed method can efficiently achieve the current state-of-the-art performance, verifying its powerful ability and effectiveness in capturing different temporal patterns in unpruned videos. BRIEF DESCRIPTION OF DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description only constitute some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0051] Figure 1 A flowchart of a time sequence action positioning method based on bidirectional interaction and dynamic feature enhancement provided by the embodiment of the present application is shown in the figure.

[0052] Figure 2 A general model architecture diagram provided by the embodiment of the present application is shown in the figure.

[0053] Figure 3 A block diagram of a time sequence action positioning device based on bidirectional interaction and dynamic feature enhancement provided by the embodiment of the present application is shown in the figure.

[0054] Figure 4 A structural schematic diagram of an electronic device provided by the embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0055] The technical solutions in the present application will be described below with reference to the drawings.

[0056] In the embodiments of the present application, the words such as "example", "for example" are used to represent as an example, illustration or description. Any embodiment or design scheme described as "example" in the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the word "example" is intended to present the concept in a specific manner. In addition, in the embodiments of the present application, the meaning expressed by "and / or" can be both, or can be one of the two.

[0057] In the embodiments of the present application, sometimes the subscript such as W1 may be written in the form of non-subscript such as W1, and when the difference is not emphasized, the meanings expressed are consistent.

[0058] In order to make the technical problems, technical solutions and advantages to be solved by the present application more clear, the following will be described in detail with reference to the drawings and specific embodiments.

[0059] The embodiment of the present application provides a time sequence action positioning method based on bidirectional interaction and dynamic feature enhancement. The method can be realized by a time sequence action positioning device based on bidirectional interaction and dynamic feature enhancement. The device can be a terminal or a server. Figure 1 A flowchart of a time sequence action positioning method based on bidirectional interaction and dynamic feature enhancement is shown in the figure. Figure 1As shown, the two-way interaction and dynamic feature enhancement based time sequence action positioning method proposed by the application can include the following steps in the processing flow:

[0060] S1, obtaining an uncropped video, processing the uncropped video through a pre-training network to extract initial features;

[0061] S2, constructing an adaptive feature enhancement strategy, and performing hierarchical network architecture integrated dynamic modeling and local and global time sequence interaction modeling through the adaptive feature enhancement strategy; the adaptive feature enhancement strategy includes a multi-scale dynamic time sequence modeling module, a global and local adaptive two-way interaction module.

[0062] In a feasible implementation, the adaptive feature enhancement strategy solves the complex time sequence dependency problem in time sequence action positioning through hierarchical network architecture integrated dynamic modeling and local and global time sequence interaction modeling. The strategy combines the core modules, multi-scale dynamic time sequence modeling module (MS-DyTM), dynamic local affine (DynL affine), global affine (Global affine) and adaptive interactive feature fusion (AIFF) module, to dynamically model and effectively balance short-term and long-term dependencies.

[0063] The overall architecture of the model is as shown in Figure 2 The adaptive feature enhancement strategy is used to solve the challenge of complex time sequence dependency in time sequence action positioning. First, the uncropped video is processed through the pre-training network to extract the initial feature representation Fpre, and then the feature is enhanced through the encoder to generate the enhanced time sequence feature FEnh. Finally, the rich feature FEnh is transmitted to the classification and regression head to predict the action category and time boundary. In order to realize dynamic modeling and effectively capture the balance between short-term and long-term dependencies, the application introduces an adaptive feature enhancement strategy in the encoder, which mainly includes MS-DyTM module, DynL affine module, Global affine module and AIFF module. Next, the core modules of the model will be described in detail.

[0064] In a feasible implementation, in S2, the multi-scale dynamic time sequence modeling module includes;

[0065] The multi-scale dynamic time sequence modeling module includes three dynamic local DynL affine; allows dynamic adjustment of the receptive field to effectively capture multi-scale time features, improve the distinguishability of the features, and enhance the accuracy and flexibility of the model in different action scenarios.

[0066] In an embodiment, the self-adaptive feature enhancement strategy integrates dynamic local and global temporal modeling through a hierarchical network architecture, addressing the complex temporal dependency issues in temporal action localization. The strategy combines three core modules - multi-scale dynamic temporal modeling (MS-DyTM) (dynamic local (DynL affine), global affine, and adaptive interactive feature fusion (AIFF) modules - to effectively balance short-term and long-term dependencies, as shown in Figure 2

[0067] To improve efficiency and effectiveness, the present application assigns different layers with different modeling strategies.

[0068] In the shallow layer, MS-DyTM consists of three DynL affine layers, focusing on dynamically capturing fine-grained local temporal features using adaptive convolution kernels, ensuring accurate modeling of short-term dependencies and enhancing action boundary detection. By focusing on local temporal segments, the shallow layer preserves detail-oriented features while avoiding premature consideration of global context, thus improving efficiency and accuracy.

[0069] In deep research, global affine captures overall temporal relationships and long-term dependencies through a self-attention mechanism, providing a comprehensive understanding of global action semantics. The AIFF module acts as a bridge, supporting bidirectional interaction between DynL affine and global affine outputs. This module adaptively fuses localized details with global context, dynamically balancing short-term and long-term information, thus enhancing feature integration. This hierarchical design leverages the advantages of shallow layers for efficient local modeling and deep layers for global context understanding, ensuring comprehensive temporal feature representation across different time scales. By combining dynamic local modeling with adaptive global integration, this framework significantly improves the accuracy and localization efficiency of action detection and localization in TAL tasks. This design ensures that the model adapts to actions of different time scales, improving the accuracy of action detection in TAL tasks.

[0070] In an embodiment, the multi-scale dynamic temporal modeling module accepts dynamic adjustments of the domain, capturing multi-scale temporal features, including:

[0071] Through dynamic local affine transformation DynL affine, the dynamic changes of actions at different time scales in TAL tasks are mitigated;

[0072] Given the input features, DynL affine uses two depth convolution branches of different scales to process the input features, obtaining multi-scale temporal information;

[0073] ​The first branch is deep-convolved with a smaller kernel to capture local temporal information, and the second branch is first convolved with a kernel k1 and then with a larger kernel k2 to gradually expand the receptive field to capture a broader context;

[0074] The outputs of the two branches are fused to obtain a multi-scale sequence feature representation.

[0075] A learnable mask generation module is constructed, and the multi-scale sequence feature representation is input into the mask generation module to generate a mask sequence, which assigns different weights to each position in the sequence, highlighting the features of the key frames while suppressing the background information.

[0076] Convolutional layers are used to extract local patterns from the sequence, and batch normalization (BN) is used to refine them to enhance their expressiveness.

[0077] An S-shaped activation function is used to compress the weights to the range [0, 1] to obtain the final mask sequence.

[0078] The mask sequence is multiplied element-wise with the multi-scale sequence features to generate a weighted sequence feature.

[0079] In a feasible implementation, the present application introduces a multi-scale dynamic temporal modeling module (MS-DyTM), which effectively alleviates the challenges posed by the dynamic changes of actions at different time scales in the TAL task using dynamic local affine transformation (DynL affine). The module integrates multi-scale adaptive kernel selection and a learnable mask mechanism, allowing dynamic adjustment of the receptive field to effectively capture multi-scale temporal features and improve the discriminability of the features, making it more accurate to detect the start and end times of actions.

[0080] The implementation of DynL affine will be introduced below, as shown in Figure 2 .

[0081] DynL affine is composed of multi-scale convolution and a mask mechanism. Given the input features, the DynL module first processes the input features using two deep convolution branches of different scales to obtain multi-scale temporal information. The first branch is deep-convolved with a smaller kernel to capture local temporal information, and the second branch is first convolved with a kernel k1 and then with a larger kernel k2 to gradually expand the receptive field to capture a broader context. Through this hierarchical convolution method, the model can capture a larger temporal context while maintaining smoothness, effectively capturing more extensive temporal information. The outputs of the two branches are then fused to obtain a multi-scale sequence feature representation.

[0082] To adaptively weigh the key positions in the sequence, the DynL module designs a learnable mask generation module. The input of this module is the fused multi-scale sequence feature, and its goal is to generate a mask sequence to assign different weights to each position in the sequence, highlighting the features of key frames while suppressing background information. A convolutional layer is used to extract local patterns from the sequence, and batch normalization (BN) is used to refine the features to enhance their expressiveness. An S-shaped activation function is used to compress the weights to the range of [0, 1] to obtain the final mask sequence. Then, the mask sequence is multiplied element-wise with the multi-scale sequence features to generate weighted sequence features. Through this weighting method, the model can emphasize the feature expression of key positions and suppress redundant background information. Then, the weighted features are processed by a deep convolution to adaptively adjust the receptive field, producing two attention maps, which are then activated to obtain the output features of the DynL module.

[0083] Compared with the static feature learning mechanism used in most existing TAL models, the DyTM module introduces an innovative dynamic feature learning method. Traditional TAD models usually use fixed receptive fields in convolution kernels or attention mechanisms to assign equal weights to all time-series features. In contrast, the DyTM module adaptively selects the receptive field through multi-scale convolution to capture features in different time ranges. In addition, DyTM uses a learnable mask to assign different weights to each frame, dynamically focusing on key moments of action while suppressing irrelevant frames. This improves the accuracy of feature representation.

[0084] In an embodiment, the global and local adaptive bidirectional interaction module in S2 includes:

[0085] The dynamic local affine block DynL affine and the global Global affine are combined to form the global and local adaptive bidirectional interaction module through bidirectional interaction of the adaptive interactive feature fusion submodule AIFF.

[0086] The global features are aggregated by the global affine module; the local features are aggregated by the dynamic local affine block DynL affine; and the global and local information is fused by the adaptive interactive feature fusion submodule AIFF.

[0087] In an embodiment, aggregating global features by the global affine module includes:

[0088] The self-attention mechanism is incorporated into the global affine module to obtain attention-based global feature aggregation; the global features are aggregated by the global affine module, and the ability of the model to identify long-term relationships within the sequence is enhanced to obtain robust action boundary localization.

[0089] In an embodiment, capturing the stochastic dependencies across time series is crucial for accurate identification of the start and end of actions in TAL. The global affine module addresses this challenge by aggregating global features and enhancing the model's ability to recognize long-term relationships within the sequence, ensuring robust action boundary localization. To better capture long-term dependencies in time-series data, the invention incorporates an attention mechanism to implement a global affine transformation. By incorporating attention into the global affine module, the global affine formula is modified to include attention-based feature aggregation. By combining self-attention with affine transformation, the global affine module can effectively capture global feature interactions, enabling the model to better understand the temporal relationships between the entire sequence. This enables more accurate action localization, especially in scenarios where actions span multiple time steps or exhibit complex temporal dependencies.

[0090] In an embodiment, the global and local information is fused by an adaptive interactive feature fusion sub-module AIFF, including:

[0091] Effective feature fusion is achieved by dynamically assigning weights, effectively balancing and fusing long-term and short-term dependencies in time series information, improving the overall performance of the TAL task.

[0092] In an embodiment, the adaptive interactive feature fusion sub-module dynamically assigns weights for feature fusion, balancing and fusing long-term and short-term dependencies in time series information, including:

[0093] The adaptive interactive feature fusion sub-module (AIFF) effectively balances and fuses long-term and short-term dependencies in time series information. This module uses a combination of convolutional operations and attention mechanisms to achieve effective feature fusion by dynamically assigning weights, improving the overall performance of the temporal action localization (TAL) task. The DynL local output features and Global module output features are obtained as input feature tensors; the adaptive interactive feature fusion sub-module connects the two input feature tensors (local and global feature tensors) along the channel dimension to form a fused feature tensor, and the connected features undergo a series of transformations to generate dynamic weights. Time average pooling is used to reduce the time dimension; the merged features are passed through a convolutional layer to extract key contextual information; an s-shaped activation function is applied to generate adaptive weights for local and global features; the resulting weights are refined by one-dimensional convolution to reduce the dimension, obtaining the final weights; the obtained weights and the original features are weighted to obtain the final output. This adaptive fusion process ensures that AIFF can effectively balance short-term and long-term dependencies based on task context, enabling the model to better capture the inherent temporal dynamics in action localization. As Figure 2 (c) as shown.

[0094] S3, introducing an adaptive feature enhancement strategy into the encoder; performing feature enhancement on the initial features through the encoder to generate enhanced time sequence features;

[0095] S4, passing the enhanced time sequence features to the classification and regression head to predict the action category and time boundary.

[0096] In a feasible implementation, as shown in Table 1 below, the performance difference of the method of the present application and the current most advanced method on the THUMOS14 dataset is compared. In order to comprehensively evaluate the effectiveness of the method of the present application, experiments are conducted under various pre-training features, including I3D, InterVideo and VideoMAE, to verify the applicability and robustness of the method in different backbone architectures.

[0097] Table 1: Comparison results with advanced methods on THUMOS14 dataset

[0098]

[0099] On the classic I3D backbone network, the method of the present application achieves an average mAP of 69.4%, which is 2.6% higher than ActionFormer, and is competitive with TriDet. On the stronger InterVideo backbone, the method of the present application further improves the average mAP to 74.6%, surpassing all existing methods and achieving SOTA performance. Compared with ActionFormer, it is improved by 3.0%, compared with DyFADet, it is improved by 2.2%, and compared with ActionMamba, it is improved by 1.9%. Based on the VideoMAE backbone, the method of the present application also performs well, with an average mAP of 1.5% higher than ActionFormer, 1.3% higher than MFAM, and 1.0% higher than TriDet.

[0100] These results show that the method of the present application performs particularly well on InterVideo and VideoMAE features, effectively overcoming their limitations from the perspective of feature enhancement. For InterVideo features, the method of the present application adaptively enhances the expression ability of time span and complex feature representation through dynamic modeling, fully exploiting rich semantic and time sequence information across videos. For VideoMAE features, the method of the present application further strengthens the interaction between global and local features, dynamically balances short-term and long-term dependencies, and ensures accurate time action modeling, effectively overcoming the inherent static limitations of pre-training features. These performance improvements fully demonstrate the superiority and wide applicability of the method of the present application in complex time sequence modeling tasks.

[0101] Table II: Comparison results on HACS dataset with state-of-the-art methods

[0102]

[0103] As shown in Table II, the performance of the method on the HACS dataset, the average mAP at [0.5, 0.75, 0.95] tIoU thresholds is reported, and the best results are marked in bold. It can be seen that the method of the present application achieves an impressive 45.1% mAP on the HACS dataset, outperforming other state-of-the-art methods. It is worth noting that even at higher tIoU thresholds, the method of the present application still shows significant advantages. At tIoU = 0.95, the method of the present application is 1.5% higher than DyFADet, 2.3% higher than ActionMamba, and 2.5% higher than TriDet, fully embodying its strong robustness in accurate action boundary positioning. Especially under the challenging high threshold condition, the consistent advantage shown by the method of the present application further proves its excellent ability in modeling complex temporal dependency and adaptively balancing short-term and long-term interactions.

[0104] Table III: Comparison results on ActivityNet-1.3 dataset with state-of-the-art methods

[0105]

[0106] As shown in Table III, the experimental results of the method of the present application on the ActivityNet-1.3 dataset verify its effectiveness under the I3D and InterVideo feature backbone. When using I3D features, the method of the present application achieves a significant improvement in average mAP compared to the ActionFormer method, with a performance gain of 1.4%. This shows that even in the case of relatively limited feature performance, the method of the present application can still effectively improve performance. When using InterVideo features, the method of the present application achieves stable performance improvement at all tIoU thresholds, especially at higher thresholds (such as tIoU = 0.95), which fully reflects its strong ability to accurately capture action boundaries. In terms of average mAP, the method of the present application is 1.1% higher than TriDet. These experimental results powerfully demonstrate the excellent performance and robustness of the method of the present application in dealing with different feature representation challenges.

[0107] In summary, extensive experiments on THUMOS14, HACS and ActivityNet-1.3 datasets show that the proposed method comprehensively outperforms the existing advanced methods in performance. The proposed algorithm provides an effective solution to the complex temporal dependence problem, and opens up new directions and ideas for further research in the field of temporal action localization.

[0108] In the embodiment of the application, a brand-new adaptive temporal enhancement framework is proposed, aiming to solve the complex temporal dependence problem in the temporal action localization (TAL) task through bidirectional dynamic interaction and balance of short-term and long-term features;

[0109] The application designs a multi-scale dynamic temporal modeling module (MS-DyTM), which integrates multi-scale adaptive convolution kernel selection and a learnable mask mechanism, can dynamically adjust the receptive field, and efficiently capture multi-scale temporal features, thereby significantly improving the distinguishability of the features and enhancing the accuracy and flexibility of the model in different action scenarios;

[0110] The application proposes an adaptive interaction feature fusion sub-module (Adaptive Interaction Feature Fusion, AIFF), which dynamically adjusts the feature weight through an attention allocation mechanism, realizes efficient balance and fusion of short-term and long-term temporal dependence, and significantly improves the overall performance of the temporal action localization task;

[0111] Extensive experiments on multiple TAL benchmark datasets show that the proposed method can efficiently achieve the current most advanced performance, verifying its powerful ability and effectiveness in capturing different temporal patterns in unpruned videos.

[0112] Figure 3 is a block diagram of a temporal action localization device 300 based on bidirectional interaction and dynamic feature enhancement according to an exemplary embodiment, the device 300 is used for a temporal action localization method based on bidirectional interaction and dynamic feature enhancement. Referring to Figure 3 The device includes a signal initial feature extraction module 310, a feature enhancement strategy construction module 320, a temporal feature enhancement module 330, and a prediction module 340. Among them:

[0113] The initial feature extraction module 310 is used to obtain an unpruned video, process the unpruned video through a pre-trained network, and extract initial features;

[0114] The feature enhancement strategy construction module 320 is configured to construct an adaptive feature enhancement strategy, and perform hierarchical network architecture integrated dynamic modeling and local and global time sequence interaction modeling through the adaptive feature enhancement strategy. The adaptive feature enhancement strategy includes a multi-scale dynamic time sequence modeling module and a global and local adaptive bidirectional interaction module.

[0115] The time sequence feature enhancement module 330 is configured to introduce the adaptive feature enhancement strategy into an encoder, and perform feature enhancement on initial features through the encoder to generate enhanced time sequence features.

[0116] The prediction module 340 is configured to pass the enhanced time sequence features to a classification and regression head to perform prediction on an action category and a time boundary.

[0117] Optionally, the multi-scale dynamic time sequence modeling module includes a dynamic local affine transformation DynL affine.

[0118] The multi-scale dynamic time sequence modeling module includes three dynamic local affine transformations DynL affine. The multi-scale dynamic time sequence modeling module is configured to accept dynamic adjustment of a domain and capture multi-scale time features.

[0119] Optionally, the multi-scale dynamic time sequence modeling module is configured to accept dynamic adjustment of a domain and capture multi-scale time features, including:

[0120] The dynamic local affine transformation DynL affine is configured to relieve dynamic changes of actions at different time scales in a TAL task.

[0121] Given input features, the DynL module is configured to process the input features using two depth convolution branches of different scales to obtain multi-scale time information.

[0122] Through a hierarchical convolution method, a first branch is configured to perform depth convolution with a smaller kernel to capture local time information. A second branch is configured to first perform convolution with a kernel k1 and then perform convolution with a larger kernel k2 to gradually expand a receptive field to capture more extensive context.

[0123] The outputs of the two branches are fused to obtain multi-scale sequence feature representations.

[0124] A learnable mask generation module is constructed. The multi-scale sequence feature representations are input into the mask generation module to generate a mask sequence, which assigns different weights to each position in the sequence, highlights the features of key frames while suppressing background information, extracts local patterns from the sequence using a convolution layer, and refines the local patterns through batch normalization BN to enhance expressiveness. The mask sequence and the multi-scale sequence features are multiplied element by element to generate weighted sequence features.

[0125] Optionally, the global and local adaptive bidirectional interaction module includes:

[0126] The dynamic local affine block DynL affine and the global Global affine are combined to form a global and local adaptive bidirectional interaction module through bidirectional interaction of an adaptive interactive feature fusion sub-module AIFF.

[0127] The global features are aggregated through the global affine module; the local features are aggregated through the dynamic local affine block DynL affine; and the global and local information is fused through the adaptive interactive feature fusion sub-module AIFF.

[0128] Optionally, aggregating the global features through the global affine module comprises:

[0129] The self-attention mechanism is introduced into the global affine module to obtain attention-based global feature aggregation; the global features are aggregated through the global affine module, and the ability of the model to identify long-term relationships within a sequence is enhanced to obtain robust action boundary positioning.

[0130] Optionally, fusing the global and local information through the adaptive interactive feature fusion sub-module AIFF comprises:

[0131] The adaptive interactive feature fusion sub-module dynamically assigns weights to perform feature fusion, balances and fuses long-term and short-term dependencies in time series information.

[0132] Optionally, the adaptive interactive feature fusion sub-module dynamically assigns weights to perform feature fusion, balances and fuses long-term and short-term dependencies in time series information, comprising:

[0133] Two input feature tensors are given; the input feature tensors comprise: output features obtained by DynL affine and output features obtained by Global affine;

[0134] The adaptive interactive feature fusion sub-module connects the two input feature tensors along the channel dimension to form a fused feature tensor, and the connected features are converted to generate dynamic weights. Time average pooling is used to reduce the time dimension;

[0135] The merged features are extracted through a convolution layer to extract key context information;

[0136] An s-shaped activation function is applied to generate adaptive weights of local and global features;

[0137] The obtained weights are reduced in dimension through one-dimensional convolution to obtain final weights; and the obtained weights and the original features are weighted to obtain the final output.

[0138] In the embodiments of the present application, a new adaptive time sequence enhancement framework is proposed, which aims to solve the problem of complex time sequence dependency in the time sequence action localization (TAL) task through bidirectional dynamic interaction and balance of short-term and long-term features.

[0139] The application designs a multi-scale dynamic timing modeling module (MS-DyTM), which integrates a multi-scale adaptive convolution kernel selection and a learnable mask mechanism, can dynamically adjust the receptive field, efficiently capture multi-scale timing features, and significantly improve the distinguishability of features, and enhance the accuracy and flexibility of the model in different action scenarios.

[0140] The application proposes an adaptive interactive feature fusion sub-module, which dynamically adjusts the feature weight through an attention allocation mechanism, realizes efficient balance and fusion of short-term and long-term timing dependence, and significantly improves the overall performance of the timing action positioning task.

[0141] Extensive experiments on multiple TAL benchmark datasets show that the proposed method can efficiently achieve the current most advanced performance, verifying its powerful ability and effectiveness in capturing different timing patterns in unpruned videos.

[0142] Figure 4 It is a structure schematic view of a timing action positioning equipment based on bidirectional interaction and dynamic feature enhancement provided by an embodiment of the application, as shown in Figure 4 The timing action positioning equipment based on bidirectional interaction and dynamic feature enhancement can include the timing action positioning device based on bidirectional interaction and dynamic feature enhancement shown in Figure 3 Optionally, the timing action positioning equipment based on bidirectional interaction and dynamic feature enhancement 410 can include a first processor 2001.

[0143] Optionally, the timing action positioning equipment based on bidirectional interaction and dynamic feature enhancement 410 can further include a memory 2002 and a transceiver 2003.

[0144] Among them, the first processor 2001 and the memory 2002 and the transceiver 2003 can be connected through a communication bus.

[0145] The following will be combined Figure 4 The various constituent components of the timing action positioning equipment based on bidirectional interaction and dynamic feature enhancement 410 will be specifically introduced:

[0146] The first processor 2001 is the control center of the timing action positioning device 410 based on bidirectional interaction and dynamic feature enhancement. It can be a single processor or a collective term for multiple processing elements. For example, the first processor 2001 can be one or more central processing units (CPUs), application-specific integrated circuits (ASICs), or one or more integrated circuits configured to implement embodiments of the present invention, such as one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs).

[0147] Optionally, the first processor 2001 can execute various functions of the timing motion positioning device 410 based on bidirectional interaction and dynamic feature enhancement by running or executing software programs stored in the memory 2002 and calling data stored in the memory 2002.

[0148] In a specific implementation, as one example, the first processor 2001 may include one or more CPUs, for example... Figure 4 CPU0 and CPU1 are shown in the diagram.

[0149] In a specific implementation, as one example, the timing action positioning device 410 based on bidirectional interaction and dynamic feature enhancement may also include multiple processors, for example... Figure 4 The first processor 2001 and the second processor 2004 are shown in the diagram. Each of these processors can be a single-core processor or a multi-core processor. Here, a processor can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0150] The memory 2002 is used to store the software program that executes the present invention, and is controlled by the first processor 2001 to execute it. The specific implementation method can be referred to the above method embodiment, and will not be repeated here.

[0151] Optionally, the memory 2002 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. The memory 2002 may be integrated with the first processor 2001 or may exist independently, and may be accessed through the interface circuit of the timing action positioning device 410 based on bidirectional interaction and dynamic feature enhancement. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0152] The transceiver 2003 is used to communicate with network devices or with terminal devices.

[0153] Alternatively, transceiver 2003 may include a receiver and a transmitter. Figure 4 (Not shown separately). The receiver is used to implement the receiving function, and the transmitter is used to implement the transmitting function.

[0154] Optionally, the transceiver 2003 can be integrated with the first processor 2001 or exist independently, and can be connected to the interface circuit of the timing action positioning device 410 based on bidirectional interaction and dynamic feature enhancement. Figure 4 (Not shown in the image) is coupled to the first processor 2001, and this embodiment of the invention does not specifically limit this.

[0155] It should be noted that, Figure 4 The structure of the timing action localization device 410 based on bidirectional interaction and dynamic feature enhancement shown in the figure does not constitute a limitation on the router. The actual knowledge structure recognition device may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0156] Furthermore, the technical effects of the timing action positioning device 410 based on bidirectional interaction and dynamic feature enhancement can be referred to the technical effects of the timing action positioning method based on bidirectional interaction and dynamic feature enhancement described in the above method embodiments, and will not be repeated here.

[0157] It should be appreciated that the first processor 2001 in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0158] It should also be understood that the memory in the embodiments of the present application can be a volatile memory or a nonvolatile memory, or can include both volatile and nonvolatile memory. Among them, the nonvolatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (direct rambus RAM, DR RAM).

[0159] The above-described embodiments can be implemented in whole or in part by software, hardware (such as a circuit), firmware, or any combination thereof. When implemented in software, the above-described embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the processes or functions described in the embodiments of the present application are wholly or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable sensor. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center through a wired (such as infrared, wireless, microwave, etc.) manner. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing one or more available medium collections. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state disk.

[0160] It should be understood that the term "and / or" herein merely describes an association relationship of associated objects, which means that there can be three relationships, for example, A and / or B can represent three cases of A alone, A and B together, and B alone, where A and B can be singular or plural. In addition, the character " / " herein generally represents that the associated objects before and after are an "or" relationship, but can also represent an "and / or" relationship, which can be understood in the context before and after.

[0161] It should be understood that in various embodiments of the present application, the size of the sequence number of each process described above does not mean the order of execution, and the execution order of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0162] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be realized in electronic hardware, or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0163] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0164] In addition, each functional unit in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit.

[0165] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the application or the part of the prior art that contributes essentially or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the application.

[0166] The above is only a specific embodiment of the application, but the protection scope of the application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the application, which should be covered within the protection scope of the application. Therefore, the protection scope of the application should be subject to the protection scope of the claims.

Claims

1. A temporal action localization method based on bidirectional interaction and dynamic feature enhancement, characterized in that, The method includes: S1. Obtain the uncropped video, process the uncropped video through a pre-trained network, and extract initial features; S2. Construct an adaptive feature enhancement strategy, and perform hierarchical network architecture integration dynamic modeling and local and global temporal interactive modeling through the adaptive feature enhancement strategy; the adaptive feature enhancement strategy includes: a multi-scale dynamic temporal modeling module and a global and local adaptive bidirectional interactive module. S3. Introduce the adaptive feature enhancement strategy into the encoder; enhance the initial features through the encoder to generate enhanced temporal features; S4. Pass the enhanced temporal features to the classification and regression heads to predict action categories and time boundaries; Wherein, the multi-scale dynamic temporal modeling module in S2 includes: The multi-scale dynamic temporal modeling module includes three dynamic local DynL affines; by dynamically adjusting the receiving domain through the multi-scale dynamic temporal modeling module, multi-scale temporal features are captured. The step of capturing multi-scale temporal features through dynamic adjustment of the receiving domain by the multi-scale dynamic temporal modeling module includes: The dynamic changes of actions at different time scales in the TAL task are mitigated by using the dynamic local affine transformation DynL affine. Given input features, DynL affines process the input features using two depthwise convolutional branches at different scales to obtain multi-scale temporal information; Using a hierarchical convolution method, the first branch performs a depthwise convolution with a smaller kernel to capture local temporal information; the second branch first convolves with kernel k1, and then with a larger kernel k2, gradually expanding the receptive field to capture a wider range of context. The outputs of the two branches are fused to obtain a multi-scale sequence feature representation; A learnable mask generation module is constructed. Multi-scale sequence feature representations are input into the mask generation module to generate a mask sequence. Different weights are assigned to each position in the sequence to highlight the features of key frames while suppressing background information. Local patterns are extracted from the sequence using convolutional layers, refined by batch normalization (BN), and the weights are compressed to the range [0,1] using a sigmoid activation function to obtain the final mask sequence. The mask sequence is then element-wise multiplied with the multi-scale sequence features to generate a weighted sequence feature weighting.

2. The temporal action localization method based on bidirectional interaction and dynamic feature enhancement according to claim 1, characterized in that, In S2, the global and local adaptive bidirectional interaction module includes: The dynamic local affine block DynL affine and the global affine are combined to form a global and local adaptive bidirectional interaction module through the adaptive interactive feature fusion submodule AIFF bidirectional interaction. Global features are aggregated through the Global affine mapping; local features are aggregated through the DynL dynamic local affine mapping; and global and local information are fused through the Adaptive Interactive Feature Fusion Submodule AIFF.

3. The temporal action localization method based on bidirectional interaction and dynamic feature enhancement according to claim 2, characterized in that, The aggregation of global features through the global affine module includes: By incorporating a self-attention mechanism into a global affine module, a global feature aggregation based on attention is obtained. The global affine module aggregates global features and enhances the model's ability to identify long-term relationships within sequences, thereby achieving robust action boundary localization.

4. The temporal action localization method based on bidirectional interaction and dynamic feature enhancement according to claim 3, characterized in that, The process of fusing global and local information through the Adaptive Interactive Feature Fusion (AIFF) submodule includes: The adaptive interactive feature fusion submodule dynamically assigns weights to perform feature fusion, balancing and integrating long-term and short-term dependencies in time series information.

5. The temporal action localization method based on bidirectional interaction and dynamic feature enhancement according to claim 4, characterized in that, The step of dynamically assigning weights through the adaptive interactive feature fusion submodule to perform feature fusion, balancing and fusing long-term and short-term dependencies in time series information, includes: Given two input feature tensors; the input feature tensors include: the output features of the DynL affine and the output features of the Global affine; The adaptive interactive feature fusion submodule connects two input feature tensors along the channel dimension to form a fused feature tensor. The connected features undergo a series of transformations to generate dynamic weights. Temporal average pooling is used to reduce the temporal dimension. The merged features are used to extract key contextual information through convolutional layers; An adaptive weighting for local and global features is generated using a sigmoid activation function. The weights are thinned using one-dimensional convolution to reduce the dimensionality and obtain the final weights. The final weights are then weighted together with the original features to obtain the final output.

6. A temporal action localization device based on bidirectional interaction and dynamic feature enhancement, wherein the temporal action localization device based on bidirectional interaction and dynamic feature enhancement is used to implement the temporal action localization method based on bidirectional interaction and dynamic feature enhancement as described in any one of claims 1-5, characterized in that, The device includes: The initial feature extraction module is used to acquire the uncropped video, process the uncropped video through a pre-trained network, and extract initial features. The feature enhancement strategy construction module is used to construct an adaptive feature enhancement strategy, and to perform hierarchical network architecture integration dynamic modeling and local and global temporal interaction modeling through the adaptive feature enhancement strategy; the adaptive feature enhancement strategy includes: a multi-scale dynamic temporal modeling module and a global and local adaptive bidirectional interaction module. The temporal feature enhancement module is used to introduce the adaptive feature enhancement strategy into the encoder; and to enhance the initial features through the encoder to generate enhanced temporal features. The prediction module is used to pass enhanced temporal features to the classification and regression heads to predict action categories and time boundaries.

7. A temporal action localization device based on bidirectional interaction and dynamic feature enhancement, the temporal action localization device based on bidirectional interaction and dynamic feature enhancement comprising: processor; The memory stores computer-readable instructions, which, when executed by the processor, implement any one of the temporal action localization methods based on bidirectional interaction and dynamic feature enhancement as described in any one of claims 1-5.

8. A computer-readable storage medium storing at least one instruction, the at least one instruction being loaded and executed by a processor to implement any one of the methods in the temporal action localization method based on bidirectional interaction and dynamic feature enhancement as described in any one of claims 1-5.

Images