Weak supervision time sequence action positioning method based on frequency domain sensing network

Abstract

The invention discloses a weak supervision time sequence action positioning method based on a frequency domain sensing network, and belongs to the field of video understanding. Constructing a feature extraction module; a frequency domain coding and interaction unit; and guiding fusion of main harmonics. And splicing the RGB features and the FLOW features after the main harmonic guided fusion processing along the channel dimension to form a fused time-frequency representation. According to the network, firstly, a frequency domain coding and interaction unit is designed, global structure semantics of a video sequence is captured through frequency domain coding, and discriminative region response is highlighted through frequency band interaction, so that the sensitivity of a model to time sequence evolution and action boundary change is enhanced. According to the method, time domain and frequency domain features are adaptively fused according to harmonic energy distribution, spectral response related to boundaries is intensified, and meanwhile, dynamic complementary information of the time domain features is reserved. The two parts cooperate with each other to realize joint modeling of a global steady state structure and local dynamic change, and the boundary sensing capability and the action positioning precision of the model are remarkably improved.

Description

A Weakly Supervised Temporal Action Localization Method Based on Frequency Domain Aware Networks Technical Field

[0001] This invention belongs to the field of video understanding, specifically a weakly supervised temporal action localization method based on deep learning. This method introduces a frequency domain modeling perspective for the first time, fusing visual and semantic features to construct a frequency domain perceptual network, aiming to reduce label dependence while improving action localization performance. The method has broad application prospects in tasks such as intelligent surveillance, abnormal behavior detection, and video question answering. Background Technology

[0002] With the rapid development of the internet, video data has exploded, and how to retrieve and understand valuable information from massive amounts of video has become an important research topic in the field of video understanding. Temporal action localization, as one of the core tasks, aims to accurately identify and locate the start and end times of actions in unedited videos. However, existing fully supervised temporal action localization methods require precise temporal boundary annotations for each training sample, a process that is both time-consuming and labor-intensive, severely limiting its practical application in large-scale video analysis. To address this, weakly supervised temporal action localization (WTAL) methods have emerged. This method can be trained solely based on video-level category annotations, significantly reducing the cost of manual annotation while still achieving competitive localization performance, thus attracting widespread attention in both academia and industry.

[0003] Existing WTAL methods typically employ a multiple instance learning (MIL) framework, utilizing RGB and FLOW features to generate class activation sequences (CAS), followed by a top-k strategy for filtering and thresholding. While this paradigm has made some progress, it relies on linear temporal aggregation in the temporal domain, making it difficult to effectively handle the dynamic changes in action boundaries in complex videos. Especially in unedited video scenes, the complex background and significant differences in action instance scale easily lead to activation shifts in CAS, resulting in incomplete or overcomplete action localization, as shown in Figure 1. Therefore, effectively balancing global steady-state and local dynamic modeling to accurately perceive temporal boundaries is the core challenge facing WTAL.

[0004] To address the aforementioned limitations, this invention proposes a weakly supervised temporal action localization method (FSNet) based on a frequency-domain sensing network, as shown in Figure 2. This method mainly comprises two parts: Frequency Domain Coding and Interaction Unit (FEIU) and Dominant Harmonic Guided Fusion (DHGF). FEIU extracts globally stable action semantics through frequency domain coding and introduces a frequency band interaction mechanism to model the complementary relationships between different frequency components, thereby enhancing the model's sensitivity to structural patterns and temporal boundary changes. Based on this, DHGF dynamically adjusts the fusion weights of temporal and frequency domain features using the dominant harmonic energy distribution, achieving adaptive time-frequency fusion. This effectively balances the local dynamics of the time domain with the global steady state of the frequency domain, enabling the model to more accurately perceive action boundaries and improving localization accuracy and robustness. Summary of the Invention

[0005] This invention proposes a weakly supervised temporal action localization method based on a Frequency Domain Aware Network (FSNet). The network first designs a Frequency Encoding and Interaction Unit (FEIU) to capture the global structural semantics of the video sequence through frequency domain encoding and utilizes frequency band interaction to highlight discriminative regional responses, thereby enhancing the model's sensitivity to temporal evolution and action boundary changes. Building upon this, a Dominant Harmonic Guided Fusion (DHGF) is proposed, adaptively fusing temporal and frequency domain features based on harmonic energy distribution, focusing on strengthening boundary-related spectral responses while preserving the dynamic complementary information of temporal features. The two parts work synergistically to achieve joint modeling of global steady-state structure and local dynamic changes, significantly improving the model's boundary awareness and action localization accuracy.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a weakly supervised temporal action localization method based on a frequency domain sensing network. The implementation steps of this method are as follows:

[0007] Step 1: Constructing the Feature Extraction Module. First, the input unedited video is divided into non-overlapping segments, and a two-stream network (I3D) is applied to extract RGB and FLOW features to simultaneously model appearance and motion information. Then, the two types of features are stacked and fused along the temporal dimension to form a holistic representation of the video, providing the input foundation for subsequent modeling.

[0008] Step 2: Frequency Domain Encoding and Interaction Unit. First, background suppression is applied to the input RGB features to prevent low-frequency components from dominating spectral energy and masking high-frequency responses. Then, a frequency domain encoder is used to transform the video feature sequence, capturing global structural semantic information from a frequency perspective. Furthermore, a frequency band interaction mechanism is introduced to distinguish and model different frequency components: low-frequency components represent stable action patterns, while high-frequency components characterize action boundaries and rapidly changing features. By integrating these complementary frequency cues, effective modeling of action structure evolution is achieved, improving the model's global consistency and boundary awareness.

[0009] Step 3: Dominant Harmonic-Guided Fusion. Although frequency domain modeling can capture global rhythmic patterns, time domain features retain fine-grained temporal dynamics. Therefore, the complementary fusion of the two is crucial for accurate action boundary localization. Based on the analysis of the spectral characteristics of time-series signals, this invention proposes a dominant harmonic-guided fusion strategy. This strategy first identifies the dominant harmonic peak in the input signal and characterizes the overall rhythmic structure and boundary-related changes based on the dominant frequency and its harmonic sequence. Subsequently, the relative energy distribution between each harmonic is mapped as a dynamic fusion weight, used to adaptively balance the global spectral information of the frequency domain features and the local temporal clues of the time domain features. Through this dynamic fusion mechanism, this invention enhances the accuracy and robustness of boundary recognition while maintaining structure awareness, thereby achieving efficient temporal action localization in complex scenarios.

[0010] Step 4: The RGB features and FLOW features, after being fused by the main harmonic guide, are concatenated along the channel dimension to form a fused time-frequency representation. Finally, a classifier is used to predict the fused features, generating a class activation sequence (CAS). This sequence reflects the correlation between each segment of the video and the action category in the time dimension, providing a high-quality response basis for subsequent action localization and boundary determination.

[0011] Step 2 captures the global structural semantics of the video, and Step 3 adaptively integrates frequency domain and temporal domain features to enhance action boundary awareness. Finally, Step 4 inputs the fused RGB and FLOW features into the classifier to generate high-quality CAS, achieving accurate action localization and boundary detection. Attached Figure Description

[0012] Figure 1 illustrates the problems with existing methods.

[0013] Figure 2 is a diagram of the overall network framework of the present invention.

[0014] Figure 3 shows the results of the qualitative visualization analysis. Detailed Implementation

[0015] The specific implementation of the present invention will be described in detail below with reference to the accompanying drawings.

[0016] The technical solution adopted in this invention is a weakly supervised temporal action localization method based on a frequency domain sensing network (FSNet). The system modules implementing this method include a feature extraction module, a frequency domain coding and interaction unit (FEIU), and a dominant harmonic guided fusion (DHGF). The feature extraction module is used for feature extraction, serving as input data for subsequent steps. The FEIU uses frequency domain coding to extract globally stable action semantics and introduces frequency band interaction to emphasize the responses of different frequency components, enhancing the model's sensitivity to structural patterns and temporal evolution. Based on this, DHGF dynamically adjusts the fusion weights between temporal and frequency domain features using the energy distribution of the dominant harmonic, achieving adaptive time-frequency fusion, thereby improving localization accuracy and robustness.

[0017] The overall framework of the technical solution of this invention is shown in Figure 2. Further, the feature extraction is performed on a dataset containing daily activities and sports events. Given an unedited video and corresponding category labels, denoted as... ,in This represents a video segment of length T. It is a one-hot vector of action category C, y c =1 indicates that action category c exists in V, y c =0 indicates none. The goal of WTAL is to use Y as a monitoring signal to locate action instances in V. s n e n c n and p n These represent the start time, end time, prediction category, and confidence score of the action instance, respectively.

[0018] Following the standard procedure, a pre-trained I3D model is first used to extract data from video segments v. t Extracting RGB features and FLOW features Where D represents the feature dimension. Then, all segment features are concatenated along the time dimension to form a complete video feature sequence, represented as... , These features will serve as input data for subsequent steps.

[0019] The implementation process of frequency domain coding and interaction unit (FEIU) is as follows:

[0020] The frequency domain perspective can explicitly reveal the periodicity and stability patterns in action signals, providing a stronger representation for modeling the internal structure and boundary changes of foreground actions. Therefore, this invention introduces a frequency domain modeling mechanism to extract stable response patterns from the frequency space, compensating for the limitations of the time domain in modeling global semantics and action boundaries.

[0021] Frequency domain encoder. In undressed video, the background typically occupies a long time segment, while action instances are often sparsely distributed along the timeline, easily masked by the dominant low-frequency components, thus weakening the high-frequency response and resulting in indistinct motion features. To highlight the representation of action regions, this invention uses X... r A foreground attention mechanism is introduced, which generates a foreground attention map through convolutional layers and a sigmoid function. Based on this, the input features are weighted to obtain the foreground enhancement features:

[0022] (1);

[0023] in, This indicates an element-wise multiplication operation.

[0024] Enhance feature X rs The spectrum is then transformed to the complex domain and projected onto the frequency space using a Fast Fourier Transform (FFT). To improve the stability of the spectral modeling, the obtained amplitude spectrum is centered and normalized to generate a structured frequency representation. .

[0025] (2);

[0026] in, Represents the complex number mapping function, Indicates complex number centralization operations, and This represents the normalization operation for complex numbers.

[0027] The resulting spectral representation F is input to a complex Transformer to model dependencies within the spectral range. First, a complex-valued learnable matrix is ​​used... , and F is linearly mapped to a query vector Q, a key vector K, and a value vector V. These vectors are then processed via dot product attention blocks. To model contextual relevance, residual connections and complex normalization are combined to maintain a stable feature distribution. Subsequently, a complex feedforward network and another set of residual normalizations are used to further refine the representation. The overall calculation formula is as follows:

[0028] (3);

[0029] in, It is a two-layer complex feedforward network. This encoding integrates global spectral dependencies, generating enhanced frequency domain feature representations. .

[0030] Frequency band interaction mechanism. Different frequency bands carry heterogeneous semantic information: low-frequency components usually reflect stable action patterns, while high-frequency components correspond to boundary changes and action transitions. To fully utilize the complementary characteristics of the two, this invention designs a frequency band interaction mechanism.

[0031] Specifically, firstly, frequency is represented The transformation is converted to polar coordinates, decomposing the complex features into amplitude and phase components, which are then concatenated to align with the real-valued attention structure. Since the initial modeling in the complex domain preserves complete spectral information, subsequent modeling will be performed in the real domain to improve computational efficiency and numerical stability. This transformation is defined as follows:

[0032] (4);

[0033] in, , This indicates a splicing operation. Represents the modulus of a complex number. The argument function of a complex number.

[0034] To further explicitly capture heterogeneous semantics across frequency bands, this invention introduces an adaptive gating mechanism, which is based on... Dynamically generated band mask It is implemented using convolutional layers and a sigmoid function. Based on this mask, the spectrum is adaptively decomposed into low-frequency and high-frequency bands:

[0035] , (5);

[0036] in, It is a single sheet with the same shape as M.

[0037] Subsequently, independent multi-head attention heads are used to model the low-frequency and high-frequency components separately to capture their internal dependencies. Their outputs are then concatenated and further processed by a fused attention head, which captures the cooperative relationships between different frequency bands, calculated as follows:

[0038] (6);

[0039] in, To ensure the consistency of the output sequence, Obtained by downsampling along the time dimension through separable convolution. Then, Segmented into amplitude along the feature dimension and phase , respectively corresponding The first and second halves. The complex-valued spectrum features are reconstructed as follows:

[0040] (7);

[0041] at last, It is restored back to the time domain using iFFT, and its real part is taken as the final frequency domain enhanced representation X.

[0042] Furthermore, the implementation process of the main harmonic guided fusion (DHGF) is as follows:

[0043] Time signals typically manifest as a set of dominant harmonic peaks in the frequency domain, reflecting stable and repetitive motion rhythms. These rhythms encode key cues for perceiving action boundaries. Based on this, this invention proposes a dominant harmonic-guided fusion strategy. This strategy adaptively adjusts the fusion weights of time-domain and frequency-domain features according to the proportion of dominant harmonic energy in the spectrum, thereby enhancing key action responses and effectively improving boundary perception and time-frequency representation quality.

[0044] First, along the time dimension, the features Perform a real-number fast Fourier transform (rFFT) to calculate the corresponding amplitude spectrum:

[0045] (8);

[0046] in, , Indicates the time average. is the output length of rFFT.

[0047] To capture the main harmonic components, this invention first suppresses the first three low-frequency components to avoid interference from the DC component and low-frequency noise. Simultaneously, an upper limit is set. To limit the frequency search range. For each channel d, the main frequency index. Defined in range The frequency position with the largest internal amplitude. Since periodic signals typically exhibit a harmonic structure, this invention further considers its… First harmonic, i.e., an integer multiple of the dominant frequency. To prevent invalid indexing, harmonic positions are truncated to ensure... Therefore, the cumulative harmonic energy of channel d is calculated as follows:

[0048] (9);

[0049] in, Let represent the spectral amplitude of channel d at the i-th harmonic frequency, and The search starts at the third frequency to exclude suppressed DC and low-frequency components.

[0050] Subsequently, the channel-level fusion weights are obtained by normalizing the harmonic energy of each channel to the proportion of the total spectral energy:

[0051] (10);

[0052] in, To stabilize the term and prevent the denominator from being zero, this weight reflects the importance of the main harmonic energy of each channel. Subsequently, the frequency domain representation X and the time domain representation X' are... rs Through weighted residual modulation:

[0053] (11);

[0054] in, , This represents an all-one matrix with the same U-shape. This fusion mechanism highlights the main harmonic responses while preserving fine-grained dynamic information in the time domain, thereby achieving complementary enhancement of frequency and time domain features and improving motion boundary perception and positioning accuracy.

[0055] Finally, the optimization and reasoning process is as follows:

[0056] Optimization. RGB features after time-frequency fusion. With FLOW feature X f The sequences are concatenated and fed into a classifier to generate class activation sequences (CAS). Subsequently, video-level prediction results were obtained through top-k aggregation. :

[0057] , (12);

[0058] This process is supervised by video-level category labels, and the classification loss is calculated as follows:

[0059] (13);

[0060] During training, the model combines multiple loss functions to optimize overall performance. Besides the classification loss L... cls It also includes background loss, action loss, and modality consistency loss. Background loss is used to constrain the class distribution of background segments and reduce false positives; action loss enhances the model's response to action regions; modality consistency loss improves the model's stable alignment with feature distribution. Joint optimization of multiple losses can effectively balance feature quality and detection accuracy.

[0061] Inference. During testing, the model first computes video-level predictions. Then, based on the predefined classification threshold... Filter out the action category set Subsequently, for each predicted action category, the model applies multiple localization thresholds to the class activation sequence S. Regions with consecutive time steps (length ≥ 2) are considered as candidate action segments, denoted as... Its confidence score q n The results are calculated using an internal-external contrast function. Finally, non-maximum suppression (NMS) is used to remove overlap, yielding the final action detection result.

[0062] Experimental Section

[0063] Experimental Datasets: This invention uses two publicly available and challenging video datasets, THUMOS14 and ActivityNet1.3, for testing. THUMOS14 contains 413 unedited videos covering 20 action classes, with approximately 15 action instances per video, featuring complex scenes and diverse time spans. ActivityNet1.3 contains 19,994 unedited videos from 200 action categories, with an average of approximately 1.6 action instances per video.

[0064] Evaluation Metric: Following standard evaluation protocols, mAP at different overlap thresholds (IoU) is used as a metric to evaluate the performance of weakly supervised temporal action localization, denoted as mAP@IoU. Specifically, in the ablation experiments, this detection evaluation metric is uniformly initialized as follows: IoU is set to [0.1:0.1:0.7] on the THUMOS14 dataset and [0.5:0.25:0.95] on the ActivityNet1.3 dataset.

[0065] Experimental setup: This invention divides an unedited video into 16-frame segments and uses an I3D network pre-trained on the Kinetics dataset to extract RGB and FLOW features, where the FLOW map is generated using the TV-L1 algorithm. For fairness, this invention does not introduce any fine-tuning operations on the I3D network.

[0066] During training, the batch size for the THUMOS14 dataset was set to 2, the batch size for the ActivityNet1.3 dataset was set to 64, and the number of training epochs was fixed at 100. Classification threshold. Set to 0.25 on THUMOS14 and 0.1 on ActivityNet1.3; Location threshold The values ​​ranged from [0.1:0.1:1]. Model parameters were updated using the Adam optimizer with a learning rate of 0.0001. All experiments were performed on an NVIDIA RTX A800 GPU using the PyTorch framework.

[0067] This invention evaluates the FSNet method on the THUMOS14 and ActivityNet1.3 datasets and compares it with state-of-the-art localization methods. Furthermore, this invention conducts ablation studies on different components and designs, with specific data analysis as follows.

[0068] Table 1: Contributions of ablation frequency domain coding and interaction unit “FEIU” and main harmonic guided fusion “DHGF” to the model on the THUMOS14 dataset. FEIU comprises two parts: the frequency domain encoder “FE” and the band interaction mechanism “IIM”. The + sign indicates the gain of each setting relative to the baseline.

[0069]

[0070] The effectiveness of ablation of each module is demonstrated. Table 1 shows the ablation experimental results of different components in the proposed model. As can be seen, the complete model achieved the best performance, validating the effectiveness and necessity of each component.

[0071] Starting from the baseline model, adding frequency coding (FE) improved performance by 1.0%, indicating that introducing frequency domain features helps enhance global semantic modeling capabilities. Building on this, replacing simple splicing with main harmonic guided fusion (DHGF) yielded a 3.6% performance gain. DHGF, by utilizing harmonic energy distribution, adaptively balances the contributions of frequency and time domain features, fully leveraging the role of frequency domain features and further validating its importance in effective fusion strategies. Simultaneously, combining it with band interaction modeling (IIM) resulted in a 3.1% improvement, demonstrating that interaction can amplify key frequency responses and capture structural changes.

[0072] Ultimately, the complete model integrating FE, IIM, and DHGF achieved the highest performance gain (+4.2%), particularly at medium to high IoU thresholds (0.5 / 0.7). This fully demonstrates the effectiveness of frequency domain enhancement combined with main harmonic guidance in modeling complex temporal structures and mitigating boundary ambiguity.

[0073] Table 2: The impact of ablation foreground attention on frequency domain modeling quality on the THUMOS14 dataset.

[0074]

[0075] Effectiveness of the Foreground Attention Mechanism. Table 2 explores the effect of introducing foreground attention (FA) before frequency domain modeling. The results show that FA brings a sustained performance improvement, increasing the average mAP from 50.0% to 50.2%, and improving it by 0.2% at a high IoU threshold (0.7). This indicates that introducing FA can effectively suppress background interference, highlight potential action regions, and thus improve the effect of subsequent spectrum modeling.

[0076] Table 3: The impact of different frequency bands on model performance in ablation modeling on the THUMOS14 dataset.

[0077]

[0078] Effectiveness of modeling across different frequency bands. Table 3 examines the impact of modeling different frequency bands in the Frequency Domain Coding and Interaction Unit (FEIU). Modeling only the low-frequency component (w / LF) captures global semantics and stable patterns, but the performance is relatively low (47.2%), indicating that relying solely on global trends is insufficient for accurate boundary modeling. Conversely, modeling only the high-frequency component (w / HF) significantly improves the average performance to 50.1%, demonstrating that high-frequency information plays a crucial role in enhancing boundary awareness and discriminability. Simultaneous modeling of both low and high frequencies (w / LF & HF) further improves the performance to 50.2%, showcasing the complementarity between low-frequency global semantics and high-frequency boundary dynamics. Overall, these results validate the design principle of the FEIU—jointly modeling multiple frequency bands can enhance boundary awareness while modeling the global structure, thereby significantly improving action localization accuracy.

[0079] Table 4: The impact of different time-frequency fusion ablation strategies on model performance on the THUMOS14 dataset.

[0080]

[0081] Effectiveness of different time-frequency fusion strategies. Table 4 compares three common fusion strategies: element-wise addition, feature concatenation, and attention mechanisms, as well as the main harmonic guided fusion (DHGF) method proposed in this invention. Experimental results show that DHGF achieves the best performance at all IoU thresholds, with an average mAP of 50.2%, which is 1.1 percentage points higher than the second-best Concat (49.1%). In particular, at thresholds of 0.1 and 0.5, DHF brings significant gains of 1.8% and 1.7% respectively, demonstrating stronger capabilities in identifying fuzzy boundaries and locating key segments.

[0082] In contrast, Add and Concat do not explicitly model the differences in importance between time-frequency domain features, and while Attention fusion is adaptive, it ignores the structural properties of the spectrum. DHGF, on the other hand, adaptively adjusts the fusion weights through the distribution of main harmonic energy, highlighting stable action patterns and suppressing redundant interference, thus exhibiting stronger discriminativeness and robustness under different threshold settings.

[0083] Table 5: The impact of ablation of different harmonic numbers a on model performance on the THUMOS14 dataset.

[0084]

[0085] Effectiveness of different harmonic orders. Table 5 analyzes the impact of the number of harmonics, *a*, on model performance. Experiments show that model performance first increases and then decreases with the increase of the number of harmonics. When *a*=3, the average mAP reaches a maximum of 50.8%, which is an improvement compared to using only the dominant frequency (*a*=1, 50.5%), indicating that appropriately introducing harmonic information helps to enhance the perception of periodic structures. However, further increasing the number of harmonics (e.g., *a*=5) leads to a decrease in performance. This may be because higher-order harmonics contain too much high-frequency noise or redundant information, interfering with feature fusion. It is evident that reasonably controlling the number of harmonics is crucial for effectively perceiving the dominant frequency structure and achieving robust fusion, with the synergistic effect of the dominant frequency and the two harmonics being most effective when *a*=3.

[0086] Qualitative Visual Analysis. To intuitively verify the effectiveness of the proposed method, Figure 3 shows the analysis of two representative samples in typical scenarios. Figure 3(a) shows that the "Pole Vault" category contains multiple action segments of different lengths. Frequency Domain Coding and Interaction Unit (FEIU) can enhance the activation intensity of each sub-action and provide clearer boundary awareness, demonstrating its advantages in capturing structural patterns and modeling action boundaries. Figure 3(b) shows that the "Front Crawl" category has strong background interference. Guided Harmonic Fusion (DHGF) strengthens the frequency components related to the action through harmonic energy distribution and adaptively combines temporal features to retain fine-grained dynamic information, thereby effectively reducing noise interference and enhancing the action localization capability with boundary awareness.

Claims

1. A weakly supervised temporal action localization method based on a frequency domain sensing network, characterized in that, The implementation steps of this method are as follows: Step 1: Construct a feature extraction module; First, the input unedited video is divided into non-overlapping segments, and a dual-stream network I3D is applied to extract RGB and FLOW features to simultaneously model appearance and motion information; then, the two types of features are stacked and fused along the temporal dimension to form an overall representation of the video, providing an input basis for subsequent modeling; Step 2: Frequency domain coding and interaction unit; First, background suppression is performed on the input RGB features to prevent low-frequency components from dominating spectral energy and masking high-frequency responses; then, the video feature sequence is transformed using a frequency domain encoder to capture global structural semantic information from a frequency perspective; a frequency band interaction mechanism is introduced to distinguish and model different frequency components: low-frequency components represent stable motion patterns, and high-frequency components characterize motion boundaries and rapidly changing features; Step 3: Main harmonic guided fusion; Based on the analysis of the spectral characteristics of the temporal signal, a main harmonic guided fusion is adopted. The main harmonic-guided fusion strategy is used. First, the dominant harmonic peak in the input signal is identified, and the overall rhythmic structure and boundary-related changes are characterized based on the main frequency and its harmonic sequence. The relative energy distribution between each harmonic is mapped to dynamic fusion weights, which are used to adaptively balance the global spectral information of the frequency domain features and the local temporal clues of the time domain features. Through the dynamic fusion mechanism, while maintaining the structural perception capability, the accuracy and robustness of boundary recognition are enhanced, thereby achieving efficient temporal action localization in complex scenes. Step 4: The RGB features and FLOW features after the main harmonic-guided fusion process are spliced ​​along the channel dimension to form a fused time-frequency representation. Finally, the fused features are predicted by a classifier to generate a class activation sequence (CAS). This sequence reflects the correlation between each segment of the video and the action category in the time dimension, providing a high-quality response basis for subsequent action localization and boundary determination.

2. The weakly supervised temporal action localization method based on a frequency domain sensing network according to claim 1, characterized in that, The feature extraction was performed on a dataset containing daily activities and sports events; given unedited videos and corresponding category labels, denoted as... ,in This represents a video segment of length T. It is a one-hot vector of action category C, y c =1 indicates that action category c exists in V, y c =0 indicates none; the goal of WTAL is to use Y as a monitoring signal to locate action instances in V. s n e n c n and p n These represent the start time, end time, prediction category, and confidence score of the action instance, respectively.

3. The weakly supervised temporal action localization method based on a frequency domain sensing network according to claim 1, characterized in that, Using a pre-trained I3D model, extract data from video clips v t Extracting RGB features and FLOW features Where D represents the feature dimension; subsequently, all segment features are concatenated along the time dimension to form a complete video feature sequence, represented as... ， ； These features will serve as input data for subsequent steps.

4. The weakly supervised temporal action localization method based on a frequency domain sensing network according to claim 1, characterized in that, The implementation process of Frequency Domain Coding and Interactive Unit (FEIU) is as follows: The frequency domain perspective can explicitly reveal the periodicity and stability patterns in motion signals, providing a stronger representation for modeling the internal structure and boundary changes of foreground motion; a frequency domain modeling mechanism is introduced to extract stable response patterns from the frequency space; to highlight the representation of the motion region, in X... r A foreground attention mechanism is introduced, which generates a foreground attention map through convolutional layers and a sigmoid function. The input features are then weighted to obtain foreground enhancement features: (1); where, This represents an element-wise multiplication operation; it enhances feature X. rs The spectrum is then transformed to the complex domain and projected onto the frequency space using a Fast Fourier Transform (FFT). To improve the stability of the spectral modeling, the obtained amplitude spectrum is centered and normalized to generate a structured frequency representation. ； (2); where, Represents the complex number mapping function, Indicates complex number centralization operations, and This represents the complex normalization operation; the resulting spectral representation F is input to a complex Transformer to model dependencies within the spectral range; firstly, a complex-valued learnable matrix is ​​used... ， and The F vector is linearly mapped to the query vector Q, the key vector K, and the value vector V; this is achieved via dot product attention blocks. To model contextual relevance, residual connections and complex normalization are combined to maintain a stable feature distribution; the representation is further refined using a complex feedforward network and another set of residual normalizations; the overall calculation formula is as follows: (3); among which, It is a two-layer complex feedforward network; the encoding integrates global spectral dependencies, generating enhanced frequency domain feature representations. 。 5. The weakly supervised temporal action localization method based on a frequency domain sensing network according to claim 4, characterized in that, Frequency band interaction mechanism: Different frequency bands carry heterogeneous semantic information: low-frequency components usually reflect stable action patterns, while high-frequency components correspond to boundary changes and action transitions; in order to make full use of the complementary characteristics of the two, a frequency band interaction mechanism is designed.

6. The weakly supervised temporal action localization method based on a frequency domain sensing network according to claim 4, characterized in that, The frequency band interaction mechanism is implemented as follows: Frequency representation... Converting to polar coordinates, the complex features are decomposed into amplitude and phase components, and these components are concatenated to align with the real-valued attention structure; defined as follows: (4); among which, ， This indicates a splicing operation. Represents the modulus of a complex number. The argument function of a complex number.

7. The weakly supervised temporal action localization method based on a frequency domain sensing network according to claim 4, characterized in that, Introduce an adaptive gating mechanism, based on Dynamically generated band mask It is implemented by convolutional layers and a sigmoid function; based on this mask, the spectrum is adaptively decomposed into low-frequency and high-frequency bands: , (5); among which, It is a full tensor with the same shape as M; subsequently, independent multi-head attention heads are used to model the low-frequency and high-frequency components separately, the outputs are concatenated and further processed by a fused attention head, which captures the cooperative relationships between different frequency bands, calculated as follows: (6); among which, To ensure the consistency of the output sequence, Obtained by downsampling along the time dimension through separable convolution. Then, Segmented into amplitude along the feature dimension and phase , respectively corresponding The first and second halves; the complex-valued spectrum features are reconstructed as: (7); Finally, It is restored back to the time domain using iFFT, and its real part is taken as the final frequency domain enhanced representation X.

8. The weakly supervised temporal action localization method based on a frequency domain sensing network according to claim 1, characterized in that, The implementation process of the main harmonic guided fusion is as follows: Time signals are usually represented by a set of dominant harmonic peaks in the frequency domain. Using a main harmonic guided fusion strategy, the fusion weights of time-domain and frequency-domain features are adaptively adjusted according to the proportion of main harmonic energy in the spectrum, thereby strengthening key action responses and effectively improving boundary perception and time-frequency representation quality. First, features are fused along the time dimension... Perform a real-number fast Fourier transform (rFFT) to calculate the corresponding amplitude spectrum: （8）； in, ， Indicates the time average. The output length of the rFFT is set; the upper limit is set to capture the main harmonic component and suppress the first three low-frequency components. To limit the frequency search range; for each channel d, the main frequency index Defined in the range The frequency location with the largest internal amplitude; consider its First harmonic, i.e., an integer multiple of the dominant frequency. ， To prevent invalid indexes, harmonic positions are truncated to ensure... The cumulative harmonic energy of channel d is calculated as follows: (9); among which, Let represent the spectral amplitude of channel d at the i-th harmonic frequency, and The search begins at the third frequency to exclude suppressed DC and low-frequency components; subsequently, the channel-level fusion weights are obtained by normalizing the harmonic energy of each channel to a proportion of the total spectral energy. (10); among which, To stabilize the term and prevent the denominator from being zero; this weight reflects the importance of the main harmonic energy of each channel; subsequently, the frequency domain representation X and the time domain representation X... rs Through weighted residual modulation: (11); among which, ， It represents a matrix of all 1s with the same shape as U.

9. The weakly supervised temporal action localization method based on a frequency domain sensing network according to claim 1, characterized in that, The optimization and inference process is as follows: The optimization process is: to integrate the RGB features after time-frequency fusion. With FLOW feature X f The sequences are concatenated and fed into a classifier to generate class activation sequences (CAS). Subsequently, video-level prediction results were obtained through top-k aggregation. ： ， (12); The classification loss is calculated as follows, supervised by video-level category labels: (13); During training, the model combines multiple loss functions to optimize overall performance; in addition to the classification loss L cls It also includes background loss, action loss, and modality consistency loss; background loss is used to constrain the category distribution of background segments and reduce false positives; action loss enhances the model's response to action regions; modality consistency loss improves the model's stable alignment with feature distribution; joint optimization of multiple losses can effectively balance feature quality and detection accuracy; the reasoning process is as follows: during testing, video-level predictions are first calculated. Then, based on the predefined classification threshold... Filter out the action category set Subsequently, for each predicted action category, the model applies multiple localization thresholds to the class activation sequence S. The region of consecutive time steps is considered as a candidate action segment, denoted as Its confidence score q n The motion detection results are obtained by calculating the internal and external contrast functions and then de-overlapping the results using non-maximum suppression (NMS).

Images