A time-series action detection method based on the interaction of frequency domain and time domain information

Abstract

This invention relates to a temporal action detection method based on the interaction of frequency and temporal domain information, belonging to the field of computer vision technology. It includes the following steps: determining a temporal action detection video and extracting pre-training video features; preprocessing the pre-training video features; fusing the processed features through frequency and temporal domain interaction to obtain fused features; adding the fused features to a feature pyramid; downsampling the fused features in the pyramid; repeating the fusion operation to obtain the final feature pyramid; classifying the feature pyramid using a semantically enhanced classification head to obtain a confidence score and decode the action category; regressing using a localization-refined regression head to obtain the start and end times of the action segment; combining the confidence score, action category, and start and end times to decode the candidate set: during training, positive and negative samples are selected, and a loss function is used to constrain the model; during inference, the final prediction result is determined based on a confidence threshold and non-maximum suppression. This invention can improve the accuracy of classifying and locating actions in videos.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, specifically relating to a temporal action detection method based on the interaction of frequency domain and time domain information. Background Technology

[0002] The goal of temporal action detection is to accurately locate and identify the start and end times of all actions in unedited video and their corresponding action category labels. With the rapid development of multimedia technology, a massive amount of video data has emerged on the internet, making temporal action detection technology increasingly important. This task has significant practical value in the field of industrial video analytics. For example, in intelligent security, it is necessary to accurately locate the critical moments of abnormal behavior from hours of surveillance footage; in sports event analysis, it is necessary to extract the precise time intervals of specific tactical actions.

[0003] The current mainstream methods are mainly divided into two parts: two-stage methods and single-stage methods. Two-stage methods include two stages: temporal action proposal generation and action classification. These methods generate video clips that may contain actions as candidate proposals through independent processes, and then classify each valid action proposal and optimize its temporal boundaries. The characteristic of two-stage methods requiring two independent stages significantly increases the computational cost and complexity, making the whole system very cumbersome and difficult to meet real-time requirements, thus limiting its practical application in industrial production. In order to overcome the inherent technical bottleneck of two-stage methods, some works have begun to try to complete the temporal action detection task using an end-to-end framework, that is, to complete the classification and localization of actions in one stage. These methods delete the candidate proposal generation stage, simplify the action detection process, and make it faster and more efficient. Although the current methods have made some progress, there are still two shortcomings: (1) they rely only on single temporal information and do not consider feature modeling from the perspective of frequency domain; (2) they focus on the feature encoding process, while the classification and localization modules are too simple, with the same structure and the same input features.

[0004] Therefore, this invention provides a timing action detection method based on the interaction of frequency domain and time domain information to solve the above problems. Summary of the Invention

[0005] To achieve the above objectives, the present invention employs the following technical solution:

[0006] This invention provides a time-series action detection method based on the interaction of frequency domain and time domain information, comprising the following steps:

[0007] S1. Determine the video to be detected for temporal action detection and extract pre-training video features from the video.

[0008] S2. Preprocess the extracted pre-trained video features, transforming them into a specified shape to obtain the processed video features;

[0009] S3. Input the processed video features into the frequency domain and time domain information interaction and fusion module to mine the frequency domain and time domain information, obtain the fused features, and add the fused features to the feature pyramid;

[0010] S4. Downsample the fused features in the feature pyramid, and then repeat the operation in S3 to obtain the final feature pyramid.

[0011] S5. Input the feature pyramid into the semantic enhancement classification head and perform classification operations on each time point of each layer of features to obtain the confidence score of each action category to which the time point belongs. Decode the confidence score to obtain the action category.

[0012] S6. Input the feature pyramid into the localization refinement regression head and perform regression operation on each time point of each feature layer to obtain the start time and end time of the action segment to which the time point belongs.

[0013] S7. Decode the candidate set based on the confidence score, action category, and start and end times of the action segment. For the training process, select positive and negative samples and constrain the model through the loss function. For the inference process, determine the final prediction result through the confidence threshold and non-maximum suppression.

[0014] Furthermore, raw video feature extraction suffers from low efficiency and poor generalization ability, and building a feature extraction network from scratch is costly and difficult to adapt to the action detection needs of different scenarios. This invention extracts pre-trained video features from videos to be used for temporal action detection through a pre-trained action recognition network. By reusing prior knowledge trained on large-scale data, the pre-trained action recognition network eliminates the need to retrain the feature extraction module, significantly reducing computational costs. At the same time, pre-trained features have stronger generalization ability and can adapt to video data from different scenarios such as surveillance and sports events.

[0015] Furthermore, the pre-trained features of different videos suffer from inconsistent temporal dimensions and mismatched channel dimensions, leading to unstable processing by subsequent modules. This invention fixes the temporal dimension of pre-trained video features through padding and truncation, resulting in pre-trained video features with fixed temporal dimensions, ensuring that all video features are aligned in the temporal dimension. Then, it fixes the channel dimension of the pre-trained video features with fixed temporal dimensions through masked convolution and layer normalization, obtaining embedded features. This ensures that the features fully adapt to the input requirements of the subsequent frequency-temporal fusion module, avoiding model training instability or accuracy loss due to inconsistent dimensions.

[0016] Furthermore, existing methods rely solely on single time-domain information, failing to capture the macroscopic trend (low frequency), main rhythm (mid frequency), and switching details (high frequency) of actions in the frequency domain; simultaneously, the feature representation is singular, lacking multi-dimensional information fusion, resulting in incomplete characterization of action features; this invention introduces frequency domain feature extraction, specifically including:

[0017] Frequency domain information is extracted by performing a Fast Fourier Transform on the embedded features to obtain frequency domain signal features. The frequency domain signals are then segmented according to their magnitude to obtain low-frequency, mid-frequency, and high-frequency signals. These signals are then processed by an Inverse Fast Fourier Transform to obtain low-frequency, mid-frequency, and high-frequency signal features. The signal features of the three frequency bands are then weighted and fused to obtain a first fused feature. This first fused feature is then interacted with the embedded features to obtain an interactive feature. This approach can supplement the frequency dimension information that cannot be covered in the time domain.

[0018] Furthermore, the temporal feature extraction process specifically includes:

[0019] The interaction features are input into the Mamba module to obtain global features; the interaction features are input into the multi-scale convolution module to obtain local features; the global features, local features, and interaction features are fused to obtain a second fused feature; the second fused feature is processed by a multilayer perceptron to obtain a fused feature; the final fused feature is added to the feature pyramid.

[0020] Furthermore, single-step fusion can only cover features at a single scale, failing to adapt to actions of different durations, resulting in low accuracy in multi-scale action detection. This invention solves the problem of poor adaptability of single-scale features by accurately detecting actions of different durations through a multi-layer pyramid. Specifically, it includes:

[0021] The features at the highest level of the feature pyramid Perform a 2x downsampling to obtain the downsampling features. Then the obtained downsampled features The input is fed into the frequency domain and time domain information interaction and fusion module, and the process in S3 is repeated to extract the frequency domain and time domain fusion features. .

[0022] Furthermore, existing methods use only single-level features in their classification modules, resulting in insufficient semantic information and low accuracy in action category judgment. Moreover, the classification and localization modules share features, making it impossible to specifically optimize the semantic expression required for classification. This invention addresses this by applying features to each level of the feature pyramid. , will upper-layer features Perform a 2x upsampling to obtain the same as Features with the same dimension are convolved and then combined with... By fusing the features, semantically enhanced features are obtained. It supplements the missing global semantic information of single-level features; and enhances semantic features. The data is input into the classification header for classification, resulting in the first... Layered pyramid time point Confidence scores for each category The action category is decoded based on the output confidence score; the classification head outputs a confidence score based on semantic enhancement features, which can improve the accuracy of decoding the action category.

[0023] Furthermore, existing methods use only single-level features in their localization module, resulting in coarse temporal granularity and difficulty in accurately predicting the start and end times of actions; moreover, they share features with the classification module, making it impossible to specifically optimize the temporal details required for localization; this invention addresses this by applying features to each level of the feature pyramid. , lower layer features Perform a 2x downsampling to obtain the same as Features with the same dimension are convolved and then combined with... By fusing the features, we can obtain refined positioning characteristics. It supplemented the temporal details of missing single-level features and refined the localization features. Inputting this into the regression header yields the distance between the current time and the start time of the action. The distance between the current time and the end time of the action The action start time boundary is obtained by decoding based on the number of feature pyramid layers. and action end time boundary The regression head outputs the distance between the current time and the start and end times of the action based on the localization refinement features. Then, by decoding the boundary through the pyramid layers, the prediction accuracy of the start and end times of the action can be improved.

[0024] Furthermore, step S7 specifically includes: setting each time point The action category and the start and end times of the action segment are combined to form a candidate action segment set. During the training process, the model is constrained by a loss function. During the inference process, action segments with confidence scores below a certain threshold are filtered out, and the final predicted action segment set is determined by a non-maximum suppression algorithm.

[0025] The advantages of this invention are:

[0026] This invention fuses the frequency domain and temporal features of video clips. During frequency domain feature extraction, multi-band decomposition is performed on the frequency information. The macroscopic direction of actions, the main body of the action, the transitions between actions, and noise information are learned from the frequency domain information. Finally, the frequency domain and temporal features are fused to form a rich feature representation encompassing two different perspectives. This invention achieves a more powerful decoding process through a semantically enhanced classification head and a localization-refined regression head. This imbues the features input to the classification head with more semantic features and the features input to the regression head with more refined temporal features, helping the model achieve more robust classification and localization results. Through comprehensive learning of both frequency domain and temporal information, this invention can more accurately classify and localize actions in videos. Attached Figure Description

[0027] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof.

[0028] Figure 1 This is a flowchart of the steps of the method of the present invention;

[0029] Figure 2 This is an experimental comparison diagram between the method of this invention and the baseline model. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0031] Example 1

[0032] In this embodiment, as Figure 1 As shown, this invention provides a time-series action detection method based on the interaction of frequency domain and time domain information, the specific steps of which include:

[0033] S1. Determine the video to be detected for temporal action detection and extract pre-training video features from the video.

[0034] Specifically, pre-trained action recognition networks extract pre-trained video features from the videos to be used for temporal action detection. The pre-trained action recognition network includes... Interactive 3D model, video self-supervised pre-trained model VideoMAEv2, and multimodal video understanding large model InternVideo2; preferably, the pre-trained video features extracted by the InternVideo2 model have a channel dimension of 3200.

[0035] S2. Preprocess the extracted pre-trained video features, transforming them into a specified shape to obtain the processed video features;

[0036] Specifically, the temporal dimension of the pre-trained video features is fixed by padding and truncation to obtain the temporally fixed pre-trained video features; the channel dimension of the temporally fixed pre-trained video features is fixed by mask convolution and layer normalization to obtain the embedded features; the specific process is as follows:

[0037] The extracted pre-trained video features are fixed to 2304 dimensions in temporal dimension. For features shorter than 2304 dimensions, they are padded with 0s to reach 2304 dimensions. For features longer than 2304 dimensions, they are truncated to fix them to 2304 dimensions, resulting in pre-trained video features with fixed temporal dimensions. By using masked convolution and layer normalization, the pre-trained video features with fixed temporal dimensions are transformed. With the channel dimension fixed at 512, the embedded features are obtained, and the calculation process is as follows:

[0038] ,

[0039] in, This represents the pre-trained video features with a fixed temporal dimension. Represents embedded features, and These represent layer normalization operation and one-dimensional mask convolution operation, respectively.

[0040] S3. Input the processed video features into the frequency domain and time domain information interaction and fusion module to mine the frequency domain and time domain information, obtain the fused features, and add the fused features to the feature pyramid;

[0041] The frequency domain feature extraction process specifically includes:

[0042] Frequency domain information is extracted by performing a Fast Fourier Transform on the embedded features to obtain frequency domain signal features. The frequency domain signals are then segmented according to their magnitude to obtain low-frequency, mid-frequency, and high-frequency signals. The low-frequency, mid-frequency, and high-frequency signals are then processed by an Inverse Fast Fourier Transform to obtain low-frequency signal features, mid-frequency signal features, and high-frequency signal features. The signal features of the three frequency bands are then weighted and fused to obtain the first fused feature. The first fused feature is then interacted with the embedded features to obtain the interactive feature.

[0043] Specifically, the low-frequency signal is obtained by segmenting the signal according to its magnitude. Intermediate frequency signal and high-frequency signals Low-frequency signals This indicates the frequency domain signal characteristics. The frequency value The result after setting the remaining parts to zero represents the macroscopic trend and posture of the action; intermediate frequency signal This indicates the frequency domain signal characteristics. The frequency value The result after setting the remaining parts to zero represents the main body and rhythm of the action; high-frequency signals This indicates the frequency domain signal characteristics. The frequency value The result after setting the remaining parts to zero represents the switching between actions and noise information; the calculation process is expressed by the following formula:

[0044] ,

[0045] ,

[0046] ,

[0047] ,

[0048] ,

[0049] ,

[0050] in, Indicates Fast Fourier Transform; Represents the characteristics of frequency domain signals; Indicates a low-frequency signal; Indicates the characteristics of low-frequency signals; Indicates intermediate frequency signal; Indicates the characteristics of intermediate frequency signals; Indicates a high-frequency signal; Indicates the characteristics of high-frequency signals; Indicates the first fusion feature; Indicates the first weighting coefficient; Indicates the second weighting coefficient; Indicates the third weighting coefficient; ; This represents the first learnable parameter; Indicates interactive features.

[0051] The temporal feature extraction process specifically includes:

[0052] The interaction features are input into the Mamba module to obtain global features; the interaction features are input into the multi-scale convolution module to obtain local features; the global features, local features, and interaction features are fused to obtain the second fused feature; the formula is expressed as follows:

[0053] ,

[0054] ,

[0055] ,

[0056] in, Represents global features; Mamba represents a state-space model; Indicates a fully connected layer; This indicates an element-wise multiplication operation on two features; Indicates local features; This represents a one-dimensional convolution with a kernel of 3. This represents a one-dimensional convolution with a kernel of 7. This represents a gating unit, which consists of a depthwise separable convolution. Indicates the second fusion feature;

[0057] The second fused feature is processed by a multilayer perceptron to obtain the fused feature; the final fused feature is then added to the feature pyramid.

[0058] The formula for obtaining the fused features is expressed as follows:

[0059] ,

[0060] in, Indicates fusion characteristics; This represents a multilayer perceptron, which consists of two one-dimensional convolutions and one... The system consists of activation functions, where the first convolution increases the channel dimension of the input features from 512 to 1024, and the second convolution reduces the channel dimension of the input features from 1024 to 512. This indicates the group normalization operation.

[0061] S4. Downsample the fused features in the feature pyramid, and then repeat the operation in S3 to obtain the final feature pyramid.

[0062] Specifically, the features at the highest level of the feature pyramid Perform a 2x downsampling to obtain the downsampling features. Then the obtained downsampled features The input is fed into the frequency domain and time domain information interaction and fusion module, and the process in S3 is repeated to extract the frequency domain and time domain fusion features. The formula is expressed as follows:

[0063] ,

[0064] ,

[0065] in, This indicates a max pooling operation. This indicates that the frequency domain and time domain information fusion module in S3 repeats the above process until the final feature pyramid is obtained. Features in the feature pyramid The channel dimension is 512, and the time dimension is... .

[0066] S5. Input the feature pyramid into the semantic enhancement classification head and perform classification operations on each time point of each layer of features to obtain the confidence score of each action category to which the time point belongs. Decode the confidence score to obtain the action category.

[0067] Specifically, for each layer of features in the feature pyramid , will upper-layer features Perform a 2x upsampling to obtain the same as Features with the same dimension are convolved and then combined with... By fusing the features, semantically enhanced features are obtained. The formula is expressed as follows:

[0068] ,

[0069] ,

[0070] in, This represents a one-dimensional convolution with a kernel of 1. Represents linear interpolation operations; semantically enhanced features The data is input into the classification header for classification, resulting in the first... Layered pyramid time point Confidence scores for each category The action category is decoded based on the output confidence score; the formula is as follows:

[0071] ,

[0072] ,

[0073]

[0074] in, Indicates activation function Operation; Indicates activation function operate; Indicates the first characteristic; for The result after concatenation along the channel dimension represents the confidence score of each pyramid time point t belonging to each category; This indicates a convolution operation with a kernel of 3 and 20 output channels equal to the number of action categories. This indicates a convolution operation with a kernel of 1. This indicates the operation of finding the index of the maximum value; Indicates at a point in time Predicted action categories;

[0075] S6. Input the feature pyramid into the localization refinement regression head and perform regression operation on each time point of each feature layer to obtain the start time and end time of the action segment to which the time point belongs.

[0076] Specifically, for each layer of features in the feature pyramid , lower layer features Perform a 2x downsampling to obtain the same as Features with the same dimension are convolved and then combined with... By fusing the features, we can obtain refined positioning characteristics. The formula is expressed as follows:

[0077] ,

[0078] ,

[0079] Refine positioning features Inputting this into the regression header yields the distance between the current time and the start time of the action. The distance between the current time and the end time of the action The calculation process is as follows:

[0080] ,

[0081] ,

[0082] in, Indicates the second characteristic; This indicates a convolution operation with a kernel of 3 and 2 output channels;

[0083] The action start time boundary is obtained by decoding based on the feature pyramid layer number. and action end time boundary The calculation process is as follows:

[0084] ,

[0085] ,

[0086] in, Indicates the number of levels in the pyramid; Indicates the current time step; Indicates the boundary of the action start time; Indicates the boundary of the action's end time.

[0087] S7. Decode the candidate set based on the confidence score, action category, and start and end times of the action segment. For the training process, select positive and negative samples and constrain the model through the loss function. For the inference process, determine the final prediction result through the confidence threshold and non-maximum suppression.

[0088] Specifically, each time point The action category, the start time, and the end time of the action segment are combined to form a candidate action segment set. ,in ;

[0089] For the training process, the model is constrained by the loss function, and the calculation process is as follows:

[0090] ,

[0091] in, Represents the relationship between real action instances and predicted action fragments. , and These represent the number of positive samples and the number of negative samples determined by the center sampling method, respectively. Represents the focus loss function. Indicates IoU loss;

[0092] For the inference process, action segments with confidence scores below a certain threshold are filtered out, and the final set of predicted action segments is determined by a non-maximum suppression algorithm.

[0093] Example 2

[0094] This invention uses a pre-trained InternVideo2 model to extract features from the THUMOS14 dataset and a pre-trained R(2+1)D model to extract features from the ActivityNet-1.3 dataset. All experiments in this chapter were performed on a GPU platform equipped with 160 GB of memory, an Intel Xeon Gold 6226 CPU @ 2.70 GHz, and an NVIDIA A100 GPU. For the THUMOS14 dataset, the initial learning rate was set to 0.0001, and training was conducted for 40 epochs; for the ActivityNet-1.3 dataset, the initial learning rate was set to 0.001, and training was conducted for 15 epochs.

[0095] The evaluation metric is the mean precision (mAP) at different time intersection-to-union (tIoU) thresholds. Consistent with the TriDet method, for the THUMOS14 dataset, the mAP at tIoU thresholds of [0.3:0.1:0.7] is used as the evaluation metric; for the ActivityNet-1.3 dataset, the mAP at tIoU thresholds of [0.5:0.05:0.95] is used as the evaluation metric.

[0096] The comparison between the method of this invention and the baseline method (i.e., the model that does not have a frequency domain information extraction module and uses ordinary classification head and regression head in the decoding process) on the THUMOS14 dataset is shown in Table 1, and the comparison between the method of this invention and the baseline method on the ActivityNet-1.3 dataset is shown in Table 2.

[0097] Table 1 Comparison of results between the method of this invention and the baseline method on the THUMOS14 dataset.

[0098]

[0099] For the THUMOS14 dataset, the method of this invention achieves an average mAP of 74.2% with tIoU thresholds of [0.3:0.1:0.7], and an mAP of 53.0% with tIoU=0.7. The frequency-time domain fusion and localization refinement regression head of this invention maintains high accuracy even under stringent localization requirements, solving the problem of inaccurate localization under high tIoU conditions in baseline models.

[0100] Table 2 Comparison of results between the method of this invention and the baseline method on the ActivityNet-1.3 dataset.

[0101]

[0102] For the THUMOS14 dataset, the method of this invention achieves an average mAP of 38.4% with tIoU thresholds of [0.5:0.05:0.95], which is significantly better than the baseline model.

[0103] Example 3

[0104] To highlight the advantages of the method in this invention, in this embodiment, such as Figure 2 The figure shows an example of the retrieval results of the method of the present invention and the baseline method on the THUMOS14 dataset. The black box part is the real label. ① represents the action detection and localization result of the baseline model, and ② represents the action detection and localization result of the method of the present invention. As can be seen from the figure, the technical solution of the present invention can achieve results that are closer to the correct answer.

[0105] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A time-series action detection method based on the interaction of frequency domain and time domain information, characterized in that, Includes the following steps: S1. Determine the video to be detected for temporal action detection and extract pre-training video features from the video. S2. Preprocess the extracted pre-trained video features, transforming them into a specified shape to obtain the processed video features; S3. Input the processed video features into the frequency domain and time domain information interaction and fusion module to mine the frequency domain and time domain information, obtain the fused features, and add the fused features to the feature pyramid; The specific steps include: Frequency domain feature extraction process: Frequency domain information is extracted by performing a Fast Fourier Transform on the embedded features to obtain frequency domain signal features. The frequency domain signals are then segmented according to their magnitude to obtain low-frequency, mid-frequency, and high-frequency signals. These signals are then processed by an Inverse Fast Fourier Transform to obtain low-frequency, mid-frequency, and high-frequency signal features. The signal features of the three frequency bands are then weighted and fused to obtain the first fused feature. Finally, the first fused feature is interacted with the embedded features to obtain the interactive feature. Temporal feature extraction process: The interaction features are input into the Mamba module to obtain global features; the interaction features are input into the multi-scale convolution module to obtain local features; the global features, local features, and interaction features are fused to obtain a second fused feature; the second fused feature is processed by a multilayer perceptron to obtain a fused feature; the final fused feature is added to the feature pyramid. S4. Downsample the fused features in the feature pyramid, then repeat the operation in S3 to obtain the final feature pyramid; the specific steps include: The features at the highest level of the feature pyramid Perform a 2x downsampling to obtain the downsampling features. Then the obtained downsampled features The input is fed into the frequency domain and time domain information interaction and fusion module, and the process in S3 is repeated to extract the frequency domain and time domain fusion features. ; S5. Input the feature pyramid into the semantic enhancement classification head and perform classification operations on each feature layer at each time point to obtain the confidence score of each action category at that time point. Decode the confidence score to obtain the action category; the specific steps include: For each layer of features in the feature pyramid , will upper-layer features Perform a 2x upsampling to obtain the same as Features with the same dimension are convolved and then combined with... By fusing the features, semantically enhanced features are obtained. ; semantic enhancement features The data is input into the classification header for classification, resulting in the first... Layered pyramid time point Confidence scores for each category Decode the action category based on the output confidence score, where, Indicates a point in time Predicted action categories; S6. Input the feature pyramid into the localization refinement regression head and perform regression operation on each time point of each feature layer to obtain the start and end times of the action segment to which that time point belongs; the specific steps include: For each layer of features in the feature pyramid , lower layer features Perform a 2x downsampling to obtain the same as Features with the same dimension are convolved and then combined with... By fusing the features, we can obtain refined positioning characteristics. ; refine positioning features Inputting this into the regression header yields the distance between the current time and the start time of the action. The distance between the current time and the end time of the action The action start time boundary is obtained by decoding based on the number of feature pyramid layers. and action end time boundary ; S7. Decode the candidate set based on the confidence score, action category, and start and end times of the action segment. For the training process, select positive and negative samples and constrain the model through the loss function. For the inference process, determine the final prediction result through the confidence threshold and non-maximum suppression.

2. The timing action detection method based on frequency domain and time domain information interaction according to claim 1, characterized in that, In S1, a pre-trained action recognition network extracts pre-trained video features from the video for which temporal action detection is to be performed.

3. The timing action detection method based on frequency domain and time domain information interaction according to claim 2, characterized in that, S2 specifically includes: The temporal dimension of the pre-trained video features is fixed by padding and truncation to obtain the temporally fixed pre-trained video features; the channel dimension of the temporally fixed pre-trained video features is fixed by mask convolution and layer normalization to obtain the embedded features.

4. The timing action detection method based on frequency domain and time domain information interaction according to claim 3, characterized in that, S7 specifically includes: Each time point The action category and the start and end times of the action segment are combined to form a candidate action segment set. During the training process, the model is constrained by a loss function. During the inference process, action segments with confidence scores below the confidence threshold are filtered out, and the final predicted action segment set is determined by a non-maximum suppression algorithm.

Images