An end-to-end timing action detection method, electronic equipment and storage medium

Abstract

An end-to-end timing action detection method, an electronic device and a storage medium, based on a ViT model, a timing action detection model ViT-TAD is constructed for completing the timing action detection task depending on long-time modeling, first, a long-time video segment is divided into multiple short-time segments, and is sent into the ViT-TAD for timing action detection, two information propagation modules in the ViT backbone network are used to guide the detection model to learn global timing semantic information, and then a long-time timing action detector is used for action detection, and the internal parameters of the information propagation module are updated in the training through an end-to-end training mode.The present application has the characteristics of strong adaptability, direct detection process and accurate detection result in the timing action detection task, and compared with the existing method, the present application has better performance in the mainstream data set and actual application.

Description

Technical Field

[0001] This invention belongs to the field of computer software technology and relates to time-series action detection technology. Specifically, it is an end-to-end time-series action detection method, electronic device and storage medium that adapts a short-term Transformer to a long-term action detection task. Background Technology

[0002] The goal of temporal action detection is to find the start and end boundaries of all actions in a video of variable length, as well as the category of each action. Existing temporal action detectors employ a two-stage approach: the first stage uses a backbone network pre-trained on action recognition tasks to extract temporal features; the second stage uses the temporal action detector to decode the temporal features extracted in the first stage, thereby obtaining the predicted action boundaries and categories. With the development of Transformer, its global and dynamic modeling capabilities have led to an increasing number of works applying it to backbone networks to extract finer-grained temporal features. In temporal action detection, "short-time" represents short-duration actions, with an action length range consistent with the sampling of video segments in action recognition, approximately 2 seconds; "long-time" represents longer actions in temporal action detection, ranging from several seconds to tens or even hundreds of seconds. However, due to computational limitations, existing works only treat the Transformer-based backbone network as an offline short-time feature extractor, resulting in an underutilization of Transformer's global modeling capabilities. Meanwhile, most existing backbone networks based on Transformer design are complex and multi-scale, making it difficult to utilize them in action recognition tasks and unable to leverage the results of large-scale model pre-training.

[0003] ViT (Vision Transformer), proposed by the Google team in 2020, is a model that applies Transformer to image classification. It achieved excellent results in image classification tasks with less computational resources compared to CNNs. This method was then transferred to the field of action recognition and achieved good results in action recognition tasks. However, action recognition tasks model short-term action features and identify action categories in short-term videos of only a few seconds, while many downstream temporal tasks rely on understanding longer video content (tens of seconds or even tens of minutes). Few works have explored how to apply this short-term modeling, simple structure, and single-scale ViT model to temporal action detection tasks that rely on long-term modeling. Summary of the Invention

[0004] The problem this invention aims to solve is: how to apply the short-time modeling ViT backbone network to long-term temporal action detection tasks. Transformers possess powerful global modeling capabilities, making them suitable for tasks like temporal action detection that rely on temporal modeling. However, due to limitations in computational resources and the current practice of short-time pre-training of Transformers for action recognition tasks, existing work cannot fully utilize the global modeling capabilities of Transformers, especially the ViT backbone network. While it benefits from the use of large models and has achieved excellent results in action recognition, it has not been fully utilized for temporal action detection tasks with long-term modeling of tens of seconds or even tens of minutes.

[0005] The technical solution of this invention is as follows: an end-to-end temporal action detection method, which constructs a temporal action detection model ViT-TAD based on the ViT model to complete the temporal action detection task that relies on long-term modeling. First, the long-term video segment is divided into multiple short-term segments, each short-term segment corresponding to a ViT backbone network. Each ViT backbone network contains n hierarchical modules. An inner-backbone information propagation module is inserted into the ViT backbone network. The insertion position and number of inner-backbone information propagation modules are determined based on the results of ablation experiments. A post-backbone information propagation module is inserted after the ViT backbone network. Then, a long-term temporal action detector is used for temporal action detection. The inserted information propagation modules are used to guide the detection model to learn global temporal semantic information, and the internal parameters of the information propagation modules are updated during training through an end-to-end training method. Specifically, the two information propagation modules are:

[0006] The inner-backbone information propagation module: For short-time feature sequences modeled from short-time modules in each backbone network, these short-time feature sequences are concatenated along the temporal dimension to form long-time features X∈R. (W×H)×T×C X = concat(x1, x2, ... x W×H ), where the feature dimension is C, the feature width is W, the height is H, the time series length is T, and x t Representing the long-term temporal features at each spatial location, t = 1, 2, ..., W × H, the inner-backbone information propagation module uses a self-attention mechanism, with the formula Y = X + attention(X). For the calculation of attention, each temporal feature x is first constructed. t The query q, key k, and value v are used, followed by self-attention calculation, as shown in the following formula:

[0007] q = x t ·w q k=xt ·w k v = x t ·w v

[0008]

[0009] These are the weight parameters q, k, and v of the self-attention mechanism, respectively, and C. q =C k =C / m, where m is the number of attention heads. After calculation within the inner-backbone information propagation module, we obtain Y = concat(y1, y2, ... y W×H )∈R (W×H)×T×C Feature Y is the feature that has undergone global temporal interaction;

[0010] The post-backbone information propagation module: For multiple short-time features modeled by the ViT backbone network, it concatenates them along the temporal dimension into a long-time feature X′∈R. T×C The operation is performed using the Transformer, which consists of a self-attention layer and a feedforward network, Y′=transformer(X′), and the specific formula is as follows:

[0011] X″ = X′ + attention(X′)

[0012] Y′=X″+FFN(X″)

[0013] The calculation of attention is consistent with the inner-backbone information propagation module. FFN consists of two fully connected layers, which are used to perform nonlinear transformations on each temporal position for feature interaction. The resulting Y′ is input to the long-term temporal action detector for action detection.

[0014] As a preferred embodiment, the present invention evenly inserts four inner-backbone information propagation modules into the ViT backbone network.

[0015] Furthermore, the implementation of the temporal action detection model ViT-TAD is as follows:

[0016] 1) Video frame generation stage: Determine the timing action detection dataset to be used. For videos in the THUMOS14 dataset, extract frames at a set frame rate to obtain video frames. Extract a video frame sequence of T frames as input samples for long-duration video segments. For videos in the ActivityNet dataset, extract video segments of length T seconds as input samples for long-duration video segments. Divide the long-duration video segment input samples into multiple short-duration segments.

[0017] 2) Network configuration phase: Configure the temporal action detection model ViT-TAD, which involves configuring a long-term temporal action detector based on the dataset. For the THUMOS14 dataset, the BasicTAD head is used, and for the ActivityNet dataset, the AFSD head is used.

[0018] 3) Training phase: Input video frames of the input samples into the temporal action detection model ViT-TAD for training. For the two long-term temporal action detectors in 2), the classification loss is calculated using the Focal loss function. BasicTAD calculates the loss for predicting nominees and ground truth using the DIoU loss function, while AFSD calculates the loss for predicting nominees and ground truth using cross-entropy and L1 distance loss functions. ViT-TAD with the BasicTAD head uses the SGD optimizer, while ViT-TAD with the AFSD head uses the AdamW optimizer. The network parameters are updated using the backpropagation algorithm, and the detection model is trained until the required number of iterations is reached.

[0019] 4) Testing phase: The video clip to be tested is input into the trained temporal action detection model ViT-TAD to obtain N prediction nominations. Finally, the reliable prediction nominations are selected according to the bounding box confidence score threshold to evaluate the performance of the trained ViT-TAD.

[0020] Furthermore, after dividing the long video segment into multiple short segments, the video frames of the short segments are scaled proportionally, and then the video frames are cropped to reduce the spatial resolution. Then, the video frames of the short segments are positionally encoded, including clip-level temporal positional encoding and segment-level temporal positional encoding. Clip-level temporal positional encoding is generated based on the length of the clip composed of multiple short segments, while segment-level temporal positional encoding is generated based on the length of the current segment. The temporal characteristics of each short segment, combined with the two types of positional encoding, make the positional encodings at different temporal positions different from each other.

[0021] Based on the above method, the present invention also provides an electronic device, which includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to realize the above-mentioned end-to-end timing action detection method.

[0022] Based on the above method, the present invention also provides a computer-readable storage medium storing at least one instruction or at least one program, wherein when the at least one instruction or the program is executed, the timing action detection model in the above-described end-to-end timing action detection method is implemented.

[0023] This invention proposes ViT-TAD (Vision Transformer-Temporal Action Detection), a simple temporal action detection framework. It aims to apply the single-scale ViT model, which can only model short-term events, to temporal action detection tasks that rely on long-term information modeling. The core idea is to insert an information propagation module into the ViT backbone network to guide the model in learning global temporal semantic information. The internal parameters of the information propagation module are updated during training using an end-to-end training method. The information propagation module allows multiple short-term modeled features to flexibly interact with each other, enabling the backbone network to model global temporal information. Furthermore, to reduce computational costs, this invention reduces the spatial resolution of the input image frames and uses interpolation techniques to align the positional encodings from the pre-trained ViT backbone network.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] The end-to-end temporal action detector ViT-TAD proposed in this invention can fully utilize the global modeling capabilities of the ViT backbone network. It is trained based on pure video RGB frames, requiring no additional optical flow. It can be adapted to different temporal action detector heads. It also has low computational cost during training.

[0026] The information propagation module designed in this invention can unleash the global modeling capabilities of the ViT backbone network. This module can simultaneously model the temporal relationships of all input short video segments, thus breaking the current limitation that the ViT backbone network can only model short video segments individually. Existing temporal action detection work uses short video segments for detection, while this invention… Figure 3 and Figure 4 As can be seen, on the THUMOS14 dataset or the ActivityNet-1.3 dataset, the ViT-TAD of this invention outperforms other existing temporal action detection methods for long-duration videos.

[0027] This invention features strong adaptability, a direct detection process, and accurate detection results in temporal action detection tasks. Strong adaptability is reflected in ViT-TAD's ability to adapt to different action detector heads, and the direct detection process refers to the end-to-end training implemented in this invention. Compared with existing methods, this invention demonstrates better performance on mainstream datasets and in practical applications. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the overall process used in this invention.

[0029] Figure 2 This is a schematic diagram of the global temporal position encoding of the present invention.

[0030] Figure 3 This is a schematic diagram comparing the present invention with other methods on the THUMOS14 dataset.

[0031] Figure 4 This is a schematic diagram comparing the present invention with other methods on the ActivityNet-1.3 dataset. Detailed Implementation

[0032] This invention is an end-to-end temporal action detection method that adapts a short-time Transformer to a long-time action detection task. First, the input long-time video segment is divided into multiple short-time segments. Then, these short-time segments are fed into a ViT backbone network for feature encoding. During encoding, an information propagation module inserted into the backbone network receives these short-time segments and enables them to perform global temporal interaction within the module. After the interaction is complete, these short-time segments are independently modeled in the backbone network. Finally, the features that have completed the interaction in the backbone network are fed into the head of the temporal action detector for action prediction, obtaining the action boundary, action category, and confidence score for that category. This invention models global temporal information by inserting an inner-backbone information propagation module into the ViT backbone network and a post-backbone information propagation module after the backbone network to further improve the modeled short-time segment features. The method includes a video frame generation stage, a network configuration stage, a training stage, and a testing stage, such as... Figure 1 As shown, the specific implementation of the present invention is described below.

[0033] 1) Video Frame Generation Stage: The mainstream temporal action datasets include THUMOS14 and ActivityNet; other datasets are less commonly used. For THUMOS14, ffmpeg is used to extract frames from the original video at a frame rate of 8 FPS, resulting in a T-frame video frame sequence that is then fed into the subsequently configured network. The frame extraction rate can be adjusted depending on the dataset used. For ActivityNet-1.3, ffmpeg is used to sample the original video into T frames. These T-frame video frames are used as input samples for long-duration video segments. Further, these video frames are proportionally scaled to a shorter side length of 180, and then cropped to a spatial resolution of 160*160.

[0034] These video frames are used to divide the video into multiple short segments and feed them into the network for temporal action detection. As an improvement, this invention also improves the temporal position encoding for video frames, such as... Figure 2As shown. This invention divides long video segments into multiple short segments. However, for the ViT-TAD temporal action detection model as a whole, it still processes long-term features. Based on the ViT model, the video segments need to be positionally encoded before temporal action detection. The positional encoding of this invention includes clip-level temporal positional encoding and segment-level temporal positional encoding. Clip-level temporal positional encoding is generated based on the length of the clip composed of multiple short segments, that is, the positional encoding of the long video segment. Segment-level temporal positional encoding is generated based on the length of the current segment. The temporal features of each short segment are added with two types of positional encoding, so that the positional encodings at different temporal positions are different from each other. This invention adds positional encodings to the features at different temporal positions of the spliced ​​short feature segments. The positional encodings at different temporal positions are different from each other. Here, "splicing" means that the value of the positional encoding is directly added to the temporal feature. In the prior art, the backbone network only processes the temporal positional encoding of short segments. When applied to long-term detection tasks, the positional encodings at the same position of multiple short segments are the same, which will cause the network to be unable to recognize the differences between these positions. This invention adds clip-level temporal position encoding. On the basis of the clip-level temporal position encoding of these short-time segments, an additional long-time position encoding is given. Because of the added clip-level position encoding, the position encodings at different temporal positions of the spliced ​​short-time feature segments are different from each other, which enables the network to accurately distinguish the differences between different temporal positions of the entire long-time feature.

[0035] 2) Network Configuration Stage: A ViT-based video backbone network is used to extract features from the video frame sequence. The backbone network for video action classification consists of n different hierarchical modules (stages). Each stage provides the features obtained from modeling in the current stage. In this invention, when constructing the temporal action detection model ViT-TAD based on the ViT model, an inner-backbone information propagation module is inserted into the ViT backbone network, and a post-backbone information propagation module is inserted after the ViT backbone network. The insertion position and number of inner-backbone information propagation modules are determined based on the results of ablation experiments. This invention preferably inserts four inner-backbone information propagation modules evenly into the ViT backbone network, resulting in the best final effect. Finally, action detection is performed using a long-term temporal action detector. Two heads from BasicTAD and AFSD are used here. The BasicTAD structure is used for the THUMOS14 dataset, and the AFSD structure is used for the ActivityNet-1.3 dataset. The entire framework and process are as follows: Figure 1 As shown, the specific operation is as follows:

[0036] 2.1) Inner-backbone information propagation module: This component receives short-term feature sequences modeled by short-term modules in the backbone network and concatenates these short-term feature sequences along the temporal dimension to form long-term features X∈R. T×W×H×C The feature dimension is C, the feature width is W, the height is H, and the time series length is T, where T represents a long-term time series. In the information propagation module, spatial information does not participate in the interaction; therefore, long-term features can be transformed into X∈R. (W×H)×T×C That is, X = concat(x1, x2, ... x W×H ). Where x t Let t represent the long-term temporal features at each spatial location, where t = 1, 2, ..., W × H. The inner-backbone information propagation module uses a self-attention mechanism, specifically the formula Y = X + attention(X). Skip connections are used to prevent gradient vanishing or exploding, improving the model's convergence speed and generalization performance. For the calculation of attention, each temporal feature x is first constructed... t The query q, key k, and value v are used, followed by self-attention calculation, as shown in the following formula:

[0037] q = x t ·w q k=x t ·w k v = x t ·w v

[0038]

[0039] here Where C q =C k =C / m, where m is the number of attention heads. After calculations within the inner-backbone information propagation module, we finally obtain Y = concat(y1, y2, ... y W×H )∈R (W×H)×T×C Features resulting from global temporal interactions.

[0040] 2.2) Post-backbone information propagation module: This component receives features from the backbone network modeling and concatenates them along the temporal dimension into long-term features X′∈R. T×C The feature is then pooled along its spatial dimension to obtain pure temporal features, which are then further modeled globally. At this point, the input feature of this module is X′∈R. T×CThe operation is performed using the Transformer, which consists of a self-attention layer and a feedforward network, Y′=transformer(X′), and the specific formula is as follows:

[0041] X″ = X′ + attention(X′)

[0042] Y′=X″+FFN(X″)

[0043] The calculations within the attention module are consistent with those in the inner-backbone information propagation module. The FFN consists of two fully connected layers, which are used to perform nonlinear transformations on each temporal position for feature interaction. The resulting Y′ is input to the long-term temporal action detector for action detection.

[0044] 2.3) Header of the temporal action detector: This component receives the temporal features Y′∈R modeled from the post-backbone information propagation module. T×C It is then decoded into the boundaries, categories, and confidence scores of the predicted actions.

[0045] When the head in BasicTAD is selected, the temporal feature D first goes through several temporal pooling layers to construct a five-layer multi-scale feature. Next, the temporal features D at each scale i The code will then proceed to the head for decoding. The head consists of two branches: a classification branch and a regression branch. Both branches are composed of four temporal convolutional layers, including normalization and activation layers, which generate... and in This represents the confidence score for each predicted action category. This represents the length of each prediction from its current temporal position to its action start and end boundaries, i.e., the temporal coordinate offset. t i This represents the output of the feature corresponding to layer i. or The temporal length is given by K, where K represents the number of actions. The final action prediction is generated. Where start and end represent the start and end boundaries of the action, c represents the action category, and s represents the action confidence score.

[0046] When the head in AFSD is selected, the temporal feature D is first processed through a temporal feature pyramid to generate six layers of multi-scale features. Next, the temporal features D at each scale i The code will then enter the head for decoding. The head consists of a two-stage detector, with the first stage performing coarse-grained prediction and the second stage performing fine-grained prediction. The coarse-grained prediction consists of two temporal convolutional layers, which generate... and in This represents the confidence score for each predicted action category. This represents the length of each prediction from its current temporal position to its action start and end boundaries, i.e., the temporal coordinate offset. K represents the number of actions. In the model, 2 represents the foreground and background classes. The final classification result of ActivityNet-1.3 relies on an additional pre-trained classifier. Two-stage fine-grained prediction is performed by a saliency enhancement module, which is an existing technique for the AFSD head. This module receives input including a temporal feature pyramid and first-stage features, and generates two-stage classification and regression results respectively. and in This represents the confidence score for each predicted action category. This represents the length of each prediction from its current temporal position to its action start and end boundaries, i.e., the temporal coordinate offset. The two-stage temporal coordinate offset will further optimize the boundary prediction.

[0047] As mentioned earlier, the mainstream temporal action datasets include the THUMOS14 dataset and the ActivityNet dataset. If other datasets are used, the corresponding long-term temporal action detector can be configured according to the design concept of this invention to complete the long-term temporal action detection task.

[0048] 3) Training Phase: The input video clips undergo four types of data augmentation: spatial random cropping, distortion, random rotation, and random flipping. Then, the network configuration phase generates action predictions. Next, the loss function is used to calculate the loss between the predictions and the ground truth, thereby guiding the model to optimize its parameters.

[0049] The loss function trained on THUMOS14 in this invention consists of the classification loss and regression loss of BasicTAD, and the loss function expression is L = L cls +αL reg α is used to balance the two loss functions and takes a value of 1. Classification loss L cls Using the focal loss function, the regression loss Li reg The DIoU loss function is used. The detailed calculation process of the classification loss is shown below:

[0050] L cls =-(1-p t ) γ log(p t )

[0051] Where p tThis represents the probability of each action category after softmax, and γ represents the modulation coefficient used to focus on hard samples. The detailed calculation process of the regression loss is shown below:

[0052]

[0053] Where B represents the boundary of the predicted action, B gt ρ represents the boundary of the action under the truth value. 2 (b,b gt c represents the square of the Euclidean distance between the center points of the predicted action and the true action. 2 This represents the square of the shortest distance between the predicted action and the ground truth action. The method uses SGD as the optimizer, updates network parameters via backpropagation, sets momentum to 0.9, and sets the decay weight to 0.0001. Training is performed in 12 rounds, with 100 video clips randomly selected each round.

[0054] The loss in this invention, trained on ActivityNet-1.3, consists of two classes of coarse-grained prediction loss and two classes of fine-grained prediction loss from AFSD, with a total loss of L = L cls +L reg +L′ cls +L′ reg The classification loss L for coarse-grained prediction is... cls Classification loss L′ for fine-grained prediction cls Both use the focal loss function, with the regression loss L for coarse-grained prediction. reg Using the tIoU loss function, the detailed calculation process is as follows:

[0055]

[0056] Where B represents the boundary of the predicted action, B gt This represents the boundary of the action at the true value. The regression loss L′ for fine-grained prediction. reg The L1 loss function is used, and its calculation process is as follows:

[0057]

[0058] Where B represents the boundary of the predicted action, B gt This represents the boundary of the action under the true value, and n represents the number of predictions. This method uses AdamW as the optimizer and updates the network parameters through backpropagation. Each epoch uses a complete video for training, and training is completed in 10 epochs.

[0059] 4) Testing Phase: The input video clip is cropped from the center and then fed into the trained temporal action detection model ViT-TAD to obtain N prediction nominations. Finally, reliable prediction nominations are selected based on the bounding box confidence score threshold, such as... Figure 1 As shown, "Penalty: 94%" represents the confidence score, indicating that the network predicts a very high probability, at 94%, that the action is a penalty kick.

[0060] This invention uses Non-Maximum Suppressor (NMW) as a filter to select reliable action predictions, which are then submitted to the evaluation module to measure mAP, in order to evaluate the performance of ViT-TAD. Figure 3 As shown, on the THUMOS14 dataset, when the backbone network uses ViT-S, the results are the best among all end-to-end temporal action detections. When the backbone network uses ViT-B, its mean average precision (mAP) is 68.5 across five thresholds from 0.3 to 0.7. Figure 4 As shown, on the ActivityNet-1.3 dataset, when the backbone network uses ViT-B, the results are the best among all temporal action detections, with an average mAP of 37.12 across ten thresholds from 0.5 to 0.95.

[0061] The timing action detection model ViT-TAD of the present invention is implemented by a program. Therefore, the present invention also provides an electronic device comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program segment, which is loaded and executed by the processor to implement the timing action detection model ViT-TAD for implementing an end-to-end timing action detection method. The present invention also provides a computer-readable storage medium storing at least one instruction or at least one program segment, which, when executed, implements the timing action detection model in the aforementioned end-to-end timing action detection method.

Claims

1. An end-to-end timing action detection method, characterized by A time sequence action detection model ViT-TAD is constructed based on the ViT model to complete the time sequence action detection task which relies on long-time modeling. First, the long-time video clip is divided into multiple short-time clips, each of which corresponds to a ViT backbone network. Each ViT backbone network contains n hierarchical modules. An inner-backbone information propagation module is inserted into the ViT backbone network. The insertion position and number of the inner-backbone information propagation module are determined according to the results of the ablation experiment. A post-backbone information propagation module is inserted after the ViT backbone network. Then a long-time time sequence action detector is used for time sequence action detection. The inserted information propagation modules are used to guide the detection model to learn global time sequence semantic information, and the internal parameters of the information propagation modules are updated through end-to-end training during training. The two kinds of information propagation modules are specifically: An inner-backbone information propagation module: for short-term feature sequences modeled by short-term modules in each backbone network, the short-term feature sequences are spliced into long-term features X ∈ R (W×H)×T×C , X = concat(x1, x2,...x W×H ), where the feature dimension is C, the feature width is W, the feature height is H, and the time sequence length is T, x t represents the long-term time sequence feature at each spatial position, t = 1, 2,..., W × H, the inner-backbone information propagation module uses a self-attention mechanism, and the formula is Y = X + attention(X). For the calculation of attention, first, the query q, key k and value v of each time sequence feature x t are constructed, and then the self-attention calculation is performed, and the specific formula is as follows: q=x t • w q k=x t • w k v=x t • w v These are the weight parameters q, k, and v of the self-attention mechanism, respectively, and C. q =C k =C / m, where m is the number of attention heads. After calculation within the inner-backbone information propagation module, we obtain Y = concat(y1, y2, ... y W×H )∈R (W×H)×T×C Feature Y is the feature that has undergone global temporal interaction; post-backbone information propagation module: for a plurality of short-time segment features modeled by the ViT backbone network, the plurality of short-time segment features are spliced along a time sequence dimension into long-time features X' ∈ R T×C , and operation in the Transformer, composed of a self-attention layer and a feedforward network, Y' = transformer(X'), and specific formulas are as follows: X''=X'+attention(X') Y'=X''+FFN(X'') Where the calculation of attention is consistent with the inner-backbone information propagation module, and FFN consists of two fully connected layers for nonlinear transformation of each time sequence position to interact features to obtain Y' for action detection by the long-time time sequence action detector.

2. The end-to-end timing action detection method of claim 1, wherein Four inner-backbone information propagation modules are uniformly inserted into the ViT backbone network.

3. The end-to-end timing action detection method of claim 1, wherein the timing is The implementation of the action detection model ViT-TAD is as follows: 1) Video frame generation stage: determine the time sequence action detection dataset to be used, extract video frames from the THUMOS14 dataset video at a set frame rate, and extract a T-frame video frame sequence as a long-time video clip input sample. For the video of the ActivityNet dataset, extract a video clip with a length of T seconds as a long-time video clip input sample. Divide the long-time video clip input sample into multiple short-time clips. 2) Network configuration stage: configure the time sequence action detection model ViT-TAD. According to the dataset, configure the long-time time sequence action detector. For the THUMOS14 dataset, use the head of BasicTAD, and for the ActivityNet dataset, use the head of AFSD. 3) Training stage: input the video frames of the input sample into the time sequence action detection model ViT-TAD for training. For the two long-time time sequence action detectors in 2), the classification loss is calculated by the Focal loss function. BasicTAD calculates the loss between the predicted nomination and the true value by the DIoU loss function. AFSD calculates the loss between the predicted nomination and the true value by the cross-entropy and L1 distance loss functions. The ViT-TAD using the head of BasicTAD uses the SGD optimizer, and the ViT-TAD using the head of AFSD uses the AdamW optimizer. The network parameters are updated by the back propagation algorithm, and the detection model is trained until the iteration number is reached. 4) Test phase: input the video clip to be tested into the trained time sequence action detection model ViT-TAD to obtain N prediction nominations, and finally screen the reliable prediction nominations according to the frame confidence score threshold, which are used to evaluate the performance of the trained ViT-TAD.

4. A method for end-to-end timing action detection according to any of claims 1-3, characterized in that After dividing the long-time video clip into multiple short-time clips, the video frames of the short-time clips are scaled proportionally, and then the video frames are cropped to reduce the spatial resolution. Then, the video frames of the short-time clips are positionally encoded, including clip-level temporal position encoding and clip-level temporal position encoding. The clip-level temporal position encoding is generated for the length of the clip composed of multiple short-time clips, and the clip-level temporal position encoding is generated for the length of the current clip. The temporal features of each short-time clip are added to the two kinds of position encodings, so that the position encodings at different temporal positions are different from each other.

5. A method for end-to-end timing action detection according to any of claims 1-3, characterized by long time The temporal action detector receives the temporal features D e R output by the post-backbone information propagation module T×C and decodes into the boundaries, class, and confidence score of the predicted action, When the head in BasicTAD is selected as the action detector, the temporal feature D is first passed through a temporal pooling layer to construct 5 layers of multi-scale features Then the temporal feature D of each layer i The input into the head is decoded, the BasicTAD head consists of two branches, namely the classification branch and the regression branch, both of which are composed of 4 layers of temporal convolution layers including normalization layers and activation layers, respectively generating and represents the confidence score of each predicted action category, represents the length of each predicted action from the current temporal position to its action start and end boundary, i.e. the temporal coordinate offset t i represents the or temporal length of the corresponding i-layer feature output, K represents the number of actions, and finally generates the action prediction where start and end represent the start and end boundaries of the action, c represents the action category, and s represents the action confidence score; When the head in AFSD is selected as the action detector, the temporal feature D is first passed through a temporal feature pyramid to generate 6 layers of multi-scale features Then the temporal feature D at each scale is passed through a 2-layer temporal convolutional layer to generate the coarse-grained prediction i The input to the AFSD head is decoded, which consists of two-stage detectors, where the first stage makes coarse-grained prediction and the second stage makes fine-grained prediction. The coarse-grained prediction consists of two layers of temporal convolutional layers to generate and represents the confidence score of each predicted action class, represents the length of each prediction from the current temporal position to its action start and end boundaries, i.e., the temporal coordinate bias The 2 in the above represents the foreground and background classes; the fine-grained prediction of the second stage is made by a saliency improvement module, which takes the input including the temporal feature pyramid and the first stage features to generate the classification and regression results of the second stage, respectively and represents the confidence score of each predicted action class, represents the length of each prediction from the current temporal position to its action start and end boundaries, i.e., the temporal coordinate bias 6. A method for end-to-end timing action detection according to any of claims 1-3, characterized by When training the time sequence action detection model ViT-TAD, the input video clip is subjected to four kinds of data enhancement of spatial random cropping, distortion, random rotation and random flipping, and then input into the time sequence action detection model ViT-TAD to generate action prediction. The loss between the prediction and the true value is calculated by the loss function, so as to guide the model to optimize the parameters.

7. An electronic device, characterized by The electronic device includes a processor and a memory, and the memory stores at least one instruction or at least one program, which is loaded and executed by the processor to implement the end-to-end time sequence action detection method of any one of claims 1-6.

8. A computer readable storage medium characterized by The computer readable storage medium stores at least one instruction or at least one program, and when the at least one instruction or program is executed, the time sequence action detection model in the end-to-end time sequence action detection method of any one of claims 1-6 is implemented.

Images