Clipping prompting method for video action detection

Abstract

The invention relates to a prompt cutting method for video action detection, which comprises the following steps of: 1) acquiring video data, and setting a non-key frame mark, a prompt mark and a key frame mark; 2) inputting the marked video data into a video action detection model to obtain a clipped key frame mark; and (3) the proposal mark and the cut prompt mark pass through a decoder of the video action detection model to output a final feature sequence, the final feature sequence comprises proposal features marked with the proposal mark and prompt features marked with the cut prompt mark, and a video action recognition result is obtained based on the final feature sequence. Compared with the prior art, the method has the advantages that the calculation complexity is remarkably reduced, the key frames are reserved, and comprehensive optimization of video action detection is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of video action detection, and in particular to a prompt cutting method for video action detection. Background Art

[0002] Video action detection aims to locate and classify actions in video sequences. It is an important and challenging task in the field of computer vision. In recent years, the Vision Transformer (ViT) model trained in an unsupervised manner has shown remarkable success in video understanding tasks. However, these ViT models still face difficulties in accurately locating actions, especially when the attention calculation is disturbed by irrelevant background and the high redundancy of video data.

[0003] At present, mainstream video action detection models generally adopt the strategy of "localization first, then classification". These models usually use a single key frame (usually an intermediate frame) to locate the object, and then use the entire video sequence to capture the motion and context information required for action recognition. However, video sequences often contain a lot of redundant information (such as Figure 2 (left), which results in inefficient attention calculation and increased computational overhead. Some previous studies have attempted to reduce redundancy by cropping non-key frames, but in order to maintain detection accuracy, these methods usually avoid cropping key frames. As a result, these methods can only achieve local optimization due to the lack of an effective guidance strategy for cross-label cropping. Summary of the invention

[0004] The purpose of the present invention is to provide a cold chain vehicle path optimization method that takes into account the on-demand recovery of turnover transport units in order to significantly reduce the computational complexity while retaining key frames to achieve comprehensive optimization of video motion detection.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A prompt cropping method for video action detection, the method comprising the following steps:

[0007] 1) Obtain video data, and the ViT model performs tokenization processing on the video data, sets key frame tags for key frames of the video data, sets non-key frame tags for non-key frames of the video data, and sets prompt tags at the same time. The sequence of prompt tags, the sequence of key frame tags, and the sequence of non-key frame tags together constitute input tags, and the video data with input tags is used as the tokenized video data;

[0008] 2) The video data after tokenization is input into the video action detection model. The transform self-attention mechanism of the video action detection model linearly transforms the input token into a query, key and value matrix, calculates the attention map between the prompt token and the key frame token, calculates the importance score of the prompt token based on the attention map between the prompt token and the key frame token, performs prompt token cropping based on the importance score of the prompt token to obtain cropped prompt token, calculates the importance score of the key frame token to crop the key frame token to obtain cropped key frame token, and crops the non-key frame token based on the importance score of the non-key frame token to obtain cropped non-key frame token;

[0009] 3) The features marked with the cropped key frame markers are respectively subjected to multi-scale adjustment and fusion, and a bounding box is used to perform a region alignment operation on the cropped key frame markers and the cropped non-key frame markers to generate a set of proposal markers. The proposal markers and the cropped hint markers are passed through a decoder of a video action detection model to output a final feature sequence. The final feature sequence includes proposal features marked with the proposal markers and hint features marked with the cropped hint markers, and a video action recognition result is obtained based on the final feature sequence.

[0010] Furthermore, the input notation is z=[P, T key , T non ], where P represents a sequence of prompt markers, T key represents a sequence of key frame markers, T non Represents a sequence of non-keyframe markers. For video data of size T×H×W×C, where T, H, W, and C represent the number of time frames, height, width, and number of channels, respectively, the video data is divided into small cubes of size t×h×w, t, h, and w are the number of time frames, height, and width of the small cube after division, respectively. The number of markers in the time dimension is The number of markers in the height dimension is The number of markers in the width dimension is Marked with The central frame of the marker in the time dimension is defined as the key frame, and all other frames are considered non-key frames. The sequence T composed of key frame markers key The dimension is 1×n h ×n w ×d, a sequence of non-keyframe markers T non The dimension is (n t -1)×n h ×n w ×d, indicating that the sequence P of markers includes N P prompt tags, d represents the dimension of the tag, and the total number of input tags is N all =N P +Nkey +N non , where the number of key frame markers is N key =1×n h ×n w , number of non-keyframe markers N non =(n t -1)×n h ×n w .

[0011] Furthermore, the importance score of the hint tag is:

[0012]

[0013] Among them, I P (i) represents the importance score of the i-th hint tag, A key represents the attention map between the prompt marker and the key frame marker. The attention map between the prompt marker and the key frame marker is obtained by the self-attention mechanism. Calculate, where Q and K represent the query matrix and the key matrix.

[0014] Furthermore, the prompt mark is cropped based on the importance score of the prompt mark, and the cropped prompt mark is specifically:

[0015] Keep the first k prompts with the largest importance scores, k = N′ P =N P ·ρ P , ρ P is the retention ratio of the prompt mark, and the remaining prompt marks are cropped to obtain the cropped prompt mark P′.

[0016] Furthermore, the importance score of the key frame marker is:

[0017]

[0018] Among them, A′ key (i, j) represents the attention map after setting a binary mask on the reserved cue marker positions.

[0019] Furthermore, the importance scores of the key frame markers are calculated to perform key frame marker cropping, and the specific steps for obtaining the cropped key frame markers are as follows:

[0020] Calculate the importance scores of key frame markers and keep the first N′ key The key frame label with the largest importance score, N′ key =N key ·ρ key , ρ keyis the retention ratio of the key frame mark, and the cropped key frame mark T′ is obtained key .

[0021] Furthermore, the importance score of non-keyframe markers is:

[0022]

[0023] Among them, α is the weight coefficient for the hint mark and key frame mark, N′ all =N′ P +N′ key +N non , A non Represents N′ all N non The attention weight map calculated from the non-keyframe labels.

[0024] Furthermore, the specific steps of cropping the non-key frame tags based on the importance scores of the non-key frame tags to obtain cropped non-key frame tags are:

[0025] Keep the first N′ non The non-keyframe tags with the largest importance scores, N′ non =N mon ·ρ non , where ρ non is the retention ratio of non-keyframe labels, and the cropped non-keyframe labels T′ are obtained non .

[0026] Furthermore, the overall loss function of the video action detection model combines the set prediction loss and the action classification loss.

[0027] Furthermore, the overall loss function is:

[0028]

[0029] Among them, λ cls , L1 , giou and λ act They are respectively the classification loss Smooth L1 box regression loss Generalized IOU regression loss and action classification loss The weight of is the overall loss function.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] Through the prompt cropping mechanism, redundant information in the video sequence is effectively eliminated, which significantly reduces the computational complexity. At the same time, key features that are crucial to the action detection task are retained, thereby improving the computational efficiency of the video action detection model. The present invention also guides attention calculations through learnable prefix prompts, thereby strengthening the model's attention to key action clues, thereby accurately locating and identifying actions under complex backgrounds or video interference, thereby comprehensively improving the detection accuracy of action detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the overall structure of the method of the present invention, wherein Figure 1 (a) is a schematic diagram of video input and prompts. Figure 1 (b) is a schematic diagram of marker cropping optimization. Figure 1 (c) is a schematic diagram of the positioning module. Figure 1 (d) is a schematic diagram of the classification module;

[0033] Figure 2 It is a schematic diagram for comparing the method of the present invention with the existing method. DETAILED DESCRIPTION

[0034] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0035] The present invention proposes a prompt cropping method for video action detection, which specifically includes the following steps: first, an adjustable prompt vector is used to optimize the input of the model so that it pays more attention to the key information related to the task; then, the attention map generated by the self-attention mechanism is used to crop the label information in the video data in stages. The cropping process is divided into three steps: the first step is to crop the prompt vector and retain the part that can most effectively guide the model; the second step is to filter the key frame labels and only retain the labels containing important spatial information; the third step is to crop the non-key frame labels and only retain the parts that are critical to action understanding. Through these steps, the cropping process can greatly reduce the amount of calculation, while retaining the key information required for action detection to ensure the accuracy of action detection. Compared with the prior art, the present invention has the advantages of high accuracy, strong robustness and high computational efficiency in action detection.

[0036] The overall architecture diagram of the method is as follows: Figure 1 As shown, the method includes:

[0037] 1) Obtain video data, and the ViT model performs tokenization processing on the video data, sets key frame tags for key frames of the video data, sets non-key frame tags for non-key frames of the video data, and sets prompt tags at the same time. The sequence of prompt tags, the sequence of key frame tags, and the sequence of non-key frame tags together constitute input tags, and the video data with input tags is used as the tokenized video data;

[0038] 2) The video data after tokenization is input into the video action detection model. The transform self-attention mechanism of the video action detection model linearly transforms the input token into a query, key and value matrix, calculates the attention map between the prompt token and the key frame token, calculates the importance score of the prompt token based on the attention map between the prompt token and the key frame token, performs prompt token cropping based on the importance score of the prompt token to obtain cropped prompt token, calculates the importance score of the key frame token to crop the key frame token to obtain cropped key frame token, and crops the non-key frame token based on the importance score of the non-key frame token to obtain cropped non-key frame token;

[0039] 3) The features marked with the cropped key frame markers are respectively subjected to multi-scale adjustment and fusion, and a bounding box is used to perform a region alignment operation on the cropped key frame markers and the cropped non-key frame markers to generate a set of proposal markers. The proposal markers and the cropped hint markers are passed through a decoder of a video action detection model to output a final feature sequence. The final feature sequence includes proposal features marked with the proposal markers and hint features marked with the cropped hint markers, and a video action recognition result is obtained based on the final feature sequence.

[0040] 1) Video input and prompts: First, the input video is processed through a sequence containing learnable prefix prompts. This prompt can help the model better understand the video content, especially when it comes to complex action recognition, the prefix prompt can provide the necessary contextual information.

[0041] The video input processing step is to represent the input video clip as a three-dimensional tensor containing time and space information. The video is divided into multiple small blocks and converted into a set of video tags. Specifically, the video is tokenized using block embedding technology to generate a video tag sequence. Then, the middle frame in the sequence is defined as a key frame, and the remaining frames are defined as non-key frames.

[0042] Vision Transformer (ViT) tokenizes videos by embedding cubes. Specifically, the size of a video segment V is T×H×W×C, where T, H, W, and C represent the number of time frames, height, width, and channels, respectively. The video segment is divided into small cubes of size t×h×w to generate a token sequence. After division, the number of tokens in each dimension is calculated as follows: The number of tokens in the time dimension is The height dimension is The width dimension is The feature dimension of each tag is d, and the tag sequence is reorganized into a one-dimensional sequence after position embedding. The central frame in the video clip is defined as the key frame, denoted as time marker, all other frames are considered non-keyframes.

[0043] The central frame in the video clip is defined as the key frame, denoted as time markers, and all other frames are considered non-key frames. The marker sequence of key frames is defined as T key , whose dimension is 1×n h ×n w ×d. The label sequence of non-keyframes is defined as T non , the dimension is (n t -1)×n h ×n w ×d. The total number of markers includes the number of key frame markers N key =1×n h ×n w and the number of non-keyframe markers N non =(n t -1)×n h ×n w , the total number is N = n t ×n h ×n w .

[0044] To enhance the representation ability of the model, a learnable prefix hint is introduced before inputting into the first layer of Transformer. Each hint token is a d-dimensional vector. Assume that the hint token set contains N P tokens, denoted as P. The final input sequence is z = [P, T key , T non ], the total dimension is Where N all =N P +N key +N non .

[0045] The step of adding learnable prefix hints is to add a set of learnable hint tags before the first layer of the Transformer model. These hint tags are composed of a set of trainable vectors, represented as a set. These hint tags are concatenated with the video tags to form an input sequence, which is used as the input content of the subsequent Transformer encoder.

[0046] 2) Marker pruning optimization: In the backbone network part of the video, the prefix prompt information is combined for labeling. Through the pruning mechanism, the labels in the video are selectively retained and redundant labels are removed to reduce the amount of calculation and improve processing efficiency. The cropped labels are optimized through different processing methods to retain key frame and important non-key frame labels to ensure that the model focuses on the most important action information.

[0047] The step of adding learnable prefix hints is to add a set of learnable hint tags before the first layer of the Transformer model. These hint tags are composed of a set of trainable vectors, represented as a set. These hint tags are concatenated with the video tags to form an input sequence, which is used as the input content of the subsequent Transformer encoder.

[0048] The cue marker pruning step is to calculate the correlation score between each cue marker and the key frame marker according to the attention map, filter out the most important cue markers, retain the cue markers with the highest scores, and prune the cue markers with low correlation, so as to obtain an optimized cue marker set.

[0049] The key frame marker cropping step is to further filter the key frame markers using the result of the prompt marker cropping. According to the correlation between the prompt marker and the key frame marker in the attention graph, the importance score of each key frame marker is calculated, and the key frame markers with the highest scores are retained to obtain the cropped key frame marker set.

[0050] The non-keyframe marker clipping step is to clip the non-keyframe markers based on the retained prompt markers and keyframe markers. By comprehensively analyzing the correlation between the non-keyframe markers, prompt markers and keyframe markers, the importance scores of the non-keyframe markers are calculated, and the non-keyframe markers with higher scores are screened out to form a clipped non-keyframe marker set.

[0051] The step of generating the final tag sequence is to merge the cropped prompt tags, key frame tags and non-key frame tags to form the final tag sequence. This sequence is used as input for subsequent models to perform video action detection tasks, improving detection performance while reducing computational complexity.

[0052] The above cropping process can significantly reduce the amount of calculation while retaining sufficient information to ensure the accuracy of action detection.

[0053] To evaluate the importance of a token, the attention map is calculated using the self-attention mechanism. In self-attention, the input token is linearly transformed into query, key, and value matrices Q, K, and V. The attention operation is calculated as:

[0054]

[0055] The attention map A is defined as For a multi-head attention layer, the attention maps of all heads are averaged.

[0056] (i) Prompt Marking Clipping

[0057] Hint marker cropping aims to retain the hint markers that contribute more to the key frame. First, the attention map A between the hint marker and the key frame marker is calculated. key , and then the importance score of each hint tag is calculated based on the attention map. Specifically, the importance score of the i-th hint tag is defined as:

[0058]

[0059] Keep the first k important reminder tags, where k = N' P =N P ·ρ P , ρ P is the proportion of the reminder mark retained. The remaining reminder marks are pruned. The pruned reminder marks are recorded as P′.

[0060] (ii) Keyframe Marking Cropping

[0061] Based on the retained cue markers, further prune the keyframe markers. Define a binary mask to mark the entries that are retained in the cue marker pruning stage. Use the mask to calculate the pruned attention map A′ key . The importance scores of the keyframe markers are then calculated:

[0062]

[0063] Keep the first N′ key key frame markers, where N′ key =N key ·ρ key , ρ key is the retention ratio of the key frame markers. The cropped key frame markers are recorded as T′ key .

[0064] (iii) Non-keyframe labeling cropping

[0065] For non-keyframe markers, the relevance to them is calculated based on the cropped hint markers and keyframe markers. Specifically, the importance score of the i-th non-keyframe marker is defined as:

[0066]

[0067] Where α is the weighting coefficient for the hint mark and key frame mark, the purpose is to give priority to retaining the non-key frame marks that are strongly related to these two types of marks. Finally, retain the first N′ non non-keyframe markers, N′ non =N non ·ρ non , where ρ non is the retention ratio of non-keyframe markers. The cropped non-keyframe markers are recorded as T′ non .

[0068] 3) Perform positioning and classification: The positioning module generates the spatial position and time period of the action, and the positioning module generates the corresponding action position prediction based on the pruned markers. The classification module is responsible for generating action category predictions based on the pruned markers and identifying different action types in the video.

[0069] The localization module is implemented based on SparseR-CNN. In the 4th, 7th, 10th and 12th layers of the backbone network, the features of the keyframe markers are adjusted at multiple scales: the features are enlarged by 4 times in the 4th layer, 2 times in the 7th layer, unchanged in the 10th layer, and reduced to 1 / 2 times in the 12th layer. These adjusted feature maps are fused at multiple scales through the feature pyramid network. Then, a certain number of learnable proposal boxes and their features are initialized to generate candidate regions, and finally a bounding box and the corresponding target confidence score are generated.

[0070] The classification module uses the bounding boxes generated by the above positioning module to perform region alignment operations on keyframe tags and non-keyframe tags to generate a set of proposed tags T prop . These proposal tags are then concatenated with the sequence features and input into the Transformer decoder. The decoder consists of a multi-head self-attention layer and a feedforward network, and outputs decoded features. The final feature sequence is divided into two parts: prompt features and proposal features. After these features are processed by the feedforward network (FFN) and averaged, the classification layer predicts the action category and outputs the classification result corresponding to each proposal.

[0071] Loss function optimization: The training of the entire model is optimized by combining the loss functions of different tasks. The loss function includes action classification loss and position prediction loss, which aims to improve the accuracy of the model in action classification and positioning.

[0072] The overall loss function combines the ensemble prediction loss and the action classification loss:

[0073]

[0074] The ensemble prediction loss includes the classification term The cross entropy loss is used to calculate the two categories (people and background), as well as the box regression term and They are defined as smooth L1 and generalized IOU loss terms respectively, and the action classification loss is Used to generate accurate action recognition results.

[0075] In order to verify the performance of the above method, the following experiment was designed in this embodiment.

[0076] This example evaluates the performance of the method of the present invention on the challenging AVA, UCF101-24, and JHMDB51-21 benchmark datasets.

[0077] AVA is a large-scale benchmark dataset containing 299 15-minute videos divided into 211,000 training clips and 57,000 validation clips. The dataset annotates bounding boxes and labels in videos at a frequency of 1 frame per second (1FPS). Evaluation was performed using two annotation versions of AVA, v2.1 and v2.2. UCF101-24 is a subset of the UCF101 dataset and contains 3,207 videos from 24 motion categories. Each video contains only one action category. JHMDB51-21 contains 928 cropped videos of 21 action categories.

[0078] The results of the entire experiment are as follows:

[0079] The proposed method is compared with recent state-of-the-art methods, including: WOO, TubeR, VideoMAE, EVAD, STMixer, BMViT, etc. The results are detailed in Table 1 and Table 2.

[0080] Table 1 Comparison with the most advanced methods on AVA

[0081]

[0082]

[0083] Table 2 Comparison with state-of-the-art methods on UCF101-24 and JHMDB51-21

[0084]

[0085] (1) Results on the AVA dataset: Table 1 compares the performance of the proposed method with the current state-of-the-art models on the AVAv2.1 and v2.2 datasets. Our method outperforms existing methods in both architectures, with ViT-B and ViT-L achieving mAPs of 40.0 and 41.9, respectively, significantly better than VideoMAE (31.8 and 39.3), without the use of an external person detector. In addition, our method also outperforms EVAD (37.7 and 39.7), STMixer, and BMViT, and significantly outperforms other end-to-end models such as TubeR (34.8) and WOO (28.3). These results fully demonstrate the effectiveness of our label pruning strategy and its overall design.

[0086] (2) Results on UCF101-24 and JHMDB51-21 datasets: Table 2 compares the performance of our method with the current state-of-the-art models on the UCF101-24 and JHMDB51-21 datasets. For a fair comparison, the results of EVAD are re-implemented here. It is worth noting that our method achieves an excellent accuracy of 89.6% on the JHMDB51-21 dataset and 84.0% on the UCF101-24 dataset, with overall performance ahead of all other models.

[0087] The trained model is able to effectively perform the task of action detection in videos.

[0088] Compared with the prior art, the present invention has the following beneficial effects:

[0089] 1. Significantly reduce redundant information: Through the prompt cropping mechanism, the redundant information in the video sequence is effectively removed, which significantly reduces the computational complexity. At the same time, the key features that are crucial to the action detection task are retained, thereby improving the computational efficiency of the model.

[0090] 2. Improve action detection accuracy: Guide attention calculation through learnable prefix cues, strengthen the model's focus on key action clues, so that actions can still be accurately located and identified under complex backgrounds or video interference, thereby improving detection accuracy.

[0091] 3. Strong versatility: Extensive experiments on multiple public datasets (such as AVA, UCF101-24, and JHMDB51-21) have verified the versatility of this method, which can adapt to different video action detection task scenarios while taking into account both efficiency and performance.

[0092] Figure 2Schematic diagram of the comparison between the proposed method and existing methods. Left: The ViT model often faces challenges in action localization because attention calculations are easily affected by irrelevant background and redundant information in video data. Right: The proposed hint-based pruning process reduces the computational burden by retaining only key information, thereby improving the performance of the model in action detection tasks.

[0093] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A prompt cropping method for video action detection, characterized in that: The method comprises the following steps: 1) Obtain video data, and the ViT model performs tokenization processing on the video data, sets key frame tags for key frames of the video data, sets non-key frame tags for non-key frames of the video data, and sets prompt tags at the same time. The sequence of prompt tags, the sequence of key frame tags, and the sequence of non-key frame tags together constitute input tags, and the video data with input tags is used as the tokenized video data; 2) The video data after tokenization is input into the video action detection model. The transform self-attention mechanism of the video action detection model linearly transforms the input token into a query, key and value matrix, calculates the attention map between the prompt token and the key frame token, calculates the importance score of the prompt token based on the attention map between the prompt token and the key frame token, performs prompt token cropping based on the importance score of the prompt token to obtain cropped prompt token, calculates the importance score of the key frame token to crop the key frame token to obtain cropped key frame token, and crops the non-key frame token based on the importance score of the non-key frame token to obtain cropped non-key frame token; 3) The features marked with the cropped key frame markers are respectively subjected to multi-scale adjustment and fusion, and a bounding box is used to perform a region alignment operation on the cropped key frame markers and the cropped non-key frame markers to generate a set of proposal markers. The proposal markers and the cropped hint markers are passed through a decoder of a video action detection model to output a final feature sequence. The final feature sequence includes proposal features marked with the proposal markers and hint features marked with the cropped hint markers, and a video action recognition result is obtained based on the final feature sequence.

2. The method for prompting and cutting video action detection according to claim 1, characterized in that: The input notation is z=[P, T key , T non ], where P represents a sequence of prompt markers, T key represents a sequence of key frame markers, T non Represents a sequence of non-keyframe markers. For video data of size T×H×W×C, where T, H, W, and C represent the number of time frames, height, width, and number of channels, respectively, the video data is divided into small cubes of size t×h×w, t, h, and w are the number of time frames, height, and width of the small cube after division, respectively. The number of markers in the time dimension is The number of markers in the height dimension is The number of markers in the width dimension is Marked with The central frame of the marker in the time dimension is defined as the key frame, and all other frames are considered non-key frames. The sequence T composed of key frame markers key The dimension is 1×n h ×n w ×d, a sequence of non-keyframe markers T non The dimension is (n t -1)×n h ×n w ×d, indicating that the sequence P of markers includes N P prompt tags, d represents the dimension of the tag, and the total number of input tags is N all =N P +N key +N non , where the number of key frame markers is N key =1×n h ×n w , number of non-keyframe markers N non =(n t -1)×n h ×n w .

3. The method for prompting and cutting video action detection according to claim 2, characterized in that: The importance scores of the hint tags are: Among them, I P (i) represents the importance score of the i-th hint tag, A key represents the attention map between the prompt marker and the key frame marker. The attention map between the prompt marker and the key frame marker is obtained by the self-attention mechanism. Calculate, where Q and K represent the query matrix and the key matrix.

4. The method for prompting and cutting video action detection according to claim 3, characterized in that: The prompt tags are cropped based on their importance scores, and the cropped prompt tags are as follows: Keep the first k prompts with the largest importance scores, k = N′ P =N P ·ρ P , ρ P is the retention ratio of the prompt mark, and the remaining prompt marks are cropped to obtain the cropped prompt mark P′.

5. The method for prompting and cutting video action detection according to claim 4, characterized in that: The importance scores of keyframe markers are: Among them, A′ key (i, j) represents the attention map after setting a binary mask on the reserved cue marker positions.

6. The method for prompting and cutting video action detection according to claim 5, characterized in that: The importance scores of key frame markers are calculated to perform key frame marker cropping. The specific steps to obtain cropped key frame markers are as follows: Calculate the importance scores of key frame markers and keep the first N′ key The key frame label with the largest importance score, N′ key =N key ·ρ key , ρ key is the retention ratio of the key frame mark, and the cropped key frame mark T′ is obtained key .

7. The method for prompting and cutting video action detection according to claim 6, characterized in that: The importance scores of non-keyframe markers are: Among them, α is the weight coefficient for the hint mark and key frame mark, N′ all =N′ P +N′ key +N non , A non Represents N′ all N non The attention weight map calculated from the non-keyframe labels.

8. The method for prompting and cutting video action detection according to claim 7, characterized in that: The specific steps of cropping the non-key frame tags based on the importance scores of the non-key frame tags to obtain cropped non-key frame tags are: Keep the first N′ non The non-keyframe tags with the largest importance scores, N′ non =N non ·ρ non , where ρ non is the retention ratio of non-keyframe labels, and the cropped non-keyframe labels T′ are obtained non .

9. The method for prompting and cutting video action detection according to claim 1, characterized in that: The overall loss function of the video action detection model combines the ensemble prediction loss and the action classification loss.

10. The method for prompting and cutting video action detection according to claim 9, characterized in that: The overall loss function is: Among them, λ cls , L1 , giou and λ act They are respectively the classification loss Smooth L1 box regression loss Generalized IOU regression loss and action classification loss The weight of is the overall loss function.

Images