A cue cropping method for video action detection

Abstract

The present invention relates to a prompt cropping method for video action detection, comprising the following steps: 1) acquiring video data and setting non-keyframe markers, prompt markers, and keyframe markers; 2) inputting the tokenized video data into a video action detection model to obtain cropped keyframe markers; 3) passing the proposed markers and the cropped prompt markers through a decoder of the video action detection model to output a final feature sequence, the final feature sequence comprising proposed features marked with the proposed markers and prompt features marked with the cropped prompt markers; and obtaining a video action recognition result based on the final feature sequence. Compared with the prior art, the present invention has the advantages of significantly reducing computational complexity while retaining keyframes to achieve comprehensive optimization of video action detection.

Description

Technical Field

[0001] The present invention relates to the technical field of video action detection, and in particular to a prompt cropping method for video action detection. Background Art

[0002] Video action detection, which aims to locate and classify actions in video sequences, is an important and challenging task in computer vision. In recent years, unsupervised Vision Transformer (ViT) models have demonstrated remarkable success in video understanding tasks. However, these ViT models still face difficulties in accurately localizing actions, especially when attention computation is affected by irrelevant background noise and the high redundancy of video data.

[0003] Currently, mainstream video action detection models generally adopt the strategy of "localization first, classification later". These models usually use a single key frame (usually an intermediate frame) to locate the object, and then use the entire video sequence to capture the motion and context information required for action recognition. However, video sequences often contain a lot of redundant information (such as Figure 2 (shown on the left), which leads to inefficient attention computation and increased computational overhead. Previous studies have attempted to reduce redundancy by cropping non-keyframes, but to maintain detection accuracy, these methods typically avoid cropping keyframes. As a result, these methods lack effective guidance strategies for cross-label cropping and can only achieve local optimization. Summary of the Invention

[0004] The purpose of the present invention is to provide a cold chain vehicle path optimization method that takes into account the on-demand recovery of turnover transport units in order to significantly reduce the computational complexity while retaining key frames to achieve comprehensive optimization of video motion detection.

[0005] The purpose of the present invention can be achieved by the following technical solutions:

[0006] A prompt cropping method for video action detection, the method comprising the following steps:

[0007] 1) Obtain video data. The ViT model tokenizes the video data by setting keyframe tags for keyframes of the video data, setting non-keyframe tags for non-keyframes of the video data, and setting prompt tags. The sequence of prompt tags, the sequence of keyframe tags, and the sequence of non-keyframe tags together constitute the input tags. The video data with the input tags serves as the tokenized video data.

[0008] 2) The tokenized video data is input into the video action detection model. The transform self-attention mechanism of the video action detection model linearly transforms the input tokens into query, key, and value matrices, calculates the attention map between the prompt token and the keyframe token, calculates the importance score of the prompt token based on the attention map between the prompt token and the keyframe token, crops the prompt token based on the importance score of the prompt token to obtain cropped prompt tokens, calculates the importance score of the keyframe token to crop the keyframe token to obtain cropped keyframe tokens, and crops the non-keyframe tokens based on the importance score of the non-keyframe tokens to obtain cropped non-keyframe tokens;

[0009] 3) The features marked with the cropped key frame markers are respectively subjected to multi-scale adjustment and fusion, and a bounding box is used to perform a region alignment operation on the cropped key frame markers and the cropped non-key frame markers to generate a set of proposal markers. The proposal markers and the cropped hint markers are passed through the decoder of the video action detection model to output a final feature sequence. The final feature sequence includes the proposal features marked with the proposal markers and the hint features marked with the cropped hint markers, and the video action recognition result is obtained based on the final feature sequence.

[0010] Furthermore, the input is marked as z=[P, T key , T non ], where P represents a sequence of prompt markers, T key represents a sequence of key frame markers, T non Represents a sequence of non-keyframe markers. For video data of size T×H×W×C, where T, H, W, and C represent the number of time frames, height, width, and number of channels, respectively, the video data is divided into small cubes of size t×h×w. T, h, and w are the number of time frames, height, and width of the small cube after division, respectively. The number of markers in the time dimension is The number of markers in the height dimension is The number of markers in the width dimension is Marked with The central frame of the marker in the time dimension is defined as the key frame, and all other frames are considered non-key frames. The sequence T composed of key frame markers key The dimension is 1×n h ×n w ×d, a sequence T consisting of non-keyframe labels non The dimension is (n t -1)×n h ×n w ×d, indicating that the sequence P of markers includes N P prompt tags, d represents the dimension of the tag, and the total number of tags N is entered. all =N P +Nkey +N non , where the number of key frame markers N key =1×n h ×n w , number of non-keyframe markers N non =(n t -1)×n h ×n w .

[0011] Furthermore, the importance score of the hint tag is:

[0012]

[0013] Among them, I P (i) represents the importance score of the i-th hint tag, A key Represents the attention map between the prompt tag and the key frame tag, and the attention map between the prompt tag and the key frame tag is obtained by the self-attention mechanism between the prompt tag and the key frame tag. Calculate, where Q and K represent the query matrix and the key matrix.

[0014] Furthermore, the prompt tags are cropped based on their importance scores, and the cropped prompt tags are specifically:

[0015] Keep the top k hint tags with the largest importance scores, k = N′ P =N P ·ρ P , ρ P is the retention ratio of the prompt marker, and the remaining prompt markers are cropped to obtain the cropped prompt marker P′.

[0016] Furthermore, the importance score of the keyframe marker is:

[0017]

[0018] Among them, A′ key (i, j) represents the attention map after setting a binary mask on the retained cue marker positions.

[0019] Furthermore, the importance scores of the key frame markers are calculated to perform key frame marker cropping. The specific steps for obtaining the cropped key frame markers are as follows:

[0020] Calculate the importance scores of keyframe tags and keep the first N′ key The key frame label with the largest importance score, N′ key =N key ·ρ key , ρ keyis the retention ratio of the key frame mark, and the cropped key frame mark T′ is obtained key .

[0021] Furthermore, the importance score of non-keyframe tags is:

[0022]

[0023] Where α is the weighting coefficient for the hint mark and key frame mark, N′ all =N′ P +N′ key +N non , A non Represents N′ all Marker pair N non The attention weight map is calculated by non-keyframe labels.

[0024] Furthermore, the specific steps of cropping the non-key frame tags based on the importance scores of the non-key frame tags to obtain the cropped non-key frame tags are:

[0025] Keep the first N′ non The non-keyframe tags with the largest importance score, N′ non =N mon ·ρ non , where ρ non is the retention ratio of non-keyframe labels, and the cropped non-keyframe labels T′ are obtained non .

[0026] Furthermore, the overall loss function of the video action detection model combines the set prediction loss and the action classification loss.

[0027] Furthermore, the overall loss function is:

[0028]

[0029] Among them, λ cls ,λ L1 ,λ giou and λ act The classification loss is Smooth L1 box regression loss Generalized IOU regression loss and action classification loss The weight of is the overall loss function.

[0030] Compared with the prior art, the present invention has the following beneficial effects:

[0031] Through the prompt cropping mechanism, redundant information in the video sequence is effectively eliminated, which significantly reduces the computational complexity. At the same time, key features that are crucial to the action detection task are retained, thereby improving the computational efficiency of the video action detection model. The present invention also guides attention calculations through learnable prefix prompts, strengthening the model's attention to key action clues, so that actions can still be accurately located and identified under complex backgrounds or video interference, thereby comprehensively improving the detection accuracy of action detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 This is a schematic diagram of the overall architecture of the method of the present invention, wherein Figure 1 (a) is a schematic diagram of video input and prompts. Figure 1 (b) is a schematic diagram of marker cropping optimization. Figure 1 (c) is a schematic diagram of the positioning module. Figure 1 (d) is a schematic diagram of the classification module;

[0033] Figure 2 Schematic diagram for comparing the method of the present invention with the existing method. DETAILED DESCRIPTION

[0034] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0035] The present invention proposes a prompt cropping method for video action detection, which specifically includes the following steps: first, using an adjustable prompt vector to optimize the input of the model so that it pays more attention to key task-related information; then, using the attention map generated by the self-attention mechanism, the label information in the video data is cropped in stages. The cropping process is divided into three steps: the first step is to crop the prompt vector, retaining the part that can most effectively guide the model; the second step is to filter the key frame labels, retaining only the labels containing important spatial information; the third step is to crop the non-key frame labels, retaining only the parts that are critical to action understanding. Through these steps, the cropping process can significantly reduce the amount of computation, while retaining the key information required for action detection to ensure the accuracy of action detection. Compared with the existing technology, the present invention has the advantages of high action detection accuracy, strong robustness and high computational efficiency.

[0036] The overall architecture diagram of the method is as follows Figure 1 As shown, the method includes:

[0037] 1) Obtain video data. The ViT model tokenizes the video data by setting keyframe tags for keyframes of the video data, setting non-keyframe tags for non-keyframes of the video data, and setting prompt tags. The sequence of prompt tags, the sequence of keyframe tags, and the sequence of non-keyframe tags together constitute the input tags. The video data with the input tags serves as the tokenized video data.

[0038] 2) The tokenized video data is input into the video action detection model. The transform self-attention mechanism of the video action detection model linearly transforms the input tokens into query, key, and value matrices, calculates the attention map between the prompt token and the keyframe token, calculates the importance score of the prompt token based on the attention map between the prompt token and the keyframe token, crops the prompt token based on the importance score of the prompt token to obtain cropped prompt tokens, calculates the importance score of the keyframe token to crop the keyframe token to obtain cropped keyframe tokens, and crops the non-keyframe tokens based on the importance score of the non-keyframe tokens to obtain cropped non-keyframe tokens;

[0039] 3) The features marked with the cropped key frame markers are respectively subjected to multi-scale adjustment and fusion, and a bounding box is used to perform a region alignment operation on the cropped key frame markers and the cropped non-key frame markers to generate a set of proposal markers. The proposal markers and the cropped hint markers are passed through the decoder of the video action detection model to output a final feature sequence. The final feature sequence includes the proposal features marked with the proposal markers and the hint features marked with the cropped hint markers, and the video action recognition result is obtained based on the final feature sequence.

[0040] In 1), video input and prompts are performed: First, the input video is processed through a sequence containing learnable prefix prompts. These prompts can help the model better understand the video content, especially when it comes to complex action recognition, where prefix prompts can provide necessary contextual information.

[0041] The video input processing step involves representing the input video clip as a three-dimensional tensor containing temporal and spatial information. This is then converted into a set of video tags by dividing the video into multiple small blocks. Specifically, the video is tokenized using block embedding techniques to generate a video tag sequence. The middle frames in the sequence are then defined as keyframes, and the remaining frames are defined as non-keyframes.

[0042] Vision Transformer (ViT) tokenizes videos using cube embedding. Specifically, the size of a video segment V is T×H×W×C, where T, H, W, and C represent the number of time frames, height, width, and channels, respectively. The video segment is divided into small cubes of size t×h×w to generate a token sequence. After division, the number of tokens in each dimension is calculated as follows: The number of tokens in the time dimension is The height dimension is The width dimension is The feature dimension of each tag is d, and the tag sequence is reorganized into a one-dimensional sequence after position embedding. The central frame in the video clip is defined as the key frame, denoted as time markers, all other frames are considered non-keyframes.

[0043] The central frame in the video clip is defined as the key frame, denoted as Time markers, all other frames are considered non-key frames. The marker sequence of key frames is defined as T key , whose dimension is 1×n h ×n w ×d. The label sequence of non-key frames is defined as T non , the dimension is (n t -1)×n h ×n w ×d. The total number of markers includes the number of key frame markers N key =1×n h ×n w and the number of non-keyframe markers N non =(n t -1)×n h ×n w , the total number is N = n t ×n h ×n w .

[0044] To enhance the representation ability of the model, a learnable prefix hint is introduced before inputting to the first layer of Transformer. Each hint token is a d-dimensional vector. Assume that the hint token set contains N P tokens, denoted as P. The final input sequence is z = [P, T key , T non ], the total dimension is where N all =N P +N key +N non .

[0045] The step of adding learnable prefix cues involves adding a set of learnable cue tokens before the first layer of the Transformer model. These cue tokens consist of a set of trainable vectors, represented as a set. These cue tokens are concatenated with the video tokens to form the input sequence, which serves as the input for the subsequent Transformer encoder.

[0046] 2) Marker pruning optimization: In the backbone network of the video, prefix cues are combined to perform marker processing. Through a pruning mechanism, markers in the video are selectively retained and redundant markers are removed to reduce computational complexity and improve processing efficiency. The pruned markers are optimized using various processing methods, retaining keyframe and important non-keyframe markers to ensure the model focuses on the most important action information.

[0047] The step of adding learnable prefix cues involves adding a set of learnable cue tokens before the first layer of the Transformer model. These cue tokens consist of a set of trainable vectors, represented as a set. These cue tokens are concatenated with the video tokens to form the input sequence, which serves as the input for the subsequent Transformer encoder.

[0048] The cue marker pruning step involves calculating the correlation score between each cue marker and the keyframe marker based on the attention map, filtering out the most important cue markers. The cue markers with the highest scores are retained, while those with low correlation are pruned, resulting in an optimized cue marker set.

[0049] The keyframe marker cropping step uses the results of the cue marker cropping to further filter the keyframe markers. Based on the correlation between the cue markers and the keyframe markers in the attention graph, the importance score of each keyframe marker is calculated, and the keyframe markers with the highest scores are retained to obtain the cropped keyframe marker set.

[0050] The non-keyframe markers are trimmed based on the retained cue and keyframe markers. The importance scores of the non-keyframe markers are calculated by comprehensively analyzing their correlation with the cue and keyframe markers. The non-keyframe markers with higher scores are then selected to form the trimmed non-keyframe marker set.

[0051] The final marker sequence is generated by combining the cropped cue markers, keyframe markers, and non-keyframe markers to form a final marker sequence. This sequence serves as input for subsequent models performing video action detection tasks, improving detection performance while reducing computational complexity.

[0052] The above cropping process can significantly reduce the amount of computation while retaining sufficient information to ensure the accuracy of action detection.

[0053] To evaluate the importance of a token, the attention map is calculated using the self-attention mechanism. In self-attention, the input token is linearly transformed into query, key, and value matrices Q, K, and V. The attention operation is calculated as:

[0054]

[0055] The attention map A is defined as For a multi-head attention layer, the attention maps of all heads are averaged.

[0056] (i) Tip mark clipping

[0057] Hint marker cropping aims to retain the hint markers that contribute more to the key frame. First, the attention map A between the hint marker and the key frame marker is calculated. key , and then calculate the importance score of each hint tag based on the attention map. Specifically, the importance score of the i-th hint tag is defined as:

[0058]

[0059] Keep the first k important prompt tags, where k = N′ P =N P ·ρ P ,ρ P is the proportion of the hint markers that are retained. The remaining hint markers are pruned. The pruned hint markers are denoted as P′.

[0060] (ii) Keyframe Marker Cropping

[0061] Based on the retained hint tags, the keyframe tags are further pruned. A binary mask is defined to mark the entries that are retained in the hint tag pruning stage. The pruned attention map A′ is calculated using the mask key . The importance scores of the keyframe markers are then calculated:

[0062]

[0063] Keep the first N′ key key frame markers, where N′ key =N key ·ρ key ,ρ key is the retention ratio of key frame markers. The cropped key frame markers are recorded as T′ key .

[0064] (iii) Non-keyframe marker cropping

[0065] For non-keyframe tags, the relevance to them is calculated based on the cropped hint tags and keyframe tags. Specifically, the importance score of the i-th non-keyframe tag is defined as:

[0066]

[0067] Where α is the weight coefficient for the hint mark and key frame mark, the purpose is to give priority to retaining the non-key frame marks that are strongly correlated with these two types of marks. Finally, retain the first N′ non non-keyframe markers, N′ non =N non ·ρ non , where ρ non is the retention ratio of non-keyframe markers. The cropped non-keyframe markers are recorded as T′ non .

[0068] In 3), localization and classification are performed: the localization module generates the spatial location and time period of the action. The localization module generates the corresponding action position prediction based on the pruned markers. The classification module is responsible for generating action category predictions based on the pruned markers and identifying different action types in the video.

[0069] The localization module is implemented based on Sparse R-CNN. In the 4th, 7th, 10th, and 12th layers of the backbone network, features from keyframe markers undergo multi-scale rescaling: features are scaled up 4x in the 4th layer, 2x in the 7th layer, unchanged in the 10th layer, and scaled down to 2x in the 12th layer. These rescaled feature maps are then fused at multiple scales using a feature pyramid network. Next, candidate regions are generated by initializing a certain number of learnable proposal boxes and their features, ultimately generating bounding boxes and corresponding object confidence scores.

[0070] The classification module uses the bounding boxes generated by the positioning module to perform region alignment on the keyframe and non-keyframe tags to generate a set of proposed tags T prop These proposed tags are then concatenated with the sequence features and fed into the Transformer decoder. The decoder, consisting of a multi-head self-attention layer and a feedforward network, outputs decoded features. The final feature sequence is divided into two parts: prompt features and proposal features. After these features are processed and averaged by the feedforward network (FFN), the classification layer predicts the action category and outputs the classification result for each proposal.

[0071] Loss function optimization: The entire model training is optimized by combining loss functions for different tasks. The loss functions include action classification loss and position prediction loss, aiming to improve the model's accuracy in action classification and localization.

[0072] The overall loss function combines the ensemble prediction loss and the action classification loss:

[0073]

[0074] The ensemble prediction loss includes the classification term The cross entropy loss is used to calculate the two categories (people and background), as well as the box regression term and They are defined as smooth L1 and generalized IOU loss terms respectively, and the action classification loss is Used to generate accurate action recognition results.

[0075] In order to verify the performance of the above method, the following experiments were designed in this embodiment.

[0076] This example evaluates the performance of the method of the present invention on the challenging AVA, UCF101-24, and JHMDB51-21 benchmark datasets.

[0077] AVA is a large-scale benchmark dataset containing 299 15-minute videos divided into 211,000 training segments and 57,000 validation segments. The dataset annotates the videos with bounding boxes and labels at a frequency of 1 frame per second (1FPS). Evaluation was performed using two annotation versions of AVA, v2.1 and v2.2. UCF101-24 is a subset of the UCF101 dataset and contains 3,207 videos from 24 motion categories. Each video contains only one action category. JHMDB51-21 contains 928 cropped videos from 21 action categories.

[0078] The results of the entire experiment are as follows:

[0079] The proposed method is compared with recent state-of-the-art methods, including WOO, TubeR, VideoMAE, EVAD, STMixer, BMViT, etc. The results are detailed in Table 1 and Table 2.

[0080] Table 1 Comparison with the most advanced methods on AVA

[0081]

[0082]

[0083] Table 2 Comparison with state-of-the-art methods on UCF101-24 and JHMDB51-21

[0084]

[0085] (1) Results on the AVA dataset: Table 1 compares the performance of the proposed method with the current state-of-the-art models on the AVAv2.1 and v2.2 datasets. The proposed method outperforms existing methods in both architectures, with ViT-B and ViT-L achieving mAPs of 40.0 and 41.9, respectively, significantly outperforming VideoMAE (31.8 and 39.3), without the use of an external person detector. In addition, the performance of the proposed method also surpasses EVAD (37.7 and 39.7), STMixer, and BMViT, and significantly outperforms other end-to-end models such as TubeR (34.8) and WOO (28.3). These results fully demonstrate the effectiveness of the label pruning strategy of the proposed method and its overall design.

[0086] (2) Results on the UCF101-24 and JHMDB51-21 datasets: Table 2 compares the performance of our method with the state-of-the-art models on the UCF101-24 and JHMDB51-21 datasets. For a fair comparison, the results of EVAD are reimplemented here. Notably, our method achieves an excellent accuracy of 89.6% on the JHMDB51-21 dataset and 84.0% on the UCF101-24 dataset, outperforming all other models overall.

[0087] The trained model is able to effectively perform the task of action detection in videos.

[0088] Compared with the prior art, the present invention has the following beneficial effects:

[0089] 1. Significantly reduce redundant information: Through the prompt cropping mechanism, redundant information in the video sequence is effectively eliminated, significantly reducing the computational complexity, while retaining the key features that are crucial for the action detection task and improving the computational efficiency of the model.

[0090] 2. Improve action detection accuracy: Guide attention calculations through learnable prefix cues, strengthen the model's focus on key action cues, and thus accurately locate and identify actions in complex backgrounds or video interference, thereby improving detection accuracy.

[0091] 3. Strong versatility: Extensive experiments on multiple public datasets (such as AVA, UCF101-24, and JHMDB51-21) have verified the versatility of this method, which can adapt to different video action detection task scenarios while balancing efficiency and performance.

[0092] Figure 2A schematic diagram comparing our method with existing methods. Left: ViT models often face challenges in action localization because attention computation is easily affected by irrelevant background and redundant information in the video data. Right: The proposed hint-based pruning process reduces the computational burden by retaining only critical information, thereby improving the model's performance in action detection tasks.

[0093] The above describes in detail the preferred embodiments of the present invention. It should be understood that those skilled in the art can make numerous modifications and variations based on the concepts of the present invention without inventive effort. Therefore, any technical solutions that can be derived by those skilled in the art through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art should be within the scope of protection defined by the claims.

Claims

1. A prompt cropping method for video action detection, characterized in that: The method comprises the following steps: 1) Obtain video data. The ViT model tokenizes the video data by setting keyframe tags for keyframes of the video data, setting non-keyframe tags for non-keyframes of the video data, and setting prompt tags. The sequence of prompt tags, the sequence of keyframe tags, and the sequence of non-keyframe tags together constitute the input tags. The video data with the input tags serves as the tokenized video data. 2) The tokenized video data is input into the video action detection model. The transform self-attention mechanism of the video action detection model linearly transforms the input tokens into query, key, and value matrices, calculates the attention map between the prompt token and the keyframe token, calculates the importance score of the prompt token based on the attention map between the prompt token and the keyframe token, crops the prompt token based on the importance score of the prompt token to obtain cropped prompt tokens, calculates the importance score of the keyframe token to crop the keyframe token to obtain cropped keyframe tokens, and crops the non-keyframe tokens based on the importance score of the non-keyframe tokens to obtain cropped non-keyframe tokens; 3) The features marked with the cropped key frame markers are respectively subjected to multi-scale adjustment and fusion, and a bounding box is used to perform a region alignment operation on the cropped key frame markers and the cropped non-key frame markers to generate a set of proposal markers. The proposal markers and the cropped hint markers are passed through the decoder of the video action detection model to output a final feature sequence. The final feature sequence includes the proposal features marked with the proposal markers and the hint features marked with the cropped hint markers, and the video action recognition result is obtained based on the final feature sequence.

2. The method for prompting and cutting video action detection according to claim 1, characterized in that: The input notation is z=[P, T key , T non ], where P represents a sequence of prompt markers, T key represents a sequence of key frame markers, T non Represents a sequence of non-keyframe markers. For video data of size T×H×W×C, where T, H, W, and C represent the number of time frames, height, width, and number of channels, respectively, the video data is divided into small cubes of size t×h×w. T, h, and w are the number of time frames, height, and width of the small cube after division, respectively. The number of markers in the time dimension is The number of markers in the height dimension is The number of markers in the width dimension is Marked with The central frame of the marker in the time dimension is defined as the key frame, and all other frames are considered non-key frames. The sequence T composed of key frame markers key The dimension is 1×n h ×n w ×d, a sequence T consisting of non-keyframe labels non The dimension is (n t -1)×n h ×n w ×d, indicating that the sequence P of markers includes N P prompt tags, d represents the dimension of the tag, and the total number of tags N is entered. all =N P +N key +N non , where the number of key frame markers N key =1×n h ×n w , number of non-keyframe markers N non =(n t -1)×n h ×n w .

3. The method for prompting and cutting video action detection according to claim 2, wherein: The importance scores of the hint tags are: Among them, I P (i) represents the importance score of the i-th hint tag, A key Represents the attention map between the prompt tag and the key frame tag, and the attention map between the prompt tag and the key frame tag is obtained by the self-attention mechanism between the prompt tag and the key frame tag. Calculate, where Q and K represent the query matrix and the key matrix.

4. The method for prompting and cutting video action detection according to claim 3, wherein: The prompt tags are cropped based on their importance scores, and the cropped prompt tags are as follows: Keep the top k hint tags with the largest importance scores, k = N′ P =N P ·ρ P , ρ P is the retention ratio of the prompt marker, and the remaining prompt markers are cropped to obtain the cropped prompt marker P′.

5. The method for prompting and cutting video action detection according to claim 4, wherein: The importance scores of keyframe markers are: Among them, A′ key (i, j) represents the attention map after setting a binary mask on the retained cue marker positions.

6. The method for prompting and cutting video action detection according to claim 5, characterized in that: The importance scores of key frame markers are calculated to perform key frame marker cropping. The specific steps to obtain cropped key frame markers are as follows: Calculate the importance score of key frame tags and keep the first N′ key The key frame label with the largest importance score, N′ key =N key ·ρ key , ρ key is the retention ratio of the key frame mark, and the cropped key frame mark T′ is obtained key .

7. The method for prompting and cutting video action detection according to claim 6, wherein: The importance scores of non-keyframe markers are: Where α is the weighting coefficient for the hint mark and key frame mark, N′ all =N′ P +N′ key +N non , A non Represents N′ all Marker pair N non The attention weight map is calculated by non-keyframe labels.

8. The method for prompting and cutting video action detection according to claim 7, characterized in that: The specific steps of cropping the non-key frame tags based on the importance scores of the non-key frame tags to obtain the cropped non-key frame tags are: Keep the first N′ non The non-keyframe tags with the largest importance score, N′ non =N non ·ρ non , where ρ non is the retention ratio of non-keyframe labels, and the cropped non-keyframe labels T′ are obtained non .

9. The method for prompting and cutting video action detection according to claim 1, wherein: The overall loss function of the video action detection model combines the set prediction loss and the action classification loss.

10. The method for prompting and cutting video action detection according to claim 9, characterized in that: The overall loss function is: Among them, λ cls ,λ L1 ,λ giou and λ act The classification loss is Smooth L1 box regression loss Generalized IOU regression loss and action classification loss The weight of is the overall loss function.

Images