Online action detection method based on lightweight transformer

Abstract

The application discloses an online action detection method based on a light-weight Transformer, and comprises the following steps: setting a detection model, wherein the detection model is based on a Transformer model; acquiring features of a video stream with a length of T; inputting the features of the video stream into the detection model; and outputting an action category of the last frame of the video stream through the detection model as a detection result. The online action detection method based on the light-weight Transformer disclosed by the application reduces the calculation amount while improving the accuracy.

Description

Technical Field

[0001] This invention relates to an action detection method, and more particularly to an online action detection method based on a lightweight Transformer, belonging to the field of artificial intelligence technology. Background Technology

[0002] Online motion detection identifies the action happening in each frame of a streaming video under conditions where the future is not visible. Since only a portion of the video stream is visible for each action, online motion detection is more difficult than offline motion detection.

[0003] Traditional online action detection mostly uses recurrent neural networks (RNNs) to model the temporal dependencies of consecutive frames. For example, the Information Discrimination Network (IDN) improves upon the GRU to better encode historical information and determine the type of action in the current frame. However, this approach suffers from the trap of recurrent neural modeling.

[0004] Existing technologies have also proposed online action detection methods based on Transformers, which leverage the excellent global modeling capabilities of Transformers to capture the temporal context information of actions in videos. Compared with RNN structures, Transformers utilize self-attention mechanisms to regress non-local temporal correlations, directly store history to avoid the pitfalls of recurrent neural models, and have better convergence and higher computational efficiency, such as Transformers-based online action detection (OadTR).

[0005] While self-attention in Transformers offers powerful temporal modeling capabilities, it also incurs high time and space complexity. Furthermore, the basic Transformer models the global relationships between input tokens, resulting in significant computational cost and a large number of parameters.

[0006] Therefore, it is necessary to conduct more in-depth research to reduce the computational load. Summary of the Invention

[0007] To overcome the above problems, the inventors conducted in-depth research and designed an online motion detection method based on a lightweight Transformer, comprising:

[0008] Set a detection model, which is based on the Transformer model;

[0009] Extract features from a video stream of length T;

[0010] The features of the video stream are input into the detection model, and the action category of the last frame of the video stream is output by the detection model as the detection result.

[0011] In a preferred embodiment, the detection model has a self-attention sub-model, which includes multiple attention layers.

[0012] The input to the attention layer is the feature of the video stream or the attention vector output from the previous attention layer, and the output is the attention vector.

[0013] In a preferred embodiment, the attention layer only represents the correlation between the current frame and each historical frame.

[0014] In a preferred embodiment, in the attention layer, the attention value is based on the cosine similarity between the key matrices of the current frame and the historical frames.

[0015] In a preferred embodiment, the self-attention quantum model includes the following steps:

[0016] S11. Take the video stream features as input, process them using a convolutional layer to obtain a key matrix and a value matrix. The key matrix contains the key vectors of each frame, and the value matrix contains the value vectors of each frame.

[0017] S12. Use the cosine similarity between the current frame key vector and the historical frame key vector as the attention value output by the current attention layer.

[0018] S13. Calculate the weighted sum of the current frame value vector and the historical frame value vector based on the attention value, and output the attention vector;

[0019] S14. Use the attention vector output from the previous attention layer as the input to the next attention layer, and repeat steps S11 to S13 to obtain the final self-attention sub-model output.

[0020] In a preferred embodiment, the self-attention sub-model further includes a temporal convolutional layer disposed between two adjacent self-attention layers.

[0021] In a preferred embodiment, a Gaussian prior is also introduced into the self-attention sub-model. By minimizing the temporal similarity between the output features of each attention layer and the KL divergence between the prior Gaussian distribution, the temporal similarity between the features in the attention layer output conforms to a Gaussian distribution on the time axis.

[0022] In a preferred embodiment, the output of the self-attention sub-model is mapped to an embedded feature matrix, and the temporal similarity of the output features of each attention layer is characterized by the embedded features.

[0023] The mapping is achieved through a mapping network h(·) consisting of a fully connected layer, a ReLU activation layer, and another fully connected layer, which is represented as the mapping of the output f″ of the self-attention sub-model. i Mapped to embedded features s i :

[0024] s i =h(f″) i )

[0025] Where s represents i The embedding features at time i, f″ i This represents the output of the self-attention sub-model;

[0026] The temporal similarity of the output features of each attention layer is represented as follows:

[0027]

[0028]

[0029] Where, μ ij Cosine similarity, denoted as μ ij =cos(s i ,s j ), where i,j represent different times, and G(·) represents a Gaussian distribution, denoted as x is the independent variable of a Gaussian distribution.

[0030] In a preferred embodiment, the detection model also includes an auxiliary classification process:

[0031] Obtain the typical characteristics of each action category;

[0032] Use the features of the current frame as query conditions to obtain the typical features of the action category corresponding to the current frame;

[0033] The acquired typical features are used as auxiliary information to assist in the action classification of the current frame.

[0034] In a preferred embodiment, the action classification assistance for the current frame is performed in the following manner:

[0035] The features of the current frame are appended to the end of the obtained class canonical features to form a new input sequence. This sequence is then fed into a new attention layer, which outputs an auxiliary feature sequence F. c , in, The auxiliary feature representing the k-th category feature. This represents the auxiliary features corresponding to the current frame;

[0036] Combine the attention value f″0 of the current frame with the auxiliary features corresponding to the current frame. The features are spliced ​​into a frame block feature f0, and the frame block feature f0 is processed by a classifier to output the recognized action.

[0037] The beneficial effects of this invention include:

[0038] (1) By modeling local motion information through convolution, the total number of self-attention layers in the model can be reduced, thereby reducing the computational load and the hardware requirements.

[0039] (2) Only the cosine similarity between the target frame and the historical frame is considered as the attention value, which reduces the computational complexity of the attention value and reduces the amount of computation.

[0040] (3) Between two adjacent self-attention layers, a convolutional layer with a stride of 2 is added to reduce the number of input frames in the next layer by half, thereby reducing the computational load and improving the detection speed.

[0041] (4) Learn category auxiliary features and further extract features useful for action classification, thereby improving detection performance. Attached Figure Description

[0042] Figure 1 The diagram illustrates a flow chart of an online motion detection method based on a lightweight Transformer according to a preferred embodiment of the present invention. Detailed Implementation

[0043] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Through these descriptions, the features and advantages of the present invention will become clearer and more apparent.

[0044] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. Although various aspects of embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated otherwise.

[0045] An online motion detection method based on a lightweight Transformer, provided by the present invention, includes:

[0046] Set a detection model, which is based on the Transformer model;

[0047] Extract features from a video stream of length T;

[0048] The features of the video stream are input into the detection model, and the action category of the last frame of the video stream is output by the detection model as the detection result.

[0049] In this invention, the features of a video stream refer to the set of frame features of each frame in the video.

[0050] In this invention, for ease of description, the video stream is written as a video stream from time -T to 0, that is, the time of the last frame is the current time, and the last frame is also called the current frame.

[0051] The characteristics of a video stream are represented as F = (f -T ,f-T+1 ,…,f0), where This represents the characteristics of the frame at time t.

[0052] In this invention, the frame feature f t The method for obtaining the frame features is not particularly limited, and those skilled in the art can use any existing frame feature extraction method to obtain them.

[0053] In this invention, the input of the detection model is the feature F of the video stream, and the output of the model is the action category y0 of the current frame, where y0∈{0,1,…K}, 1-K are different action categories, that is, the total number of categories is K, and 0 indicates that no action has occurred.

[0054] Furthermore, the detection model has a self-attention sub-model, which includes multiple attention layers. The input of each attention layer is a feature of the video stream or an attention vector output by the previous attention layer, and the output is an attention vector.

[0055] Wherein, the input feature F = (f -T ,f -T+1 After passing through an attention layer, local and global temporal information is fused, and then through a temporal convolutional layer to reduce the number of input tokens for the next layer.

[0056] In traditional Transformers models, self-attention has powerful temporal modeling capabilities, but its time and space complexity is also high.

[0057] Unlike traditional Transformers models, in this application, the attention value only represents the correlation between the current frame and historical frames, and does not include the correlation between historical frames at different locations.

[0058] The inventors discovered that the current frame is particularly important in online action detection tasks. An action should continue within a local time frame, thus adjacent frames provide rich contextual information to the target frame. Furthermore, over a longer temporal range, historical frames also provide useful temporal action information to the target frame. However, the inventors found that calculating attention values ​​between all arbitrary pairs of tokens in a self-attention mechanism is unnecessary. In this invention, the operation of calculating attention values ​​between all pairs of tokens in the traditional attention mechanism is abandoned, so that the attention value only represents the correlation between the current frame and historical frames.

[0059] Furthermore, in this invention, by calculating only the attention values ​​of the current frame and each historical frame, the computational complexity of attention is reduced from O(N) to O(N). 2 The computational complexity of attention values ​​is reduced to O(N), which greatly reduces the computational complexity of attention values.

[0060] Specifically, the self-attention sub-model includes the following steps:

[0061] S11. Using video stream features as input, a temporal convolutional layer is applied to process them, extracting local action information to obtain a key matrix and a value matrix. The key matrix contains the key vectors of each frame, and the value matrix contains the value vectors of each frame.

[0062]

[0063]

[0064] Where K is the key matrix, V is the value matrix, and F is the input video stream feature. k Convolution operations for keys, Conv v Convolution operation with values.

[0065] In this invention, by setting convolutional layers, the ability to model local correlations is improved, which reduces the number of attention layers, thereby reducing the amount of computation and parameters.

[0066] S12. Use the cosine similarity between the current frame key vector and the historical frame key vectors as the attention value λ. t , represented as:

[0067]

[0068] Where, k t Let t be the key vector at time t, and k0 be the key vector at the current time.

[0069] S13. Calculate the weighted sum of the current frame value vector and the historical frame value vectors based on the attention value, and output the attention vector f″. t This is used as the input to the next layer, and is represented as:

[0070] f′0=V·softmax(λ)+v0

[0071]

[0072] f″ t =λ t ·f′0+f′ t

[0073]

[0074] Where λ=[λ -T ,λ -T+1 ,…,λ0] T v t Let f′0 and f′ be the value vectors at time t. t F″ represents the process value, and F″ represents the output of the attention layer.

[0075] S14. Use the attention vector output from the previous attention layer as the input to the next attention layer, and repeat steps S11 to S13 to obtain the final self-attention sub-model output.

[0076] According to a preferred embodiment of the present invention, the self-attention sub-model further includes a temporal convolutional layer, which is set between two adjacent self-attention layers to model local motion information. The number of frames is reduced by half by adjusting the convolution stride, thereby further reducing the feature length input to the next self-attention layer. Specifically, in S14, the attention vector output by the previous attention layer is passed through the temporal convolutional layer and used as the input to the next attention layer.

[0077] In a preferred embodiment, the temporal convolutional layer is a convolutional layer with a kernel size of 3 and a stride of 2.

[0078] In a preferred embodiment, a Gaussian prior is introduced into the self-attention sub-model to ensure that the temporal similarity between features in the attention layer output conforms to the prior distribution on the time axis, and to minimize the KL divergence between this temporal similarity and the prior Gaussian distribution. That is, by minimizing the KL divergence between the temporal similarity and the prior Gaussian distribution, the temporal similarity between features in the attention layer output conforms to a Gaussian distribution on the time axis.

[0079] The above method ensures that the features of each timestamp have high similarity with its neighboring frames and low similarity with distant frames.

[0080] Specifically, a mapping network h(·) consisting of a fully connected layer, a ReLU activation layer, and another fully connected layer is used to map the output f″ of the self-attention sub-model. i Mapped to embedded features s i :

[0081] s i =h(f″) i )

[0082] The temporal similarity is represented as:

[0083]

[0084]

[0085] Where, μ ij Cosine similarity, denoted as μ ij =cos(s i ,s j ), where i,j represent different times, and G(·) represents a Gaussian distribution, denoted as x is the independent variable of a Gaussian distribution.

[0086] The inventors discovered that each type of action has its own unique appearance and movement characteristics. This invention also includes an auxiliary classification process:

[0087] The typical features of each action category are obtained. Using the features of the current frame as query conditions, the typical features of the corresponding category for the current frame are retrieved. These retrieved typical features are used as auxiliary information to assist in the action classification of the current frame. Dynamically learning the typical features of categories not only supplements information that is helpful for action classification but also enhances the generalization ability of the model.

[0088] The typical features of the category are obtained through a decoder, which preferably adopts the standard Transformer decoder structure. The decoder can output the feature vector of each action category, represented as follows:

[0089] Furthermore, the decoder is trained, that is, trained on... Then perform K-class classification using a fully connected layer and a softmax layer, minimizing the sum of the classification result and y. k The cross-entropy loss between the two elements allows the decoder to output typical features of the action category. Let y represent the k-th category feature. k For one-hot encoding, the value of the k-th position is 1, indicating the k-th type of action.

[0090] By inputting the features of the current frame into the trained decoder, the typical features of the category corresponding to the current frame can be obtained.

[0091] Furthermore, the features of the current frame are added to the end of the obtained class-specific typical features to form a new input sequence. This sequence is also input into the attention layer, which outputs an auxiliary feature sequence F. c , This represents the auxiliary feature of the k-th category. This represents the auxiliary feature corresponding to the current frame.

[0092] Furthermore, the attention vector f″0 of the current frame and auxiliary features are... The features f0 are concatenated into a frame block, then passed through a fully connected layer and a ReLU activation layer, and fed into a classifier consisting of a fully connected layer and a Softmax layer to achieve action recognition, as shown below:

[0093] p0=Softmax(W2(RELU(W1f0))),

[0094] Where W1 and W2 represent the parameters of the fully connected layer. This indicates the probability that the current frame block belongs to each action category, and the action category with the highest probability is taken as the detection result.

[0095] According to a preferred embodiment of the present invention, the loss function during the training of the detection model is:

[0096]

[0097]

[0098]

[0099] Where α and β are balance coefficients, Cross-entropy loss constraints for action classification Represents temporal similarity constraints. This represents the cross-entropy loss constraint for categorical features, where CE represents the cross-entropy loss.

[0100] Example

[0101] Example 1

[0102] Experiments were conducted on the THUMOS'14 dataset, which defines 20 actions, with an average of 15.8 actions per video segment and an average share of 71% for background information, where T is set to 63.

[0103] Furthermore, during the experiment, to facilitate comparison with other methods, video frame features were extracted using a TSN model pre-trained on ActivityNet v1.3 (denoted as -Anet) and video frame features were extracted using a TSN model pre-trained on Kinetics (denoted as -Kinetics).

[0104] The experiment includes the following steps:

[0105] Set a detection model, which is based on the Transformer model;

[0106] Extract features from a video stream of length T;

[0107] The features of the video stream are input into the detection model, and the action category of the last frame of the video stream is output by the detection model as the detection result.

[0108] The detection model has a self-attention sub-model, which includes multiple attention layers.

[0109] The input to the attention layer is the feature of the video stream or the attention vector output from the previous attention layer, and the output is the attention vector.

[0110] The attention layer only represents the correlation between the current frame and each historical frame.

[0111] In the attention layer, the cosine similarity between the key matrices of the current frame and the historical frames is used as the attention value.

[0112] The self-attention sub-model includes the following steps:

[0113] S11. Take the video stream features as input, process them using a convolutional layer to obtain a key matrix and a value matrix. The key matrix contains the key vectors of each frame, and the value matrix contains the value vectors of each frame.

[0114] S12. Use the cosine similarity between the current frame key vector and the historical frame key vector as the attention value output by the current attention layer.

[0115] S13. Calculate the weighted sum of the current frame value vector and the historical frame value vector based on the attention value, and output the attention vector;

[0116] S14. Use the attention vector output from the previous attention layer as the input to the next attention layer, and repeat steps S11 to S13 to obtain the final self-attention sub-model output.

[0117] The self-attention sub-model also includes a temporal convolutional layer, which is set between two adjacent self-attention layers.

[0118] In the self-attention sub-model, a Gaussian prior is also introduced. By minimizing the temporal similarity between the output features of each attention layer and the KL divergence between the prior Gaussian distribution, the temporal similarity between the features in the attention layer output conforms to a Gaussian distribution on the time axis.

[0119] The output of the self-attention sub-model is mapped to an embedded feature matrix, and the temporal similarity of the output features of each attention layer is characterized by the embedded features.

[0120] The mapping is implemented through a mapping network h(·) consisting of a fully connected layer, a ReLU activation layer, and another fully connected layer, and is represented as:

[0121] s i =h(f″) i )

[0122] Among them, s i f″ represents the embedding feature at time i. i This represents the output of the self-attention sub-model;

[0123] The temporal similarity of the output features of each attention layer is represented as follows:

[0124]

[0125]

[0126] The detection model also includes an auxiliary classification process:

[0127] Obtain the typical characteristics of each action category;

[0128] Use the features of the current frame as query conditions to obtain the typical features of the action category corresponding to the current frame;

[0129] The acquired typical features are used as auxiliary information to assist in the action classification of the current frame.

[0130] The action classification for assisting the current frame is performed in the following manner:

[0131] The features of the current frame are appended to the end of the obtained class canonical features to form a new input sequence. This sequence is then fed into a new attention layer, which outputs an auxiliary feature sequence F. c , Auxiliary features representing the k-th category feature;

[0132] Add the attention value f″0 and auxiliary features of the current frame The features are spliced ​​into a frame block feature f0, and the frame block feature f0 is processed by a classifier to output the recognized action.

[0133] Comparative Example 1

[0134] The same experiments as in Example 1 were conducted, using the RED, TRN, and IDN methods from the RNN modeling temporal context approach, respectively.

[0135] For the RED method, see the paper J. Gao, Z. Yang, and R. Nevatia. RED: Reinforced encoder-decoder networks for action anticipation. In BMVC, 2017.

[0136] For the TRN method, see the paper M. Xu, M. Gao, Y.-T. Chen, L.S. Davis, and D.J. Crandall. Temporal recurrent networks for online action detection. In Proc. IEEE International Conference on Computer Vision (ICCV), 2019, pp 5532–5541.

[0137] For the IDN method, see the paper Hyunjun Eun, Jinyoung Moon, Jongyoul Park, Chanho Jung, and Changick Kim. Learning to discriminate information for online action detection. In CVPR, pages 809–818, 2020.

[0138] Comparative Example 2

[0139] The same experiments as in Example 1 were conducted, using both the standard Tansformer and the additional storage Colar methods.

[0140] For the standard Tansformer method, see the paper Wang, Xiang and Zhang, Shiwei and Qing, Zhiwu and Shao, Yuanjie and Zuo, Zhengrong and Gao, Changxin and Sang, Nong. OadTR::Online Action Detection with Transformers. In ICCV, pages 7565–7575, 2021.

[0141] For additional storage of the Colar method, see the paper L. Yang, J. Han, D. Zhang. Colar: Effective and Efficient Online Action Detection by Consulting Exemplars, CVPR 2022.

[0142] Experimental Example

[0143] Comparing the results of Example 1 with those of Comparative Examples 1 and 2, the average precision (mAP) was used to evaluate the model's performance at the frame level, and the computational load of the model was represented by 1 billion floating-point operations per second (GFLOPs).

[0144] Table 1

[0145] method mAP(%) GFLOPs Comparative Example 1 (RED) 45.3 _ Comparative Example 1 (TRN) 47.2 1.46 Comparative Example 1 (IDN) 50.0 _ Comparative Example 2 (Standard Tansformer-Anet) 58.3 2.44 Comparative Example 2 (Additional Storage: Colar-Anet) 59.4 2.56 Example 1 - Anet 62.9 1.76 Comparative Example 2 (Standard Tansformer-Kinetics) 65.2 2.54 Comparative Example 2 (Additional Storage for Colar-Kinetics) 66.9 2.71 Example 1 - Kinetics 68.5 1.97

[0146] As shown in Table 1, the method in Example 1 generally improves the performance of online action detection. The model proposed in Example 1 achieves mAP of 61.2% and 68.0% on ActivityNet v1.3 pre-trained features and Kinetics pre-trained features, respectively, demonstrating better performance (at least 11.2% higher) compared to methods using RNNs to model temporal context (RED, TRN, and IDN). Even compared to the state-of-the-art Colar, which employs the standard Tansformer and additional storage, the method in Example 1 achieves 1.8% and 1.1% higher mAP using ActivityNet v1.3 and Kinetics pre-trained features, respectively.

[0147] Meanwhile, judging from the computational cost (GFLOPs) of each model shown in Table 1, the method in Example 1 requires less computation and performs better than other Transformer-based methods (standard Tansformer and Colar).

[0148] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," "front," and "rear," etc., indicate the orientation or positional relationship based on the orientation or positional relationship in the working state of this invention, and are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention. Furthermore, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0149] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0150] The present invention has been described above with reference to preferred embodiments; however, these embodiments are merely exemplary and illustrative. Various substitutions and modifications can be made to the present invention based on these embodiments, all of which fall within the scope of protection of the present invention.

Claims

1. A lightweight Transformer-based online action detection method, characterized in that, The method comprises the following steps: setting a detection model based on a Transformer model; obtaining features of a video stream with a length of T; inputting the features of the video stream into the detection model, and outputting an action category of a last frame of the video stream by the detection model as a detection result, the detection model has a self-attention sub-model, and the self-attention sub-model comprises a plurality of attention layers, the input of the attention layer is the features of the video stream or an attention vector output by a previous attention layer, and the output is the attention vector, the self-attention sub-model comprises the following steps: S11, taking the features of the video stream as input, processing the features by using a convolution layer to obtain a key matrix and a value matrix, the key matrix comprising key vectors of each frame, and the value matrix comprising value vectors of each frame; S12, taking a cosine similarity of the key vectors of a current frame and historical frames as an attention value output by a current attention layer; S13, weighting and summing the value vectors of the current frame and the historical frames according to the attention value to output an attention vector; S14, taking the attention vector output by a previous attention layer as input of a next attention layer, and repeating steps S11-S13 to obtain a final output of the self-attention sub-model, a Gaussian distribution prior is further introduced into the self-attention sub-model, and the time sequence similarity between the features output by each attention layer is made to conform to a Gaussian distribution on a time axis by minimizing the KL divergence between the time sequence similarity and the prior Gaussian distribution.

2. The online action detection method based on the lightweight Transformer according to claim 1, wherein the attention layer only represents the relevance of the current frame and the historical frames.

3. The online action detection method based on the lightweight Transformer according to claim 1, wherein in the attention layer, the cosine similarity of the key matrix of the current frame and the historical frames is taken as the attention value.

4. The online action detection method based on the lightweight Transformer according to claim 1, wherein the self-attention sub-model further comprises a time sequence convolution layer arranged between adjacent two self-attention layers.

5. The online action detection method based on the lightweight Transformer according to claim 1, wherein the output of the self-attention sub-model is mapped into an embedding feature matrix, and the time sequence similarity of the features output by each attention layer is represented by the embedding features, The mapping is realized by a mapping network h(·) consisting of a fully connected layer, a RELU activation layer, and another fully connected layer, denoted as h(f i "mapping to embedding features s i : s i = h(f i ") wherein, represents s i embedding feature at time i, f i represents the output of the self-attention submodel; the time sequence similarity of the features output by each attention layer is represented as: wherein μ ij is the cosine similarity, denoted as μ ij = cos(s ii , s j ), i,j denote different time instants, G(·) denotes a Gaussian distribution, denoted as x is the argument of the Gaussian distribution.

6. The online action detection method based on the lightweight Transformer according to claim 5, wherein an auxiliary classification process is further included in the detection model, typical features of each action category are obtained, the typical features of the action category corresponding to a current frame are obtained by taking the features of the current frame as a query condition, and the obtained typical features are taken as auxiliary information to assist the action classification of the current frame.

7. The online action detection method based on the lightweight Transformer according to claim 6, wherein the assistance of the action classification of the current frame is performed by the following manner: The new input sequence formed by adding the features of the current frame to the end of the obtained typical features of the category is input to a new attention layer, and an auxiliary feature sequence F is output by the attention layer C , wherein, Fk represents the auxiliary feature of the kth category feature, F represents the auxiliary feature corresponding to the current frame; The attention value f0" of the current frame and the auxiliary feature corresponding to the current frame Spliced into a frame block feature f0, the frame block feature f0 is output by the classifier to recognize the action.

Images