An action recognition method based on fast-slow decomposition network

Abstract

The application discloses a kind of action recognition methods based on fast and slow decomposition network.Method includes: collecting the action video of several action categories, and action video and preprocessed action video are constructed as enhanced dataset;Fast and slow decomposition network is constructed;Input preprocessed action dataset is trained, and the network that training is completed is obtained;Collect the action video to be identified, input the network that training is completed, and the probability of action category is obtained after the output result is passed through a fully connected layer, classification is carried out, and the recognition of action is realized.The method of the application not only reduces the parameter quantity and calculation amount, but also improves the feature expression capability, can adaptively learn the correlation and importance of features, thereby enhancing the representation capability and generalization capability, avoiding overfitting low-level features while ensuring sufficient high-level features, so that the recognition of video action is more rapid and accurate.

Description

TECHNICAL FIELD

[0001] The application relates to a motion recognition method, and relates to the technical field of motion recognition based on deep learning. BACKGROUND

[0002] With the continuous development of deep learning technology, the application scenarios of motion recognition are increasing, and the image classification technology has been very mature, which creates favorable conditions for the development of motion recognition technology. Motion recognition technology is an artificial intelligence technology capable of recognizing human posture, gestures, body movements, etc. It is one of the important technologies for realizing natural interaction, human-computer collaboration and other artificial intelligence applications. Previous researches are mainly based on traditional machine learning algorithms such as support vector machine and hidden Markov model, but these methods often need to manually design features and are limited in accuracy, and cannot handle complex motions. In recent years, with the rise of deep learning technology, motion recognition technology based on deep learning has gradually become the mainstream.

[0003] At present, the motion recognition technology based on deep learning mainly includes recurrent neural network RNN, convolutional neural network CNN and self-attention mechanism. Recurrent neural network RNN is a deep learning model suitable for sequence data, which can capture the time sequence information in the sequence. In motion recognition, RNN can convert a video sequence into a fixed-length vector representation, and then use a fully connected layer for classification. In recent years, some improved RNN models such as long short-term memory network LSTM and gated recurrent unit GRU have also been widely applied to motion recognition tasks. Convolutional neural network CNN is a deep learning model suitable for image and video data processing, which can effectively extract local features. In motion recognition, CNN can use convolutional layers to extract local features in video sequences, and then use fully connected layers for classification. In order to improve the recognition accuracy, some improved CNN models such as Inception and ResNet have also been applied to motion recognition tasks. In addition to RNN and CNN, self-attention mechanism has also been widely applied to motion recognition tasks. Self-attention mechanism can weight different parts of input data, improving the accuracy and robustness of motion recognition. Self-attention mechanism has been widely applied in motion recognition.

[0004] The recurrent neural network RNN requires a large amount of calculation for processing a long sequence, which leads to slow speed and is not suitable for real-time applications. The model based on the convolutional neural network is a locally connected network, which has difficulty in long-distance information transmission. The convolutional layer of the CNN requires a large number of convolution kernels, which leads to a large number of model parameters and a long training and inference time. In addition, the model based on the self-attention mechanism needs to calculate the similarity between all input features, which leads to high computational complexity, easy overfitting to small-scale data sets, and limitation on the length of the input sequence. When the sequence length is long, it will lead to excessive memory consumption. In summary, both the convolutional neural network and the self-attention mechanism have some shortcomings, and need to be selected and improved according to the specific task and scene to achieve better performance and effect. SUMMARY

[0005] In order to solve the problems in the background art, in view of the defects that the convolution network operation obtained by the above action recognition method cannot capture global information, the model parameter quantity is large, and the self-attention mechanism has high computational complexity and is easy to overfit to small-scale data sets, etc., the application provides an action recognition method based on a slow-fast factorized dot-product attention (SFDA) network. Here, the SlowFast network is used to obtain spatial semantic information and action semantic information. The 3D group convolution is used to solve the problem of large parameter quantity of the slow path in SlowFast, and the FDA_Drop key mechanism is added after the two paths of SlowFast, which is used to calculate the importance of each element in the feature sequence, so that the model can better focus on the important part of the sequence, thereby improving the effect of sequence modeling. In addition, in order to solve the problems of easy overfitting to small-scale data sets and high computational complexity of the self-attention mechanism, the key is used as a random inactivation unit in the FDA_Drop key module in the application, which solves the problem well, and the random inactivation probability value adjustment strategy is used, in which the random inactivation probability value decreases with the increase of the depth of the FDA_Drop key module.

[0006] The technical scheme adopted by the application is:

[0007] The action recognition method based on the slow-fast factorized dot-product attention network of the application comprises:

[0008] Step 1) Collect action videos of several action categories labeled with action category labels, and construct an enhanced data set by combining each action video and each action video after preprocessing.

[0009] Step 2) Construct a slow-fast factorized dot-product attention network.

[0010] Step 3) input the pre-processed action data set into the fast-slow decomposition network for training to obtain the trained fast-slow decomposition network; specifically, the pre-processed action data set can be divided into a training set train and a validation set validation in a ratio of 7:3 for training and validation, respectively. After training is completed, the model needs to be evaluated using the validation set to calculate the accuracy of the model on the validation data set, until the accuracy of the fast-slow decomposition network reaches a stable state, and the training and validation are completed.

[0011] Step 4) collect the action video to be identified, input the action video to be identified into the trained fast-slow decomposition network, and the output result of the trained fast-slow decomposition network is input into a fully connected layer to obtain the probability of several action categories, and the classification of the action video to be identified is performed according to the probability of each action category to realize the recognition of the action. Specifically, the input fusion features are fused in the dimension dim=1 using the cat operation, then the fused features are subjected to a layer normalization operation, and finally sent to a linear layer (Linear) to output the category probability for action classification.

[0012] In step 1), each action video is pre-processed, specifically for each action video, a predetermined number of frames are extracted from the action video using sparse sampling; for each action video frame, data enhancement processing is performed on the action video frame, including one or several methods of size normalization, center cropping and random flipping processing, to finally obtain several frames of data enhanced action video frames; each frame of action video frame and each frame of data enhanced action video frame of each action video is collectively constructed into an enhanced data set.

[0013] In step 2), the constructed fast-slow decomposition network includes an improved fast-slow SlowFast model and two improved FDA_Drop key attention mechanisms, the input of the fast-slow decomposition network is processed by the improved fast-slow SlowFast model to output two results and input into two improved FDA_Drop key attention mechanisms, respectively, and the output of the two improved FDA_Drop key attention mechanisms is fused by cat to obtain a fusion result as the output of the fast-slow decomposition network.

[0014] The improved SlowFast model includes a fast path and an improved slow path, and the input of the improved SlowFast model is divided according to a preset ratio and input into the fast path and the improved slow path for processing and fusion. The two output results of the fast path and the improved slow path are taken as the output of the improved SlowFast model and input into two improved FDA_Drop key attention mechanisms respectively. Specifically, the training set is sent into the slow path and the fast path according to a ratio of 1:8; the features extracted through the slow path and the fast path are sent into the FDA_Drop key module again to obtain the correlation and importance between the features.

[0015] The fast path includes a fast path and a plurality of residual blocks with a Resnet network as a backbone network connected in sequence, and the slow path includes a slow path and a plurality of improved residual blocks with a Resnet network as a backbone network connected in sequence. The output result of the slow path is input into the plurality of improved residual blocks in sequence, the output result of the fast path is input into the first residual block, the output result of the nth improved residual block and the output result of the nth residual block are fused after feature fusion and taken as the input of the (n+1) th residual block, and the outputs after processing of the last residual block and the improved residual block are taken as the outputs of the fast path and the slow path respectively.

[0016] The slow path operating at a low rate is used to capture spatial semantics, and the convolution kernel of the first layer convolution is k=(1, 7, 7). Compared with the fast path, the slow path uses fewer convolution operations in the time dimension; in the fast path operating at a high rate, 3D convolution operation with a convolution kernel k=(5, 7, 7) is used to capture temporal semantics, which uses more convolution operations in the time dimension than the slow path to capture temporal information, and finally spatial semantic feature and motion information feature are output respectively. The number of residual blocks in the slow path and the improved residual blocks in the slow path is the same, which can be 4.

[0017] The improved residual block is specifically a residual block obtained by using group convolution for the second convolution operation Conv2 in the residual block. The group number is group=32; in the slow path, group convolution is used instead of ordinary convolution to reduce the parameter amount and the calculation amount. In the fast path, group convolution operation is not used, and group convolution is only used in the slow path.

[0018] The improved FDA_Drop key attention mechanism is specifically to take the key as a random inactivation unit of the FDA_Drop key attention mechanism, and to set the FDA_Drop key attention mechanism to a probability of random inactivation that decreases with the deepening of the layer.

[0019] The specific operation of the FDA_Drop key is to down-sample the input feature map through 3D convolution, and then convert it into a sequence (token) required in self-attention through a rearrangement operation; the feature maps of the two paths obtained through the residual block are respectively input into the FDA_Drop key, and the multi-head dot-product self-attention mechanism in the FDA_Drop key is divided into a spatial head and a temporal head, then different keys and values (keys and values in space and time, respectively) are used to realize parallel operation in the time and space dimensions; the self-attention is calculated separately, and the spatial head and the temporal head are calculated for the sequence, the spatial head calculates different sequences token in the same frame, and the temporal head calculates the sequence token in the same position of different frames.

[0020] In the FDA_Drop key operation, unlike directly randomly inactivating attention weights, the random inactivation operation is performed before calculating the attention matrix, and the key is taken as the basic random inactivation unit. The feature is that it can punish high attention areas and distribute attention weight values to other areas of interest, thereby enhancing the model's ability to capture global information. In the FDA_Drop key operation, a fixed random inactivation probability value is not used, but a random inactivation probability value that decreases with the increase of the depth of the FDA_Drop key is set.

[0021] In step 4), the classification of the to-be-identified action video is performed according to the probabilities of each action category, specifically, for each frame of the to-be-identified action video, the probabilities of each action category of the action video frame are sorted, and the action category with the highest probability is selected as the action category of the action video frame, thereby realizing the recognition of the action.

[0022] The improvement of the method is that the group convolution method is used in the slow path, which can not only reduce the calculation amount of the convolution layer and the parameter amount of the model, but also help to increase the nonlinear ability and feature expression ability of the model, then the features after each residual block in the fast path are fused into the slow path, finally the features of the fast path and the slow path are sent into the FDA_Drop key of the self-attention mechanism module again, in the module, the input features are decomposed into multiple independent parts, which can adaptively learn the correlation and importance of the features, thereby enhancing the representation ability and generalization ability of the model, in the FDA_Drop key, the multi-head attention is divided into two parts, one half is the spatial head Spatial Heads, and the other half is the temporal head Temporal Heads, the spatial head Spatial Head calculates different sequence tokens in the same frame, and the temporal head Temporal Head calculates the same position sequence tokens in different frames.

[0023] The beneficial effects of the present application are:

[0024] 1) The present application proposes to use group convolution operation in the slow path of SlowFast, which not only reduces the parameter amount and calculation amount, but also improves the feature expression ability.

[0025] 2) The FDA_Drop key self-attention mechanism module designed by the present application can adaptively learn the correlation and importance of the features, thereby enhancing the representation ability and generalization ability of the model.

[0026] 3) In addition, with the deepening of the FDA_Drop key layer, the probability of random inactivation is set to decrease, which can not only avoid overfitting of low-level features, but also ensure sufficient high-level features for stable training. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 The technical scheme flowchart of the method of the present application is;

[0028] Figure 2 The improved SlowFast network structure diagram of the present application is;

[0029] Figure 3 The structure diagram of the improved FDA_Drop key module of the present application is;

[0030] Figure 4 The improved SFDA network structure diagram of the present application is;

[0031] Figure 5 The action recognition accuracy curve diagram of the present application is; DETAILED DESCRIPTION

[0032] The application will be described in further detail below with reference to the drawings and specific embodiments.

[0033] As shown in the figure, the action recognition method based on the fast-slow decomposition network of the application comprises: Figure 1

[0034] Step 1) Collect action videos of several action categories labeled with action category labels, and jointly construct each action video and each action video after preprocessing as an enhanced data set.

[0035] In step 1), each action video is preprocessed, specifically, for each action video, the action video is extracted by a sparse sampling method to a preset frame number; for each action video frame, the action video frame is processed by data enhancement, the data enhancement includes processing by one or several methods of size normalization, center cropping and random flipping, and finally a plurality of frames of data enhanced action video frames are obtained; each frame of action video frame and each frame of data enhanced action video frame of each action video are jointly constructed as an enhanced data set.

[0036] Step 2) Construct a fast-slow decomposition network.

[0037] In step 2), the constructed fast-slow decomposition network includes an improved fast-slow SlowFast model and two improved FDA_Drop key attention mechanisms, the input of the fast-slow decomposition network is processed by the improved fast-slow SlowFast model to output two results and input into the two improved FDA_Drop key attention mechanisms, respectively, and the output of the processing of the two improved FDA_Drop key attention mechanisms is obtained after cat fusion as the output of the fast-slow decomposition network.

[0038] The improved fast-slow SlowFast model includes a fast path and an improved slow path, the input of the improved fast-slow SlowFast model is divided according to a preset ratio and input into the fast path and the improved slow path for processing and fusion, the two output results after processing of the fast path and the improved slow path are the output of the improved fast-slow SlowFast model and input into the two improved FDA_Drop key attention mechanisms, respectively. Specifically, the training set is sent into the slow path and the fast path according to a ratio of 1:8; the features extracted by the slow path and the fast path are sent into the FDA_Drop key module again to obtain the correlation and importance between the features.

[0039] ​The fast path comprises a plurality of residual blocks with Resnet network as a backbone network connected in sequence, and the slow path comprises a plurality of improved residual blocks with Resnet network as a backbone network connected in sequence, the output of the slow path is sequentially input into the plurality of improved residual blocks, the output of the fast path is input into the first residual block, the output of the nth improved residual block and the output of the nth residual block are fused as the input of the (n+1)th residual block, and the outputs of the last residual block and the improved residual block are respectively the outputs of the fast path and the slow path.

[0040] The slow path operating at a low rate is used to capture spatial semantics, and the convolution kernel of the first layer convolution is k=(1, 7, 7); compared with the fast path, the slow path uses fewer convolution operations in the time dimension; in the fast path operating at a high rate, 3D convolution operation with a convolution kernel k=(5, 7, 7) is used to capture time semantics, which uses more convolution operations in the time dimension than the slow path to capture time information, and finally outputs spatial semantic information features and motion information features. The number of residual blocks and improved residual blocks in the slow path and the slow path is the same, which can be 4.

[0041] The improved residual block is specifically a residual block obtained by using group convolution for the second convolution operation Conv2 in the residual block. The group number is group=32; in the slow path, group convolution is used instead of ordinary convolution to reduce the parameter amount and the calculation amount. In the fast path, group convolution operation is not used, and group convolution is only used in the slow path.

[0042] The improved FDA_Drop key attention mechanism is specifically that the key is used as a random inactivation unit of the FDA_Drop key attention mechanism, and the FDA_Drop key attention mechanism is set to a probability of random inactivation that decreases with the deepening of the layer.

[0043] The specific operation of the FDA_Drop key is to input the feature map to perform 3D convolution for down-sampling, and then convert it into a sequence (token) required in self-attention through a rearrangement operation; the feature maps of the two paths obtained through the residual block are input into the FDA_Drop key, and the multi-head dot-product self-attention mechanism in the FDA_Drop key is divided into a spatial head (Spatial Head) and a temporal head (Temporal Head), and then different keys and values (keys and values in space and time, respectively) are used to realize parallel operation in the time and space dimensions; the self-attention is calculated separately, and the spatial head Spatial Head and the temporal head Temporal Head are calculated for the sequence, respectively. Spatial Head calculates different sequences token in the same frame, and Temporal Head calculates the same position sequence token in different frames.

[0044] In the FDA_Drop key operation, unlike directly randomly inactivating the attention weight, the random inactivation operation is performed before the attention matrix is calculated, and the key is used as the basic random inactivation unit. The feature is that it can punish high attention areas and distribute attention weight values to other areas of interest, thereby enhancing the model's ability to capture global information. In the FDA_Drop key operation, a fixed random inactivation probability value is not used, but a random inactivation probability value that decreases with the increase of the depth of the FDA_Drop key is set.

[0045] Step 3) input the preprocessed action data set into the fast-slow decomposition network for training to obtain a trained fast-slow decomposition network; specifically, the preprocessed action data set can be divided into a training set train and a validation set validation in a ratio of 7:3 for training and validation, respectively. After training is completed, the model needs to be evaluated using the validation set to calculate the accuracy of the model on the validation data set, until the accuracy of the fast-slow decomposition network reaches a stable state, and the training and validation are completed.

[0046] Step 4) collect the action video to be identified, input the action video to be identified into the trained fast-slow decomposition network, and the output result of the trained fast-slow decomposition network is input into a fully connected layer to obtain the probability of a plurality of action categories, and the classification of the action video to be identified is performed according to the probability of each action category to realize the recognition of the action. Specifically, the input fusion features are fused in the dimension dim=1 using the cat operation, then the fused features are subjected to a layer normalization operation, and finally sent to a linear layer (Linear) to output the category probability for action classification.

[0047] In step 4), the classification of the action video to be recognized is performed according to the probabilities of the respective action categories, specifically, for each frame of the action video frame of the action video to be recognized, the probabilities of the respective action categories of the action video frame are sorted, and the action category with the highest probability is selected as the action category of the action video frame, so that the action recognition is realized.

[0048] The present application is mainly directed to video action recognition, and since the recognition object of the present application is a video action sequence, the HMDB-51 dataset is selected, which contains 51 human daily behavior actions, including walking, jogging, cycling, boxing, etc., and the video background is relatively complex, so it is more difficult to train.

[0049] As shown in the method of the present application, Figure 1 First, the videos of each category in the HMDB-51 dataset are read to obtain the category label, then the video data is preprocessed: the video frames of the specified parameters are extracted by sparse sampling, the video frames are randomly flipped, and the video is randomly cropped for data enhancement and sliding window sampling. Then, the pixel values are standardized to the range of [0, 1] for input into the deep learning model. After that, the continuous video frame sequence is used as the input of the improved SlowFast to capture spatial semantic information and action semantic information respectively, and fast and slow action information is obtained, then the obtained feature information is sent into the FDA (Factorised dot-product attention) key module to obtain the importance between elements, finally the features of the two paths are fused to obtain a more comprehensive video information representation, then the features are subjected to layer normalization (Layernorm), and finally sent into the fully connected layer to obtain the category probability of each action.

[0050] The specific steps of the present application are as follows:

[0051] The videos of each category are read to obtain the category label, then each video is preprocessed to obtain the number of frames of each video action, then the specified number of video frames is obtained by sparse sampling, and finally the preprocessed video frames are divided into a training set train and a validation set validation in a ratio of 7:3; as shown in Figure 2 The training set train is sent into the fast path and the slow path in a ratio of 8:1, wherein the fast path and the slow path are based on the Resnet skeleton.

[0052] The slow path is used to capture spatial and semantic information, low frame rate, large parameter quantity. The low frame rate is reflected in that the frame density of the slow path is 1 / α times that of the fast path; the large parameter quantity is reflected in that its channel capacity is relatively large compared with the fast path, that is, it needs to capture more spatial semantic information than the fast path, so the slow path is designed to not sample the time dimension at the lower layer and to sample the time dimension at the higher layer. Therefore, the convolution kernel of the first layer convolution of the slow path is k = 1 in the time dimension, 7 in the high h and wide w, followed by the normalization BatchNorm3d, the activation function ReLU and the maximum pooling layer MaxPooled. Next are four residual blocks resnet block of the slow path, each of which contains three 3D convolutions, in which the slow path samples in the spatial dimension, and in the first two residual blocks, the convolution kernels of the three 3D convolutions are k = (1, 1, 1), k = (1, 3, 3), k = (1, 1, 1) respectively, and in the last two residual blocks, the convolution kernels of the three 3D convolutions are k = (3, 1, 1), k = (1, 3, 3), k = (1, 1, 1) respectively. In order to solve the problem of large parameter quantity in the slow path, the second convolution operation in each residual block in the slow path uses grouped convolution, and the group number is set to groups = 32 from the default groups = 1.

[0053] The fast path is used to capture the rapid change of action, high frame rate, small parameter quantity, and the goal is to have a good representation in the time dimension. The high frame rate is reflected in that it samples αT frames, where α > 1 is the frame rate ratio between the fast and slow paths, and T is the number of frames sampled by the slow path, that is, the frame density of the fast path is α times that of the slow path; the small parameter quantity is reflected in the low channel capacity, and the low channel capacity can also be explained as weak spatial semantic expression ability, and the fast path does not have special processing in the spatial dimension, so its spatial modeling ability should be lower than that of the slow path due to fewer channels. The fast path needs to weaken its spatial modeling ability while enhancing its time modeling ability. First, the first layer needs to obtain more time sequence information, so the convolution kernel of the first 3D convolution is k = 5 in the time dimension, and k is 7 in the h, w dimensions, followed by the batch normalization BatchNorm3d, the activation function ReLU and the maximum pooling layer MaxPooled. Next are four residual blocks resnet block of the fast path, each of which contains three 3D convolutions, in which the fast path samples in the time dimension, that is, the convolution kernels of the three 3D convolutions in each residual block are k = (3, 1, 1), k = (1, 3, 3), k = (1, 1, 1) respectively.

[0054] For the feature fusion module, the information of the fast path is fused into the slow path. Since the two path image sequences are inconsistent in time, the fast path needs to be down-sampled in the time dimension, that is, a convolution operation with a convolution kernel of k=(7, 1, 1) and a stride of (4, 1, 1) is adopted, and then the features are fused from the fast path to the slow path after each residual block, and horizontal connection is performed, as shown in Figure 2

[0055] As Figure 3 ​The structure diagram of the FDA_Drop key module is shown, and the method performs a random inactivation operation before calculating the attention matrix, and takes the key as the basic random inactivation unit, thereby avoiding that in the previous self-attention mechanism, the random inactivation is usually set after the softmax activation function, but the random inactivation after the normalization of the softmax activation function breaks the probability distribution of the attention weight and cannot punish the weight peak, thereby causing the model to still overfit to local specific information. The method theoretically verifies that the regularizer Drop key can punish high attention areas and distribute attention weight values to other areas of interest, thereby enhancing the model's ability to capture global information. Secondly, setting the random inactivation probability value gradually decreasing with the deepening of the FDA_Drop key structure can avoid the case that a larger random inactivation probability in the deep layer of the network will cause the loss of high-level semantic information, and a smaller random inactivation probability in the shallow layer will cause overfitting to the bottom-level detail features, so a constant random inactivation probability will cause unstable conditions in the training process. In this module, an input tensor with a size of (batch_size, num_tokens, embedding_dim) is accepted, and an output tensor with the same size as the input tensor is returned. Among them, batch_size is the batch size, num_tokens is the length of the input tensor, and embedding_dim is the embedding dimension of each sequence token. In the module, first, a linear layer is used to map the input tensor to a tensor with a size of (batch_size, num_heads, num_tokens, embedding_dim_per_head), where num_heads is the number of heads, and embedding_dim_per_head is the embedding dimension of each head. Then the mapped tensor is divided into query, key and value three tensors, then the dot product of the query and the key is performed, and then the softmax operation is performed, the scaled dot product attention is obtained, the attention matrix is obtained, and the final output tensor is obtained by weighting and summing the attention matrix and the value. Unlike the standard self-attention layer, the module divides the attention operation into two parts: spatial attention and temporal attention. Spatial attention is performed in the spatial dimension of the input, and temporal attention is performed in the time dimension of the input. Then the spatial attention Y s and the temporal attention Y t are fused, and finally fed into a fully connected layer, which includes a linear layer and a random inactivation layer. The linear layer maps the input tensor to a tensor with a size of (batch_size, num_tokens, embedding_dim), and the random inactivation layer randomly discards the output of some neurons, thereby forcing the model to learn more robust and generalizable features, thereby avoiding overfitting.

[0056] As Figure 4 shown, the improved SlowFast features are sent to the FDA_Drop key, then the feature fusion operation is performed, and then the features are sent to the normalization LayerNorm layer for normalization processing of the feature vector. Finally, the features are sent to a linear layer for mapping the feature vector to the dimension of the number of categories for the final classification prediction. Finally, the SFDA network action classification result is obtained. As Figure 5 shown, the accuracy diagram of the fast and slow decomposition network trained by the method of the present application is shown. The accuracy of action recognition of the method of the present application and the accuracy of other methods are compared as shown in Table 1. It can be seen that the action recognition accuracy of the method of the present application is significantly improved.

[0057] Table 1 Comparison of results of the present application and other models

[0058]

[0059]

[0060] The present application proposes to fuse CNN and self-attention mechanism, and adds a group convolution operation in the slow path of SlowFast, so as to reduce the parameter quantity and reduce the calculation amount. Then the obtained features are sent to the FDA_Drop key designed by us again to obtain the importance between elements. In the FDA_Drop key, we add the strategy that the random inactivation probability decreases with the increase of depth, which can avoid the model overfitting to low-level features and at the same time ensure that there are sufficient high-level features for stable training.

[0061] The above specific technical process is used to explain and illustrate the present application, but not to limit the present application. Any modification and change made to the present application within the spirit and protection scope of the claims of the present application all fall within the protection scope of the present application.

Claims

1. An action recognition method based on a fast-slow decomposition network, characterized in that: include: Step 1) Collect action videos of several action categories marked with action category labels, and construct each action video and each action video after preprocessing into an enhanced dataset; Step 2) Construct a fast and slow decomposition network; Step 2.1) Use grouped convolution in the slow path to reduce the computational effort of the convolutional layer and the number of model parameters, and enhance the model's nonlinearity and feature expression capabilities. Step 2.2) After each residual block in the fast path is processed, its output features are fused into the slow path, thereby achieving feature interaction between the fast and slow paths; Step 2.3) The features extracted by the fast and slow paths are fed into the self-attention mechanism module FDA_Drop. In the FDA_Drop module, the input features are decomposed into multiple independent parts to adaptively learn the relevance and importance of features, thereby enhancing the model's representation and generalization capabilities. In the FDA_Drop module, the multi-head attention is divided into two categories: spatial heads, which are used to calculate different sequence tokens within the same frame; and temporal heads, which are used to calculate sequence tokens at the same position in different frames. Step 3) Input the preprocessed action data set into the fast-slow decomposition network for training to obtain a trained fast-slow decomposition network; Step 4) Collect the action video to be recognized and input it into the trained fast-slow decomposition network. The output of the trained fast-slow decomposition network is then passed through a fully connected layer to obtain the probabilities of several action categories. The action video to be recognized is classified according to the probabilities of each action category to achieve action recognition.

2. The method for motion recognition based on a fast-slow decomposition network according to claim 1, characterized in that: In the step 1), each action video is preprocessed. Specifically, for each action video, a preset number of frames are extracted from the action video using a sparse sampling method; for each action video frame, data enhancement processing is performed on the action video frame, and the data enhancement processing includes processing using one or more methods selected from size normalization, center cropping, and random flipping, and finally obtaining several frames of data-enhanced action video frames; each action video frame and each data-enhanced action video frame of each action video are jointly constructed into an enhanced data set.

3. The method for motion recognition based on a fast-slow decomposition network according to claim 1, characterized in that: In the step 2), the constructed fast-slow decomposition network includes an improved fast-slow SlowFast model and two improved FDA_Drop key attention mechanisms. The input of the fast-slow decomposition network is processed by the improved fast-slow SlowFast model and outputs two results which are respectively input into the two improved FDA_Drop key attention mechanisms. The outputs processed by the two improved FDA_Drop key attention mechanisms are fused by cat to obtain a fusion result as the output of the fast-slow decomposition network.

4. The method for motion recognition based on a fast-slow decomposition network according to claim 3, characterized in that: The improved SlowFast model includes a fast path fast path and an improved slow path slow path. The input of the improved SlowFast model is divided according to a preset ratio and input into the fast path fast path and the improved slow path slow path for processing and fusion. The two output results after processing of the fast path fast path and the improved slow path slow path are used as the output of the improved SlowFast model and are respectively input into the two improved FDA_Drop key attention mechanisms.

5. The method for motion recognition based on a fast-slow decomposition network according to claim 4, characterized in that: The fast path includes a fast path and several residual blocks connected in sequence, and the slow path includes a slow path and several improved residual blocks connected in sequence. The output results of the slow path are input into several improved residual blocks in sequence, and the output result of the fast path is input into the first residual block. The output result of the nth improved residual block and the output result of the nth residual block are feature fused as the input of the n+1th residual block, and finally the output after processing by the last residual block and the improved residual block are used as the output of the fast path and the slow path, respectively.

6. The method for motion recognition based on a fast-slow decomposition network according to claim 5, characterized in that: The improved residual block is specifically a residual block obtained by performing grouped convolution on the second convolution operation Conv2 in the residual block.

7. The method for motion recognition based on a fast-slow decomposition network according to claim 3, characterized in that: The improved FDA_Drop key attention mechanism specifically uses the key Key as the random deactivation unit of the FDA_Drop key attention mechanism, and sets the FDA_Drop key attention mechanism so that the probability of random deactivation decreases as the number of layers increases.

8. The method for motion recognition based on a fast-slow decomposition network according to claim 1, characterized in that: In the step 4), the action video to be identified is classified according to the probability of each action category. Specifically, for each action video frame of the action video to be identified, the probabilities of each action category of the action video frame are sorted, and the action category with the highest probability is selected as the action category of the action video frame to achieve action recognition.

Images