A motion recognition method and electronic device based on multi-task self-supervised learning

Abstract

The present invention discloses a method and electronic device for motion recognition based on multi-task self-supervised learning, including: using an encoder to encode motion data to be recognized; classifying the encoded data, and determining the category with the highest probability as the motion category of the motion data to be recognized; wherein the self-supervised tasks when the encoder is trained include: motion prediction self-supervised tasks, jigsaw puzzle self-supervised tasks, and contrast learning self-supervised tasks. The present invention designs self-supervised tasks and training strategies suitable for motion data, and improves the motion recognition accuracy of motion data through multiple self-supervised tasks.

Description

Technical Field

[0001] The present invention belongs to the field of action recognition, and specifically designs an action recognition method and an electronic device based on multi-task self-supervised learning. Background Art

[0002] The self-supervised task aims to extract data features through a large amount of unlabeled data, generate labeled data through corresponding self-supervised tasks, extract meaningful features through supervised methods, and assist in the learning of the target task.

[0003] Self-supervised tasks can be divided into three categories: (1) Pseudo-label based methods. Pseudo-labels are generated from data through various methods, and the generated labels are used to learn the classification task under supervision. For example, a picture is cut into blocks to sample a series of sub-pictures, two sub-pictures are randomly selected and their relative position relationship is predicted by the network. Or an image is rotated at different angles, and the angles are used as the generated classification labels; (2) Reconstruction based methods. This method masks part of the original data and uses the network to reconstruct the masked part of the data. The selected loss function is generally the mean square error, and a generative adversarial network can also be used for reconstruction; (3) Contrastive learning based methods. Contrastive learning means that after an original picture is generated into a new picture through a data augmentation method, the feature distance extracted from the same picture and its transformed picture is reduced, and the feature distance between different data is increased.

[0004] The existing self-supervision methods on videos mainly use the prediction method of shuffling the frame order in the spatiotemporal domain and reconstruction in the spatiotemporal domain for self-supervision. However, the current use of action data is limited to fine-grained extraction, and less attention is paid to long-term features that are more important for action recognition. At the same time, the current self-supervision tasks are relatively simple and cannot extract richer and more generalized features. Summary of the invention

[0005] In response to the above technical problems, the present invention proposes an action recognition method and electronic device based on multi-task self-supervised learning. Through the coordination of three self-supervised tasks, the global and local feature extraction of action data in time and space is enhanced, thereby improving the accuracy of sequential action recognition.

[0006] The technical solution adopted by the present invention is as follows:

[0007] An action recognition method based on multi-task self-supervised learning, the steps of which include:

[0008] 1) Use an encoder to encode the action data to be recognized;

[0009] 2) Classify the encoded data and determine the category with the highest probability as the action category of the action data to be identified;

[0010] Among them, the self-supervision tasks during encoder training include:

[0011] a) Action prediction self-supervision task: mask part of the action data of each sample, and predict the masked data through the unmasked data;

[0012] b) Jigsaw puzzle self-supervision task: divide each sample action data into several sub-segments in time and randomly shuffle them, and predict the shuffle order;

[0013] c) Contrastive learning self-supervised task: convert each sample action data into enhanced data respectively, reduce the feature distance between each sample action data and the corresponding enhanced data, and increase the feature distance between each sample action data.

[0014] Furthermore, the network structure of the encoder includes: one or more layers of bidirectional gated recurrent units.

[0015] Furthermore, the action prediction self-supervision task is completed through a reconstruction head composed of one or more layers of fully connected networks; the jigsaw puzzle self-supervision task is completed through a classification head composed of one or more layers of fully connected networks; and the contrastive learning self-supervision task is completed through a mapping head composed of one or more layers of fully connected networks.

[0016] Furthermore, the encoded data is classified by a classifier consisting of one or more layers of gated recurrent units and one or more layers of fully connected networks.

[0017] Furthermore, methods for training a convolutional neural network composed of an encoder, a reconstruction head, a classification head, a mapping head and a classifier include: a sliding training method and a joint training method.

[0018] Furthermore, the loss function L of the sliding training method is moving =θL cls +(1-θ)L self , where θ is a parameter that grows linearly from 0 to 1, L cls is the action recognition loss function of the classifier, and the total loss function of the self-supervised task L self =L m +L j +L c , L m is the loss function for the action prediction self-supervision task; L j is the loss function of the jigsaw puzzle self-supervision task; L c For contrast learning self-supervised task loss function. Further, the action recognition loss function of the classifier includes: cross entropy loss function.

[0019] Furthermore, the action prediction self-supervision task loss function includes: a mean square loss error function.

[0020] Furthermore, the loss function of the joint training in, is the weight parameter, L cls is the action recognition loss function of the classifier, and the total loss function of the self-supervised task L self =L m +L j +L c , L m is the loss function for the action prediction self-supervision task; L j is the loss function of the jigsaw puzzle self-supervision task; L c Loss function for contrastive learning self-supervised tasks.

[0021] Furthermore, each sample motion data is converted into enhanced data by covering the last few frames of data in the sample motion data and disrupting the sample motion data in segments.

[0022] A storage medium stores a computer program, wherein the computer program is configured to execute the method described above when running.

[0023] An electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer to execute the method described above.

[0024] Compared with the prior art, the present invention has the following advantages:

[0025] 1) Improve the accuracy of action recognition of action data through various self-supervision tasks;

[0026] 2) We designed self-supervised tasks and training strategies suitable for motion data. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a structural diagram of the action recognition network used in the present invention.

[0028] Figure 2 A diagram of the method of scrambling actions used in the present invention. DETAILED DESCRIPTION

[0029] In order to make the above features and advantages of the present invention more obvious and easy to understand, embodiments are given below and described in detail with reference to the accompanying drawings.

[0030] The action recognition method based on multi-task self-supervised learning of the present invention comprises a feature extractor and an action classifier, wherein the feature extractor is used to extract features, and the action classifier classifies actions based on the extracted features.

[0031] The feature extractor uses self-supervised training to learn. By masking the input data, shuffling the time sequence, and performing other processing, three self-supervised tasks are trained to enhance feature representation learning and extract more generalized feature information. The three self-supervised tasks are:

[0032] (1) Action prediction: The trajectory of subsequent actions is predicted based on the data of the previous frame of each action.

[0033] (2) Jigsaw puzzle task: The original data is divided into three segments in time. After the three sub-segments are randomly shuffled, the network is trained to predict the shuffled order.

[0034] (3) Contrastive learning: The original data is transformed to generate enhanced data, which reduces the feature distance between the original data and the enhanced data, while increasing the feature distance between different data.

[0035] After pre-training the feature extractor, the present invention designs two training strategies to optimize the action classifier to further improve the action recognition performance. The training strategies include:

[0036] (1) Sliding training. Sliding training adjusts the weight ratio between the auxiliary task and the target task through a dynamically adjusted parameter, so that the training can transition more smoothly from the auxiliary task to the target task.

[0037] (2) Joint training. Joint training requires training different tasks simultaneously, so that the network can complete multiple tasks at the same time, thereby sharing useful information to extract more generalized features.

[0038] The present invention is specifically described below by identifying the motion of a human skeleton, and the steps include:

[0039] Step 1: Collect data and obtain a skeleton-based action database.

[0040] Step 2: Build an action recognition network.

[0041] The network structure is as follows Figure 1 As shown, the model is divided into an encoder f and a reconstruction head h m , classification head h j , mapping header h c and classifier C. The encoder f consists of three bidirectional gated recurrent units (BiGRU); the reconstruction head h m , classification head h j and the mapping header h c Both are composed of three layers of fully connected networks (MLP); the classifier C is composed of three layers of gated recurrent units (GRU) followed by three layers of fully connected networks (MLP).

[0042] Step 3: Train the self-supervised assisted action recognition network.

[0043] The loss for the self-supervised task is:

[0044] L self =L m +L j +L c ,

[0045] Among them, L self is the total loss function of the self-supervised task. m , L j , L c , respectively adopting three self-supervised tasks, action prediction, jigsaw task, and contrastive learning corresponding loss functions.

[0046] When predicting action, let the original input data be X, and the data after masking several frames be X m Then the encoder f extracts the features and reconstructs the head h m The reconstructed data obtained is The present invention uses mean square error (MSE) to complete the reconstruction task:

[0047]

[0048] The MSE is the sum of the squares of the corresponding elements of the matrix.

[0049] like Figure 2 As shown in the figure, the jigsaw task is to divide the original data into three segments in time, randomly shuffle the three sub-segments, and use the network to predict the shuffle order. Let the input original data be X, then the shuffled data is X j , the label of the scrambling method is y, then:

[0050] L j =-ylog(h j (f(X j ))).

[0051] Contrastive learning is to generate enhanced data by transforming the original data, reducing the feature distance between the original data and the enhanced data, and increasing the feature distance between different data. There are two enhancement methods used here, namely covering the last few frames of data and shuffling the original data into segments. Two enhanced data can be obtained for each data.

[0052]

[0053] Where z is the feature extracted by the encoder f through the mapping head h c The mapping result. is the mean of the features of the same data and its enhanced data, N is the total number of data, and the sim function is the cosine similarity, which is defined as:

[0054]

[0055] The action recognition loss is the cross entropy loss function, y i If the data category is correct, it is 1, and if it is wrong, it is 0; i is the probability distribution of each category after the data passes through the encoder f and the classifier C; c is the number of action categories:

[0056]

[0057] The present invention can select two training strategies, namely a sliding training strategy or a joint training strategy, both of which can be used to train self-supervised task-assisted action recognition.

[0058] The loss function of sliding training is:

[0059] L moving =θL cls +(1-θ)L self ,

[0060] Here, θ is a parameter that increases linearly from 0 to 1.

[0061] The loss function for joint training is:

[0062]

[0063] in is the weight parameter.

[0064] Step 4: In the recognition stage, the skeleton data to be recognized is input. After passing through the encoder f and the classifier C, the probability distribution of each category is output. The category with the highest probability is selected as the predicted action category.

[0065] The present invention is applicable to different experimental settings and can achieve better results.

[0066] (1) Supervised learning: The dataset uses all skeleton data and corresponding action labels to test the accuracy of action recognition.

[0067] (2) Unsupervised learning: Only skeleton data is used to train the encoder f and the reconstruction head h m , classification head h j , mapping header h c . Use the features extracted by encoder f to perform action recognition and retrain classifier C to test the action recognition accuracy.

[0068] (3) Semi-supervised learning: All skeleton data and 10% labeled data are used during training to test the accuracy of action recognition.

[0069] (4) Transfer learning: Train the encoder f and reconstruction head h in PKUMMD part I and NTU respectively m , classification head h j , mapping header h c Afterwards, action recognition training was performed on the PKUMMD part II dataset and the action recognition accuracy was tested on the PKUMMD part II dataset.

[0070] Table 1, Table 2, and Table 3 are the action recognition accuracy rates of supervised learning on NW-UCLA, PKUMMD part I, PKUMMD part II, and NTU, respectively. Table 4 is the action recognition accuracy rate under unsupervised learning. Table 5 is the action recognition accuracy rate under semi-supervised learning. Table 6 is the action recognition accuracy rate under transfer learning.

[0071] Compared with other existing methods, the present invention has achieved a significant improvement in the accuracy of action recognition.

[0072] Table 1

[0073] Model NW-UCLA HBRNN-L (Hierarchical Bidirectional Recurrent Neural Network) 78.50 SK-CNN (Skeleton Convolutional Neural Network) 86.10 VA-LSTM (View Adaptive Long Short-Term Memory Neural Network) 70.71 Denoised-LSTM (Denoised Long Short-Term Memory Neural Network) 80.30 The present invention (sliding training) 85.32 The present invention (joint training) 86.75

[0074] Table 2

[0075]

[0076] Table 3

[0077]

[0078] Table 4

[0079]

[0080] Table 5

[0081]

[0082]

[0083] Table 6

[0084]

[0085] The above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the same. A person skilled in the art may modify or make equivalent substitutions for the technical solutions of the present invention without departing from the spirit and scope of the present invention. The protection scope of the present invention shall be subject to the claims.

Claims

1. An action recognition method based on multi-task self-supervised learning, the steps of which include: Build an action recognition network, which includes an encoder f and a reconstruction head h m , classification head h j , mapping header h c and a classifier C; wherein the encoder f is composed of three bidirectional gated recurrent units BiGRU; the reconstruction head h m , classification head h j and the mapping header h c They are all composed of a three-layer fully connected network MLP, and the classifier C is composed of a three-layer gated recurrent unit GRU followed by a three-layer fully connected network MLP; Train the self-supervised action recognition network, and the loss of the self-supervised task is: L self =L M +L j +L c , where L self is the total loss function of the self-supervised task, L M , L j , L c Three self-supervised tasks are used: action prediction, jigsaw puzzle task, and contrastive learning corresponding loss functions; In the recognition stage, the skeleton data to be recognized is input, and after passing through the encoder f and the classifier C, the probability distribution of each category is output. The category with the highest probability is selected as the predicted action category. Among them, the self-supervision tasks during training include: a) Action prediction self-supervision task: mask part of the action data of each sample, and predict the masked data through the unmasked data; b) Jigsaw puzzle self-supervision task: divide each sample action data into several sub-segments in time and randomly shuffle them, and predict the shuffle order; c) Contrastive learning self-supervised task: convert each sample action data into enhanced data respectively, reduce the feature distance between each sample action data and the corresponding enhanced data, and increase the feature distance between each sample action data.

2. The method according to claim 1, characterized in that Methods for training a convolutional neural network composed of an encoder, a reconstruction head, a classification head, a mapping head and a classifier include: a sliding training method and a joint training method.

3. The method according to claim 2, characterized in that Loss function L of the sliding training method moving =θL cls +(1-θ)L self , where θ is a parameter that grows linearly from 0 to 1, L cls Action recognition loss function for the classifier.

4. The method according to claim 3, characterized in that The action recognition loss function of the classifier includes: cross entropy loss function; the action prediction self-supervised task loss function includes: mean square loss error function.

5. The method according to claim 2, characterized in that Loss function for joint training in, is the weight parameter, L cls Action recognition loss function for the classifier.

6. The method according to claim 1, characterized in that Each sample action data is converted into enhanced data by covering the last few frames of data in the sample action data and shuffling the sample action data into segments.

7. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 6.

Images