Self-supervised three-dimensional action prediction method based on past completion and future trends

Abstract

The application relates to a self-supervised three-dimensional action prediction method based on past completion degree and future trend, belonging to the field of computer vision. Firstly, an action completion degree perception task and a motion prediction task are proposed to supervise the model to learn a feature representation containing action state information and trend information. A multi-task self-supervised learning framework is designed, a contrast learning auxiliary task is introduced to extract high-order semantic features about actions, and under the premise of not needing manual labels, a plurality of self-supervised task losses are fused to guide the network to learn discriminative action features. Finally, a complete action sequence generation module is proposed on the basis, the action completion degree perceiver and the motion predictor trained by self-supervision are fully utilized to generate a corresponding complete sequence from an incomplete sequence, missing information in the sequence is complemented, and finally prediction is obtained by fusing partial prediction and global prediction, so that the accuracy of action prediction is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision, mainly aiming at a three-dimensional action prediction method, and particularly relates to a self-supervised three-dimensional action prediction method based on past completion and future trend. BACKGROUND

[0002] Skeleton-based three-dimensional action prediction is one of the important research topics in the field of intelligent behavior understanding, which can predict the category of action before the action is completely performed. With the popularity of depth sensors and the development of pose estimation algorithms, it plays an increasingly important role in intelligent security and other application scenarios. In the big data era, there are vast amounts of video data on the Internet, and the corresponding skeleton data can be easily obtained by applying pose estimation algorithms. How to train a deep neural network for action prediction using a large number of unprocessed skeleton sequences without manual annotation is an urgent research problem in the field of three-dimensional vision.

[0003] Three-dimensional action prediction can predict the action category during the execution of the action, and has wide practical value, but also faces the following problems: 1) the same action has different feature integrity under different observation rates; 2) the same action has different appearance under different viewing angles; 3) the same category of action has differences in duration and motion rate when performed by different subjects; 4) different categories of actions may have similarities in the early stage of action execution; 5) due to the incompleteness of the action sequence, some discriminative information is often missing; 6) different actions may have similar appearance due to the change of viewing angle.

[0004] At present, the research on three-dimensional action prediction has just started, and few studies have considered these difficulties, and all of them are concentrated in the supervised learning framework, which requires a large number of manual labels. The existing supervised three-dimensional action prediction methods all use various strategies during training, such as using soft labels or observation rate-related losses to prevent fitting, using regularization or adversarial learning to force the network to learn the implicit global information, and storing difficult-to-distinguish instances to enable the network to mine subtle discriminative information, but all of them follow the action recognition idea in the design of network structure, i.e. the encoder-classifier structure: first extract the features of part of the sequence, and then send it to the classifier to get the prediction result. These methods ignore the past and future information about the action contained in the incomplete sequence, and such information has been proven by psychological research to be the key to human action prediction. In fact, when humans watch a certain action sequence, they observe for a certain time, i.e. reach a certain observation rate, and already know what the target has done and what the target will do in the future. This information about the past completion and the future trend can help humans accurately predict the action category. SUMMARY

[0005] In order to solve the above problems existing in the prior art, the present application provides a self-supervised three-dimensional motion prediction method based on past completion degree and future trend, which does not require manual labeling, and based on the past completion degree and future trend information contained in the incomplete motion sequence, designs a motion completion degree perception and motion prediction proxy task, extracts the past and future information in the incomplete skeleton sequence, introduces state and trend information into the feature representation of the skeleton, and designs a contrast learning auxiliary task to introduce high-order semantic information about the motion into the feature representation of the skeleton, and enhances the representation ability of the model to the features. In addition, the present application designs a three-dimensional motion prediction network based on a complete motion sequence generation module, generates a skeleton sequence motion trajectory through an encoder-decoder network structure, extracts features from the partial sequence and the generated sequence respectively to make a prediction, and fuses the prediction results of the two to obtain the final prediction.

[0006] The present application comprises the following steps:

[0007] 1) Preprocess the three-dimensional skeleton sequence dataset to generate a three-dimensional motion prediction dataset;

[0008] 2) Design a motion completion degree perception task and a motion prediction task to supervise the model to extract past completion degree information and future trend information in the incomplete motion sequence;

[0009] 3) Design a contrast learning auxiliary task to supervise the model to extract high-order semantic information in the incomplete motion sequence, optimize the feature space, and design a multi-task self-supervised learning framework to further enhance the robustness of the model;

[0010] 4) Design a three-dimensional motion prediction network based on a complete sequence generation module to generate a complete sequence without introducing additional parameters, complete the missing information, and fuse the partial prediction and global prediction to obtain the final prediction;

[0011] 5) Calculate each loss in the multi-task self-supervised learning framework on the three-dimensional motion prediction dataset, train the network through the back propagation algorithm and the stochastic gradient descent method, and obtain the finally trained model;

[0012] 6) Test the recognition effect of three-dimensional motion prediction by using the trained model, the model outputs the prediction value of each action category, and the output value is normalized to obtain the final action prediction result.

[0013] In step 1), the specific steps of preprocessing the three-dimensional skeleton sequence dataset and generating the three-dimensional motion prediction dataset can be:

[0014] 1.1 First, normalize the joint coordinates on a large-scale skeleton sequence data, and apply a downsampling technique to unify the number of sample frames;

[0015] 1.2 Define nine different observation rates from 0.1 to 0.9, traverse all samples in the data set, generate a sequence for each sample under nine different observation rates, record the observation rate corresponding to each sequence, unify the length of all sequences, and fill in the blank frames with the last frame.

[0016] In step 2), the specific steps of the design action to complete the action completion perception task and the motion prediction task can be:

[0017] 2.1 Build a feature encoder based on a two-layer bidirectional gated recurrent unit network, input the incomplete action sequence into the encoder, extract features, and then input them into different downstream task heads;

[0018] 2.2 Build an action completion perception head based on a multilayer perception mechanism and a motion prediction head based on a recurrent neural network;

[0019] 2.3 Input the extracted features into the action completion perception head to obtain the predicted observation rate, measure the loss between the predicted observation rate and the true observation rate using the mean square loss function, and input the extracted features into the motion prediction head to obtain the predicted human joint trajectory, measure the loss between the predicted human joint trajectory and the true skeleton sequence using the mean square loss function.

[0020] In step 3), the design of the multi-task self-supervised learning framework is based on the contrastive learning auxiliary task, which supervises the model to extract high-order semantic information from incomplete action sequences, optimizes the feature space, and the specific steps of the multi-task self-supervised learning framework are as follows:

[0021] 3.1 For incomplete action sequence samples, input their corresponding complete action sequences into the feature encoder to obtain feature representations;

[0022] 3.2 Build a contrastive learning task head based on a multilayer perception mechanism, respectively project the features of the incomplete action sequence and the features of the corresponding complete action sequence into the contrastive learning feature space, and calculate the distance between the two feature vectors in terms of direction and value to obtain the contrastive learning loss;

[0023] 3.3 Build a multi-task self-supervised learning framework, weight the sum of the action completion perception loss, the motion prediction loss, and the contrastive learning loss to obtain the total loss, and supervise the weight training of the feature encoder and the three downstream task heads.

[0024] In step 4), design a complete sequence generation module and fuse partial and global prediction based on it, the specific steps of the three-dimensional action prediction network based on the complete sequence generation module are as follows:

[0025] 4.1 The action completion perception head and the motion prediction head are fused to obtain a complete sequence generation module, that is, an incomplete action sequence is input, and a corresponding complete action sequence is output;

[0026] 4.2 The trained feature extractor is applied to extract the features of the incomplete action sequence and the generated action sequence respectively, and make a prediction, and the final prediction result is obtained by fusing two predictions.

[0027] Compared with the prior art, the present application has the following outstanding technical effects:

[0028] Firstly, the action completion perception task and the motion prediction task are proposed to supervise the model to learn the feature representation containing the action state information and the trend information. And a multi-task self-supervised learning framework is designed, a contrast learning auxiliary task is introduced to extract high-order semantic features about the action, and under the premise of not needing manual labels, the network is guided to learn discriminative action features by fusing multiple self-supervised task losses. Finally, a complete action sequence generation module is proposed on this basis, which fully utilizes the self-supervised trained action completion perception and motion predictor to generate the corresponding complete sequence from the incomplete sequence, complete the missing information in the sequence, fuse the partial prediction and the global prediction to obtain the final prediction, and effectively improve the accuracy of action prediction. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 A three-dimensional action prediction sample diagram is constructed.

[0030] Figure 2 A framework diagram of the embodiment of the present application.

[0031] Figure 3 A three-dimensional action prediction network diagram based on the complete sequence generation module.

[0032] Figure 4 A complete sequence generation module diagram. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical scheme and advantages of the present application clearer, the following embodiments will further illustrate the present application with reference to the drawings. It should be understood that the specific embodiments described herein are only used to explain the present application, and are not used to limit the present application. On the contrary, the present application covers any substitution, modification, equivalent method and scheme defined by the claims within the essence and scope of the present application. Further, in order to make the public better understand the present application, some specific details are described in detail in the following detailed description of the present application. The present application can also be completely understood without the description of these details by those skilled in the art.

[0034] 1) Preprocess the three-dimensional skeleton sequence dataset, and generate a three-dimensional action prediction dataset. For example,Figure 1 As shown, given a skeleton sequence X = {x1, x2, x3, ..., x6} with a total of T frames, T-1 ,x T}, where x t Let represent the skeleton coordinates of frame t. Assuming an observation rate of O (O ∈ (0,1)), the input for 3D motion prediction is a partial sequence. That is, only the beginning of the complete action sequence The frames are known. 3D motion prediction aims to predict the complete motion based on the known incomplete motion.

[0035] The observation rate is defined as having a range of values ​​0 ∈ {0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9}. For each sample in the 3D skeleton sequence dataset, it is first normalized, then uniformly downsampled to the same number of frames, and then truncated at each of the nine observation rates. The frames obtained are incomplete sequences, and blank frames are filled with the last frame to ensure a consistent number of sample frames for different observation rates. By traversing the 3D skeleton sequence dataset, a 3D action prediction dataset is obtained, which has nine times the sample size of the 3D skeleton sequence dataset.

[0036] 2) Based on the past completion rate and future trend information contained in incomplete action sequences, design action completion rate perception tasks and motion prediction tasks, and supervise the model to extract state and trend information from the incomplete action sequences. See also Figure 2 The motion completion perception and motion prediction components in the program.

[0037] 2.1 As Figure 2 As shown in Figures (a) and (b), a feature encoder is constructed based on a two-layer bidirectional gated recurrent unit network. The incomplete action sequence is used as input to the encoder for feature extraction, and the calculation formula is as follows:

[0038] F = E(X) O )

[0039] After obtaining the feature representation F, it is sent to different downstream task headers.

[0040] 2.2 A motion completion perception head is constructed based on a multilayer perceptron, and a motion prediction head is constructed based on a recurrent neural network. For example... Figure 2 As shown in Figure (c), H O For action completion sensor, H M For motion predictors.

[0041] A two-layer multilayer perceptron is used as the action completion sensor H. O, the last layer of the multi-layer perceptron converts the feature dimension to 1 dimension and uses a sigmoid function as the activation function to obtain the predicted observation rate, and the calculation process is as follows:

[0042] O P = sigmoid(H O (F))

[0043] The structure of the motion predictor H M is a GRU network and a fully connected layer, and a residual design is adopted. The incomplete action sequence (the unobserved part is masked) is input into the GRU, and then a fully connected layer is used to convert the dimension thereof, so as to obtain the complete action sequence and add the original sequence to obtain the generated complete action sequence X P , and the network structure can be represented as:

[0044] X P = H M (F) = FC(GRU(XO)) + X O

[0045] 2.3. The loss of the action completion awareness task and the motion prediction task is calculated respectively, so as to supervise the feature extractor to learn the past completion degree and future trend information in the skeleton sequence.

[0046] As shown in Fig. (c) in Figure 2 , according to the observation rate O P regressed by the network and the real observation rate label O, the loss of the action completion awareness task is calculated as follows:

[0047]

[0048] Where N is the number of samples in a batch. Through the back propagation of the loss of the self-supervised task, the weights of the shared feature encoder E and the action completion awareness H O are trained.

[0049] According to the complete action sequence X P generated by the network and the real complete sequence X, the loss of the motion prediction task is calculated as follows:

[0050]

[0051] Where N is the size of the batch. Through the back propagation of the loss of the self-supervised task, the weights of the shared feature encoder E and the motion predictor H M are trained.

[0052] 3) A multi-task self-supervised learning framework is designed to integrate the action completion awareness task, the motion prediction task and the contrast learning auxiliary task, so as to further enhance the robustness of the model. The overall structure of the framework is shown in Figure 2 .

[0053] 3.1 For incomplete action sequence samples, their corresponding complete action sequence input feature encoders are obtained, and the original complete feature representation is calculated as follows:

[0054] F original = E(X)

[0055] The feature F is compared with the feature F extracted from the incomplete sequence, which contains more rich and complete information.

[0056] 3.2 As shown in Fig. (c) in the Figure 2 , a two-layer multilayer perceptron is used as a contrast learning mapper H R , respectively, to map F and F original , expecting them to have similar feature distributions in the contrast learning feature space, so that the model has robustness to the change of observation rate. The calculation process of the mapping is as follows:

[0057]

[0058] Where F observed is the mapping of the partial sequence feature in the contrast learning feature space, and F full is the mapping of the complete sequence feature in the contrast learning feature space.

[0059] In order to comprehensively measure the similarity of the features, and consider the distance of the features in direction and value, the loss of the contrast learning auxiliary task is as follows:

[0060]

[0061] Where N is the size of the batch, and the first term is the cosine loss, whose calculation formula is as follows:

[0062]

[0063] Where x1 and x2 are two feature vectors, and ∈ is a constant. This function can measure the difference between the partial feature and the complete feature in direction. The second and third terms respectively supervise the mean and variance of the partial feature and the complete feature, where mean(·) is the mean of the feature vector, and var(·) is the variance of the feature vector. These two losses can force the partial feature and the complete feature to be close in value. Through the back propagation of the contrast learning task loss, the shared feature encoder E and the contrast learning mapper H RThe weights are assigned to learn a feature space where partial sequences and their corresponding complete sequences have similar feature representations. For samples belonging to the same action category, even if the action completion rate (observation rate) is different, they still have common high-order features about action information. The feature encoder trained by the contrastive learning task will have a stronger ability to express this abstract information.

[0064] 3.3 Constructing a multi-task self-supervised learning framework. For example... Figure 2 As shown, the perceived loss of action completion L O Motion prediction loss L M And contrastive learning loss L R Simultaneously, backpropagation is performed to supervise the training of the feature encoder E, and the weight training of the three downstream task heads is supervised separately. The total loss for multi-task learning is as follows:

[0065] L=αL O +βL M +γL R

[0066] Where α, β and γ are the weights of each loss, and are all set to 1 in this invention.

[0067] 4) Design a 3D action prediction network based on a complete sequence generation module. The overall structure of the network is as follows: Figure 3 As shown in Figure (b), compared to the traditional structure that makes predictions based solely on incomplete sequence features... Figure 3 Figure (a) shows the addition of a complete sequence generation module, which integrates predictions based on partial sequences and predictions based on complete sequences to obtain the final prediction result.

[0068] 4.1 Action Completion Perceptor H Based on Multi-Task Self-Supervised Learning Framework O and motion predictor H M Construct a complete sequence generation module, the specific structure of which is as follows: Figure 4 As shown. In actual action prediction, the network input only contains an incomplete action sequence, and the action completion rate (O) of this sequence is unknown. Because the observation rate is unknown, the length L of the complete action sequence is unknown. full It is also unknown. First, the incomplete sequence X... O The input feature encoder obtains partial sequence features F = E(X). O Then, the observation rate of the sequence is predicted using an action completion perceptron, calculated as follows:

[0069] O P =sigmoid(H O (F))

[0070] The length L of the complete action can be obtained by combining the predicted observation rate and the number of frames of the observed sequence full , and the calculation process is as follows:

[0071]

[0072] wherein is a floor operation. The feature F of the partial sequence is sent to the motion predictor to generate the human motion trend in a certain time range in the future, and the first L full frames of the future motion trajectory can generate the complete skeleton sequence X P of the incomplete action sequence.

[0073] 4.2 A single-layer fully connected layer is used as the action predictor P. The generated complete sequence X P is sent to the feature encoder E to obtain the global feature F global contained in the complete sequence = E(X P ). The partial feature F and the global feature F global are sent to the action predictor P, respectively, to obtain the action prediction predict based on the partial sequence and the action prediction predict global based on the generated complete sequence, respectively. The network structure is as follows:

[0074]

[0075] The prediction results based on the partial sequence and the prediction results based on the generated complete sequence are fused to obtain the final prediction result. The overall structure of the network is as follows:

[0076]

[0077] wherein P converts the dimension size of the feature to the number of action categories to obtain the prediction result.

[0078] 5) On the three-dimensional action prediction data set, each loss in the multi-task self-supervised learning framework is calculated, and the network is trained by the back propagation algorithm and the stochastic gradient descent method to obtain the finally trained model. The back propagation process is shown in Figure 2 , wherein the action completion awareness loss L O provides a supervision signal for the feature encoder E and the action completion awareness H O , the motion prediction loss L M provides a supervision signal for the feature encoder E and the motion predictor H M , and the contrast learning loss L R provides a supervision signal for the feature encoder E and the contrast learning mapper H R .

[0079] 6) Test the effect of three-dimensional action prediction using the trained model, the model outputs the prediction value of each action category, and the output value is normalized to obtain the final action prediction result.

[0080] On the NTU RGB+D and SYSU 3D HOI three-dimensional action prediction data sets, the method proposed in the application trains the model to learn the feature representation without manual annotation, and the experimental results are shown in Tables 1 and 2.

[0081] Table 1

[0082]

[0083] Table 2

[0084]

[0085] The application designs a self-supervised three-dimensional action prediction method based on past completion degree and future trend, which can guide the learning of discriminative features through a multi-task self-supervised learning framework, and introduce past completion degree information and future trend information about actions into feature representation. The application preprocesses and down-samples a three-dimensional action dataset, and generates samples under different observation rates. Based on the past completion degree and future trend information contained in the incomplete action sequence, the application designs an action completion degree perception task and a motion prediction task to guide the model to learn the state information and trend information of the action, and designs a contrast learning auxiliary task to guide the model to learn the high-order semantic information of the action and enhance the temporal consistency of the features. A multi-task self-supervised learning framework is constructed, and the above loss supervision is used for training to enhance the robustness of the model. On this basis, a three-dimensional action prediction network based on a complete sequence generation module is proposed, which fuses partial prediction and global prediction to complete the missing information in the incomplete action sequence. Through experimental analysis, it can be known that the self-supervised three-dimensional action prediction method can reduce the dependence on labels, learn features with discriminative and strong representation ability, alleviate the lack of information in incomplete action sequences, strengthen the identity of action sequence features of the same category and the distinguishability of action sequence features of different categories, and has good prediction performance on multiple three-dimensional action prediction data sets.

Claims

1. A self-supervised three-dimensional motion prediction method based on past completion and future trends, The features include the following steps: 1) preprocessing the three-dimensional skeleton sequence dataset to generate a three-dimensional action prediction dataset; 2) designing an action completion awareness task and a motion prediction task supervised model training, the supervised model extracts past completion information and future trend information in the incomplete action sequence; 3) designing a contrastive learning auxiliary task, the supervised model extracts high-order semantic information in the incomplete action sequence, optimizes the feature space, and designs a multi-task self-supervised learning framework to enhance the robustness of the model, the specific steps are as follows: (1) for the incomplete action sequence sample, input the corresponding complete action sequence into the feature encoder to obtain the feature representation; (2) based on the multi-layer perception mechanism, build a contrastive learning task head, respectively project the features of the incomplete action sequence and the corresponding complete action sequence into the contrastive learning feature space, and calculate the distance between the two feature vectors from the direction and the value, to obtain the contrastive learning loss; (3) build a multi-task self-supervised learning framework, weight the sum of the action completion awareness loss, the motion prediction loss and the contrastive learning loss to obtain the total loss, and supervise the weight training of the feature encoder and the three downstream task heads; 4) design a three-dimensional action prediction network based on a complete sequence generation module, generate a complete sequence without introducing additional parameters, complete the missing information, and fuse partial prediction and global prediction to obtain the final prediction; The three-dimensional action prediction network based on the complete sequence generation module is designed, and the specific steps are as follows: (1) fuse the action completion awareness head and the motion prediction head to obtain the complete sequence generation module, that is, input the incomplete action sequence to output the corresponding complete action sequence; (2) apply the trained feature extractor to extract the features of the incomplete action sequence and the generated action sequence respectively and make a prediction, and fuse the two predictions to obtain the final prediction result; 5) on the three-dimensional action prediction dataset, calculate each loss in the multi-task self-supervised learning framework, train the network through the back propagation algorithm and the stochastic gradient descent method, and obtain the finally trained model; 6) use the trained model to test the recognition effect of three-dimensional action prediction, the model outputs the prediction value of each action category, and the output value is normalized to obtain the final action prediction result.

2. The self-supervised 3D motion prediction method based on past completion rate and future trend as described in claim 1, characterized in that... In step 1), the specific steps of preprocessing the three-dimensional skeleton sequence dataset to generate a three-dimensional action prediction dataset are as follows: (1) on a large-scale skeleton sequence data, normalize the joint coordinates, and apply down-sampling technology to unify the frame number of all samples; (2) define nine different observation rates from 0.1 to 0.9, traverse all samples in the data set, generate sequences under nine different observation rates for each sample, record the observation rate corresponding to each sequence, and unify the length of all sequences by filling blank frames with the last frame. 3.The self-supervised 3D motion prediction method based on past completion and future trend according to claim 1, wherein In step 2), the specific steps of designing an action completion awareness task and a motion prediction task supervised model training are as follows: (1) based on a two-layer bidirectional gated recurrent unit network, build a feature encoder, input the incomplete action sequence into the encoder for feature extraction, and then send it to different downstream task heads; (2) Construct an action completion perception head based on a multi-layer perception mechanism and a motion prediction head based on a recurrent neural network; (3) Send the extracted features into the action completion perception head to obtain a predicted observation rate, measure the loss between the predicted observation rate and the real observation rate with a mean square loss function, and send the extracted features into the motion prediction head to obtain a predicted human joint trajectory, and measure the loss between the predicted human joint trajectory and the real skeleton sequence with a mean square loss function.

Images