A motion prediction method based on human skeleton sequence
Through the knowledge distillation technology of graph convolution network and multi-generation student models, adaptive selection of teacher models and weighted fusion strategies, the problem of insufficient action prediction performance at low observation rates in the existing technology is solved, and the accuracy of action prediction and robustness.
Patent Information
- Application Number
- CN202210255594.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-15
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-03-15
AI Technical Summary
The existing action prediction methods based on human skeleton sequences are insufficient in prediction performance at low observation rates, and fail to fully utilize the observation prior knowledge and the potential of multiple teacher models, resulting in insufficient capture of the importance of the model on different examples.
The graph convolution network model is used to extract spatiotemporal features, and through the knowledge distillation technology of multi-generation student models, the teacher model is adaptively selected according to the observation rate for guidance, the regeneration network is designed and iteratively generated student models, and the prediction performance is improved through weighted fusion strategies.
It improves the accuracy and robustness of action prediction, makes full use of observation rates and the knowledge of multiple teacher models, and improves the classification performance of the model at low observation rates.
Smart Images

Figure CN114582024B_ABST
Abstract
Description
Technical Field
[0001] The invention discloses a method for predicting motion based on human skeleton sequence, and relates to the technical field of motion prediction based on computer vision. Background Art
[0002] Human actions usually occur in three-dimensional space, and three-dimensional skeleton sequences provide more comprehensive information than RGB videos captured by two-dimensional cameras. Nowadays, with the rapid development of low-cost depth sensors such as Microsoft Kinect and ASUS Xtion, accurate skeleton data can be directly generated in real time, and capturing human skeleton movements has become more accurate, simple and convenient. Therefore, in the field of computer vision, motion prediction research based on human skeleton sequences is becoming more and more attractive. Research on motion prediction based on human skeleton sequences is cutting-edge and challenging, and it has a very broad practical application prospect, such as human-computer interaction, autonomous driving, intelligent medical care, etc. It is a key technology for the popularization of intelligent information technology in the future.
[0003] The task of action prediction is to infer the category of the action performed as early as possible from a video of an incomplete action that has been observed. Since there is less key information available in the early stages of action execution and the complete execution sequence structure of the action is unknown, "how to improve the prediction performance of the model under low observation rates" has become a key difficulty in action prediction research. Most of the existing action prediction research methods obtain the correspondence between partial videos and complete videos, supplement the prior knowledge of complete videos to partial videos, and thus improve the prediction performance of the model.
[0004] The patent "A method for action prediction based on multi-task random forest" (2019108569846) provides an action prediction method based on multi-task random forest. It uses training videos annotated with action type labels and observation rate labels to build an action prediction model based on multi-task random forest. For a newly input video containing incomplete actions, the multi-task random forest is used to predict the action type of the video. Qiuhong Ke et al. proposed an action prediction method based on human skeleton sequence (Learning Latent Global Network for Skeleton-Based ActionPrediction, in IEEE Transactions on Image Processing, 2020). Specifically, each skeleton sequence is converted into a color pseudo-image as input, and by finding a feature space similar to the local sequence and the global sequence, complementary global information is provided to the local sequence, so as to more accurately infer the action category. This method converts the skeleton sequence into a pseudo-image as the model input, which does not fully utilize the three-dimensional human skeleton sequence data to have stronger robustness to viewpoint changes, illumination changes and cluttered backgrounds, and can provide more comprehensive motion information.
[0005] Xionghui Wang et al. proposed a new teacher-student learning framework to improve early action prediction (Progressive Teacher-student Learning for Early Action Prediction, in Internaltional Conference on Computer Vision and Pattern Recogintion, 2019). Specifically, the teacher-student learning framework includes a teacher model that recognizes actions from complete videos, a student model that predicts early actions from partial videos, and a teacher-student learning block that extracts progressive knowledge from the teacher to the student across different tasks. In addition, there are some similar related research works, such as Yijun Cai et al. proposed a two-stage knowledge transfer learning framework (Action Knowledge Transfer for Action Prediction with PartialVideos, in the Association for the Advance of Artificial Intelligence, 2019). Specifically, in the first stage, feature embeddings and discriminative action classifiers are learned from complete videos; in the second stage, the knowledge in the learned embeddings and classifiers is transferred to partial videos. However, such action prediction research methods only consider full video pre-training as the teacher model. Regardless of the observation rate of the current input sample of the student model, it is strongly constrained to align with the features or soft labels output by the full video pre-training teacher model. It does not fully and reasonably utilize the prior knowledge of observation rate, ignores the potential of a student model to learn from multiple teacher models, and cannot capture the different importance of different teacher models for each sample instance.
[0006] Therefore, how to develop a new motion prediction method based on human skeleton sequence has become an urgent problem to be solved. Summary of the invention
[0007] In view of this, the present invention provides a motion prediction method based on human skeleton sequence to achieve accurate prediction of motion.
[0008] The technical solution provided by the present invention is specifically a method for motion prediction based on human skeleton sequence, characterized in that it comprises the following steps:
[0009] S1: Using a graph convolutional network model to extract the spatiotemporal features of the human skeleton sequence data to be predicted, and inputting the spatiotemporal features into the trained multi-generation student model to obtain multiple classification results;
[0010] S2: Fusion of multiple classification results to obtain a final prediction result;
[0011] Among them, in step S1, the training process of the multi-generation student model is specifically as follows:
[0012] S101: collecting existing human skeleton sequence data for preprocessing to generate complete skeleton sequence samples and incomplete skeleton sequence samples, wherein the incomplete skeleton sequence samples include high observation rate skeleton sequence samples and low observation rate skeleton sequence samples;
[0013] S102: Using a graph convolutional network model to extract the spatiotemporal features of the complete skeleton sequence samples and the high observation rate skeleton sequence samples, respectively, and using the spatiotemporal features extracted from the complete skeleton sequence samples and the spatiotemporal features extracted from the high observation rate skeleton sequence samples for pre-training to obtain a corresponding complete teacher model Teacher 完整 And high observation teacher model Teacher 高观测 ;
[0014] S103: Design the first generation student model based on the graph convolutional network deep learning method, and the first generation student model is consistent with the pre-trained complete teacher model Teacher 完整 And high observation teacher model Teacher 高观测 Have the same network structure and number of parameters;
[0015] S104: Both the complete skeleton sequence samples and the incomplete skeleton sequence samples are used as the input samples of the first generation student model, and the graph convolutional network model is used to extract the spatiotemporal features of the input samples. According to the observation rate of the input human skeleton sequence samples, the pre-trained complete teacher model Teacher is adaptively selected. 完整 Or high observation teacher model Teacher 高观测 Provide guidance, use the true category labels as supervision, adjust the parameters in the first-generation student model, and obtain the trained first-generation student model;
[0016] S105: Based on the first generation student model, the remaining multiple generations of student models are designed and trained in sequence, wherein the design and training of the mth generation student model are specifically as follows:
[0017] An m-th generation student model with the same network structure and parameter amount as the m-1-th generation student model is designed, and both the complete skeleton sequence samples and the incomplete skeleton sequence samples are used as input samples of the m-1-th generation student model and the m-th generation student model. A graph convolutional network model is used to extract the spatiotemporal features of the input samples, and the soft labels output by the m-th generation student model are used as guidance and the true category labels are used as supervision to adjust the parameters in the m-th generation student model to obtain the trained m-th generation student model, where m is a positive integer greater than or equal to 2.
[0018] Preferably, in step S101, existing human skeleton sequence data is collected for preprocessing to generate complete skeleton sequence samples and incomplete skeleton sequence samples, specifically:
[0019] Each human skeleton sequence sample is evenly divided into 10 segments, and the corresponding skeleton data under different observation rates are generated;
[0020] The frame cycle sampling method is adopted to take the skeleton data frame at different observation rates as 300 frames. If the skeleton data frame is less than 300 frames, the previous frame is used for filling. If the skeleton data frame is greater than 300 frames, the content after 300 frames is discarded.
[0021] Further preferably, in step S101, the high observation rate skeleton sequence sample is a sequence data content whose observed content accounts for more than 50% of the complete sequence data content;
[0022] The low observation rate skeleton sequence samples are sequence data samples in which the observed data content accounts for 50% or less of the complete sequence data content.
[0023] Further preferably, in step S102, the complete teacher model Teacher 完整 And high observation teacher model Teacher 高观测 The loss function is:
[0024]
[0025] Where N is the number of samples; C is the number of categories; when the true category label of the i-th sample is category c, y i,c =1, otherwise 0; p i,c It represents the probability of predicting sample i as category c.
[0026] Further preferably, the graph convolution network model used for the spatiotemporal feature extraction is a shifted graph convolution network, and the shifted graph convolution network has a total of 10 graph convolution layers, and each graph convolution layer includes a spatial domain shifted graph convolution operation and a temporal domain shifted graph convolution operation.
[0027] Further preferably, in step S104, according to the observation rate of the input human skeleton sequence sample, the pre-trained complete teacher model Teacher is adaptively selected. 完整 Or high observation teacher model Teacher 高观测 Provide guidance, specifically:
[0028] When the observation rate of the input human skeleton sequence sample is less than or equal to 0.5, the first generation student model is based on the pre-trained high observation teacher model Teacher 高观测 For guidance;
[0029] When the observation rate of the input human skeleton sequence sample is greater than 0.5, the first generation student model is trained with the complete teacher model Teacher 完整 For guidance.
[0030] Further preferably, in step S104, the loss function of the first generation student model is specifically:
[0031]
[0032]
[0033]
[0034] Where N is the number of samples; C is the number of categories; when the true category label of the i-th sample is category c, y i,c =1, otherwise 0; p i,c Indicates the probability of predicting sample i as category c; α is used to balance the classification loss and distillation loss; Output' i,t Output' is the soft label output by the pre-trained teacher model; i,s Output is the soft target output by the first generation student model, that is, the smoothed probability distribution; i,t Output is the output of the fully connected layer of the pre-trained teacher model; i,s is the output of the fully connected layer of the first generation student model; T represents the temperature parameter in the knowledge distillation method.
[0035] Further preferably, in step S105, the loss function of the m-th generation student model is:
[0036]
[0037] ρ' i =k×ρ i +b;
[0038] Where N is the number of samples; C is the number of categories; when the true category label of the i-th sample is category c, yi,c =1, otherwise 0; p i,c represents the probability of predicting sample i as category c; α is used to balance the classification loss and distillation loss; k represents the linear scaling factor; ρ i represents the observation rate; b represents the bias.
[0039] More preferably, in step S2, a weighted approach is used to fuse the classification results.
[0040] The motion prediction method based on human skeleton sequence provided by the present invention makes full use of the human topological graph structure, adopts graph convolutional network to extract rich and representative spatiotemporal features, and adopts the knowledge distillation technology of multi-teacher model to learn prior knowledge from complete sequence and high observation rate sequence. Specifically, each sample instance adaptively selects the corresponding pre-trained teacher model according to the observation rate to perform knowledge transfer learning. The student model generated by the multi-teacher model adaptively guides the generation of multiple student models through iterative training by adopting the regeneration network technology. In the regeneration network training process, due to the weak classification performance of the model under low observation rate, the cross entropy loss function is weighted based on the observation rate of the sample instance to optimize the student model. By adopting the weighted fusion strategy, the prediction results of multiple student models are fused as the final prediction classification result.
[0041] The motion prediction method based on human skeleton sequence provided by the present invention has the following advantages:
[0042] 1. The present invention adopts a graph convolutional network model, makes full use of the topological structure of human beings, and extracts spatiotemporal features from the observed human skeleton sequence data. The spatiotemporal features have rich spatial information and temporal information and are more representative. If the human skeleton sequence data is converted into a pseudo-image and a convolutional neural network CNN is used for feature extraction, although rich spatial features can be extracted, the expression of temporal information is lacking; if a recurrent neural network RNN is used to extract features from the human skeleton sequence data, although it can fit the serialization characteristics of the skeleton sequence and perform action classification by learning the temporal context relationship, the spatial information expression ability of the skeleton sequence is relatively weak.
[0043] 2. The present invention proposes a knowledge distillation learning method for adaptive multi-teacher models. According to the observation rate of the student model input sample, the method adaptively selects a suitable complete pre-trained teacher model or a high observation rate pre-trained teacher model to provide soft label guidance information. This method makes full use of the prior knowledge of the observation rate of the sample instance and fully explores the potential of a student model to learn from multiple teacher models. By adaptively selecting the pre-trained teacher model through the sample instance, the different importance of different teacher models to each sample instance is captured, which has a certain degree of flexibility.
[0044] 3. The present invention proposes to use regenerative network technology to iteratively generate student models, and considers the differences in classification performance under different observation rates. It designs to weight the cross entropy loss function based on the sample instance observation rate, and fuses multiple student models with a weighted fusion strategy to improve the model prediction classification performance.
[0045] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0048] Figure 1 A flowchart of a method for motion prediction based on human skeleton sequence provided by an embodiment of the present invention;
[0049] Figure 2 A training flow chart of a multi-generation student model provided for an embodiment of the present invention. DETAILED DESCRIPTION
[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0051] This embodiment provides a method for motion prediction based on human skeleton sequence. Figure 1 , the method comprises the following steps:
[0052] S1: A graph convolutional network model is used to extract the spatiotemporal features of the human skeleton sequence data to be predicted, and the spatiotemporal features are input into the trained multi-generation student model to obtain multiple classification results;
[0053] S2: Fusion of multiple classification results to obtain the final prediction result.
[0054] The result fusion in step S2 can be performed in a weighted fusion manner, which can be either an average value or multiplying the classification result of each model by a corresponding coefficient based on experience, but the sum of the coefficients before all student models is required to be equal to 1.
[0055] Combined with the above prediction method, if the prediction is completed, it is necessary to pre-train multiple generations of student models. The training process of multiple generations of student models is as follows:
[0056] S101: collecting existing human skeleton sequence data for preprocessing to generate complete skeleton sequence samples and incomplete skeleton sequence samples, wherein the incomplete skeleton sequence samples include high observation rate skeleton sequence samples and low observation rate skeleton sequence samples;
[0057] S102: Use the graph convolutional network model to extract the spatiotemporal features of complete skeleton sequence samples and high-observation rate skeleton sequence samples respectively, and use the spatiotemporal features extracted from the complete skeleton sequence samples and the spatiotemporal features extracted from the high-observation rate skeleton sequence samples to pre-train and obtain the corresponding complete teacher model Teacher 完整 And high observation teacher model Teacher 高观测 ;
[0058] S103: Design the first generation student model based on the graph convolutional network deep learning method, and the first generation student model is consistent with the pre-trained complete teacher model Teacher 完整 And high observation teacher model Teacher 高观测 Have the same network structure and number of parameters;
[0059] S104: Both the complete skeleton sequence samples and the incomplete skeleton sequence samples are used as the input samples of the first generation student model, and the graph convolutional network model is used to extract the spatiotemporal features of the input samples. According to the observation rate of the input human skeleton sequence samples, the pre-trained complete teacher model Teacher is adaptively selected. 完整 Or high observation teacher model Teacher 高观测 Provide guidance, use the true category label as supervision, adjust the parameters in the first generation student model, and obtain the trained first generation student model; when the student model adaptively selects the complete teacher model Teacher 完整 When used as a teacher model to guide training, it can learn the prior knowledge of the complete skeleton sequence; when the student model adaptively selects the high-observation teacher model Teacher 高观测 When used as a teacher model to guide training, it can learn the prior knowledge of high-observation rate skeleton sequences;
[0060] S105: Based on the first generation student model, the remaining multiple generations of student models are designed and trained in sequence. This step uses the regenerative network technology to use the generated student model as the teacher model for a new round of knowledge distillation learning. The design and training of the mth generation student model are as follows:
[0061] An m-th generation student model with the same network structure and parameters as the m-1-th generation student model is designed, and both the complete skeleton sequence samples and the incomplete skeleton sequence samples are used as input samples of the m-1-th generation student model and the m-th generation student model. A graph convolutional network model is used to extract the spatiotemporal features of the input samples, and the soft labels output by the m-th generation student model are used as guidance and the true category labels are used as supervision to adjust the parameters in the m-th generation student model to obtain the trained m-th generation student model, where m is a positive integer greater than or equal to 2.
[0062] In step S101, the known human skeleton sequence data is preprocessed to generate a complete skeleton sequence data set and an incomplete skeleton sequence data set. First, each sequence sample is evenly divided into 10 segments, and the skeleton data under different observation rates are generated accordingly. For example, if the first segment corresponds to 10% of the observed content in the complete sequence, the observation rate is 0.1; if the first two segments correspond to 20% of the observed content in the complete sequence, the observation rate is 0.2; and so on, the first 10 segments represent a complete skeleton sequence with an observation rate of 1.0. Then, a frame cycle sampling method is adopted to take 300 frames for each sequence sample frame under different observation rates. When the sequence sample frame is less than 300 frames, the previous frame is used for filling; if the sequence sample frame is greater than 300 frames, the content after 300 frames is discarded. A complete skeleton sequence data set refers to data with an observation rate of 1.0; incomplete skeleton sequence data includes observation rates of 0.1 to 0.9.
[0063] In this implementation scheme, high observation rate skeleton sequence data samples and complete observation rate skeleton sequence data samples are selected as training samples of the teacher model. In order to extract representative deep feature information of human skeleton sequence with rich spatiotemporal information and effectively represent comprehensive motion information, this implementation scheme uses a shift graph convolutional network to extract spatiotemporal information from human skeleton sequence data with different observation rates to obtain the feature vector F t , and use the extracted feature vector as the input of the fully connected layer to get the output Output t , and then input it into the Softmax multi-classifier, and iteratively train it with the real label as the supervision information, and finally train to obtain the complete teacher model Teacher 完整 And high observation teacher model Teacher 高观测 , where the loss function is shown in formula (1):
[0064]
[0065] Where N is the number of samples; C is the number of categories; when the true category label of the i-th sample is category c, yi,c =1, otherwise 0; p i,c It represents the probability of predicting sample i as category c.
[0066] The shifted graph convolutional network model used to extract the spatiotemporal features of human skeleton sequence data has a total of 10 graph convolution layers, and each graph convolution layer includes spatial domain shifted graph convolution operations and temporal domain shifted graph convolution operations. Among them, the model downsamples the human skeleton sequence feature map in the frame dimension at the 5th and 8th layers, setting stride = 2. The shifted graph convolutional network uses shift operations and point convolution operations instead of traditional graph convolution operations, which greatly reduces the amount of calculation.
[0067] When training the first generation student model, both the complete skeleton sequence data and the incomplete skeleton sequence data are used as the input of the first generation student model, and the graph convolutional network model is used to extract spatiotemporal features to obtain the feature vector F s , use the feature vector as the input of the fully connected layer to get the output Output s Then, the pre-trained teacher model is selected according to the observation rate ρ of the input sample. When the input sample observation rate is 0.1≤ρ≤0.5, the pre-trained high observation teacher model Teacher is selected. 高观测 The output soft labels are used as guidance information to obtain prior knowledge in high-observation-rate skeleton sequence data; when the input sample observation rate is 0.6≤ρ≤1.0, the pre-trained complete teacher model Teacher is selected 完整 , to obtain the prior knowledge in the complete skeleton sequence data. In the training process of the first generation student model, the real category label and the soft labels output by different pre-trained teacher models are used as guidance supervision information. The total loss function of the first generation student model is shown in formula (2):
[0068]
[0069]
[0070]
[0071] In formula (2), N represents the number of samples; C represents the number of categories; when the true category label of the i-th sample is category c, y i,c =1, otherwise 0; p i,c Indicates the probability of predicting sample i as category c; α is used to balance the classification loss and distillation loss; Output' i,t Output' is the soft label output by the pre-trained teacher model; i,s is the soft target output by the student model. In formula (3) and formula (4), Output i,tis the output of the fully connected layer of the pre-trained teacher model; Output i t ,s is the output of the fully connected layer of the student model; T represents the temperature parameter in the knowledge distillation technique.
[0072] When training the remaining multiple generations of student models, the regeneration network technology is used, and the generated student model is used as the pre-trained teacher model. The student model with the same network structure and parameter quantity as the teacher model is designed, and the complete human skeleton sequence and the incomplete skeleton sequence are used as the input of the pre-trained teacher model and the student model. The knowledge distillation learning method is used for training. Specifically, the soft labels output by the pre-trained teacher model are used as guidance, and the real category labels are used as supervision information for training. When designing the total loss function of the regenerated student model, considering that the classification performance of the low observation rate model is weak, the classification loss is weighted based on the observation rate of the input sample, as shown in formula (6). The total loss of the regenerated student model is as shown in formula (5):
[0073]
[0074] ρ' i =k×ρ i +b (6)
[0075] Where N is the number of samples; C is the number of categories; when the true category label of the i-th sample is category c, y i,c =1, otherwise 0; p i,c represents the probability of predicting sample i as category c; α is used to balance the classification loss and distillation loss; k represents the linear scaling factor; ρ i represents the observation rate; b represents the bias.
[0076] Experiments have shown that the final prediction performance and accuracy are significantly improved by using the above method for prediction.
[0077] Those skilled in the art will readily appreciate other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses or adaptations of the present invention that follow the general principles of the present invention and include common knowledge or customary techniques in the art that are not disclosed by the present invention. The specification and examples are to be considered exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.
[0078] It should be understood that the present invention is not limited to what has been described above and that various modifications and changes may be made without departing from its scope. The scope of the present invention is limited only by the appended claims.
Claims
1. A motion prediction method based on human skeleton sequence, characterized in that: The steps include: S1: Using a graph convolutional network model to extract the spatiotemporal features of the human skeleton sequence data to be predicted, and inputting the spatiotemporal features into the trained multi-generation student model to obtain multiple classification results; S2: Fusion of multiple classification results to obtain a final prediction result; Among them, in step S1, the training process of the multi-generation student model is specifically as follows: S101: collecting existing human skeleton sequence data for preprocessing to generate complete skeleton sequence samples and incomplete skeleton sequence samples, wherein the incomplete skeleton sequence samples include high observation rate skeleton sequence samples and low observation rate skeleton sequence samples; S102: Using a graph convolutional network model to extract the spatiotemporal features of the complete skeleton sequence samples and the high observation rate skeleton sequence samples, respectively, and using the spatiotemporal features extracted from the complete skeleton sequence samples and the spatiotemporal features extracted from the high observation rate skeleton sequence samples for pre-training to obtain a corresponding complete teacher model Teacher 完整 And high observation teacher model Teacher 高观测 ; S103: Design the first generation student model based on the graph convolutional network deep learning method, and the first generation student model is consistent with the pre-trained complete teacher model Teacher 完整 And high observation teacher model Teacher 高观测 Have the same network structure and number of parameters; S104: Both the complete skeleton sequence samples and the incomplete skeleton sequence samples are used as the input samples of the first generation student model, and the graph convolutional network model is used to extract the spatiotemporal features of the input samples. According to the observation rate of the input human skeleton sequence samples, the pre-trained complete teacher model Teacher is adaptively selected. 完整 Or high observation teacher model Teacher 高观测 Provide guidance, use the true category labels as supervision, adjust the parameters in the first-generation student model, and obtain the trained first-generation student model; S105: Based on the first generation student model, the remaining multiple generations of student models are designed and trained in sequence, wherein the design and training of the mth generation student model are specifically as follows: An m-th generation student model with the same network structure and parameter amount as the m-1-th generation student model is designed, and both the complete skeleton sequence samples and the incomplete skeleton sequence samples are used as input samples of the m-1-th generation student model and the m-th generation student model. A graph convolutional network model is used to extract the spatiotemporal features of the input samples, and the soft labels output by the m-th generation student model are used as guidance and the true category labels are used as supervision to adjust the parameters in the m-th generation student model to obtain the trained m-th generation student model, where m is a positive integer greater than or equal to 2.
2. The motion prediction method based on human skeleton sequence according to claim 1, characterized in that: In step S101, existing human skeleton sequence data is collected for preprocessing to generate complete skeleton sequence samples and incomplete skeleton sequence samples, specifically: Each human skeleton sequence sample is evenly divided into 10 segments, and the corresponding skeleton data under different observation rates are generated; The frame cycle sampling method is adopted to take the skeleton data frame at different observation rates as 300 frames. If the skeleton data frame is less than 300 frames, the previous frame is used for filling. If the skeleton data frame is greater than 300 frames, the content after 300 frames is discarded.
3. The motion prediction method based on human skeleton sequence according to claim 1, characterized in that: In step S101, the high observation rate skeleton sequence sample is a sequence data content whose observed content accounts for more than 50% of the complete sequence data content; The low observation rate skeleton sequence samples are sequence data samples in which the observed data content accounts for 50% or less of the complete sequence data content.
4. The motion prediction method based on human skeleton sequence according to claim 1, characterized in that: In step S102, the complete teacher model Teacher 完整 And high observation teacher model Teacher 高观测 The loss function is: Where N is the number of samples; C is the number of categories; when the true category label of the i-th sample is category c, y i,c =1, otherwise 0; p i,c It represents the probability of predicting sample i as category c.
5. The motion prediction method based on human skeleton sequence according to claim 1, characterized in that: The graph convolution network model used for spatiotemporal feature extraction is a shifted graph convolution network. The shifted graph convolution network has a total of 10 graph convolution layers, and each graph convolution layer includes a spatial domain shifted graph convolution operation and a temporal domain shifted graph convolution operation.
6. The motion prediction method based on human skeleton sequence according to claim 1, characterized in that: In step S104, according to the observation rate of the input human skeleton sequence samples, a pre-trained complete teacher model Teacher is adaptively selected. 完整 Or high observation teacher model Teacher 高观测 Provide guidance, specifically: When the observation rate of the input human skeleton sequence sample is less than or equal to 0.5, the first generation student model is based on the pre-trained high observation teacher model Teacher 高观测 For guidance; When the observation rate of the input human skeleton sequence sample is greater than 0.5, the first generation student model is trained with the complete teacher model Teacher 完整 For guidance.
7. The motion prediction method based on human skeleton sequence according to claim 1, characterized in that: In step S104, the loss function of the first generation student model is specifically: Where N is the number of samples; C is the number of categories; when the true category label of the i-th sample is category c, y i,c =1, otherwise 0; p i,c Indicates the probability of predicting sample i as category c; α is used to balance the classification loss and distillation loss; Output' i,t Output' is the soft label output by the pre-trained teacher model; i,s Output is the soft target output by the first generation student model; i,t Output is the output of the fully connected layer of the pre-trained teacher model; i,s is the output of the fully connected layer of the first generation student model; T represents the temperature parameter in the knowledge distillation method.
8. The motion prediction method based on human skeleton sequence according to claim 1, characterized in that: In step S105, the loss function of the m-th generation student model is: r' i =k×ρ i +b; Where N is the number of samples; C is the number of categories; when the true category label of the i-th sample is category c, y i,c =1, otherwise 0; p i,c represents the probability of predicting sample i as category c; α is used to balance the classification loss and distillation loss; k represents the linear scaling factor; ρ i represents the observation rate; b represents the bias; Output' i,t Output' is the soft label output by the pre-trained teacher model; i,s is the soft target output by the first generation student model.
9. The motion prediction method based on human skeleton sequence according to claim 1, characterized in that: In step S2, a weighted approach is used to fuse the classification results.
