A human motion prediction method

Abstract

The present invention provides a human motion prediction method, comprising dividing human motion sequence samples into a historical sequence, a sequence to be predicted, and a future sequence; inputting the historical sequence and the future sequence into a motion feature extraction module; extracting and inputting the historical motion features and the future motion features into a preset motion prediction module for training to construct a dual-stream motion prediction network; adding the dual-stream motion prediction network and a classification module into a preset dictionary module to construct a dual-stream information storage network; constructing a motion prediction test network through the motion feature extraction module, the classification module, the motion prediction module, and the stored dictionary module; inputting the historical motion features into the classification module of the motion prediction test network to obtain motion categories, and searching for future motion features in a storage dictionary corresponding to the motion categories; adding the historical motion features and the future motion features, and inputting the sum into the motion prediction module of the motion prediction test network to realize motion prediction.

Description

Technical Field

[0001] The present invention relates to the technical field of motion prediction, and in particular to a method for predicting human motion. Background Art

[0002] Human motion prediction is currently a key topic in computer vision. It has widespread applications in areas such as human-computer interaction and autonomous driving. Compared to video-based human motion prediction, human skeleton-based human motion prediction is less susceptible to factors such as complex backgrounds, lighting variations, and appearance changes. Consequently, many researchers are researching human skeleton-based human motion prediction.

[0003] Traditional methods typically use Markov models to model the dynamic environment of human motion, predicting future human motion based on historical human motion. However, this approach often struggles with long-term motion prediction, and experimental results have been less than ideal. Currently, mainstream methods rely on historical motion sequences to achieve motion prediction, aiming to analyze the body's inertial motion. As the prediction time increases, human motion becomes more susceptible to external influences, leading to higher prediction errors.

[0004] Therefore, to address the above motion prediction algorithm problems, a human motion prediction method is proposed. Summary of the Invention

[0005] The present invention provides a human motion prediction method to solve the above problems.

[0006] The present invention provides a method for predicting human motion:

[0007] according to Figure 2 As shown in , a human action sequence sample contains 100 frames of human skeletons. Take any human action sequence sample V as an example:

[0008] V=[X1,...,X t ,...,X 100 ],t∈[1,100],

[0009] Among them, t represents the time index frame number of the human skeleton, the human action sequence sample V has a total of 100 frames, X t ∈R 20×60 is the matrix representation of the human skeleton of the t-th frame of the human action sequence sample V. The number of rows and columns of the matrix representation of the human skeleton of this frame are 20 and 60 respectively. R indicates that the matrix is a real matrix. The human skeleton of this frame is represented by 20 key points, and each key point is represented by a 60-dimensional vector. The human skeleton of this frame is as follows Figure 10 shown.

[0010] The human motion sequence sample V is divided into a historical sequence, a sequence to be predicted, and a future sequence. The historical sequence has 50 frames, the sequence to be predicted has 25 frames, and the future sequence has 25 frames. At this time, the human motion sequence sample can be represented as a set consisting of a historical sequence, a sequence to be predicted, and a future sequence, which can be expressed as follows:

[0011] V={V history ,V predict ,V future},

[0012] Among them, V history Represents the historical sequence, V predict represents the sequence to be predicted, V future Represents a future sequence.

[0013] Historical Sequence V history The formula is as follows:

[0014] V history =[X1,...,X h ,...,X 50 ], h∈[1,50],

[0015] Among them, h represents the time index, the history sequence has a total of 50 frames, and the history sequence consists of the human skeleton of the first frame to the 50th frame of the human action sequence sample V. h ∈R 20×60 For historical sequence V history The matrix representation of the h-th frame human skeleton;

[0016] Sequence to be predicted V predict The formula is as follows:

[0017] V predict =[X 51 ,...,X p ,...,X 75 ],p∈[51,75],

[0018] Among them, p represents the time index, and the sequence to be predicted has a total of 25 frames. The sequence to be predicted consists of the human skeleton from the 51st frame to the 75th frame of the human action sequence sample V. is the sequence to be predicted V predict The matrix representation of the human skeleton of the pth frame,

[0019] Future Sequence V future The formula is as follows:

[0020] V future =[X 76 ,…,X f ,…,X 100],f∈[76,100],

[0021] Among them, f represents the time index. The future sequence has a total of 25 frames. The future sequence consists of the human skeleton of the 76th frame to the 100th frame of the human action sequence sample V. f ∈R 20×60 is the sequence to be predicted V predict The matrix representation of the human skeleton of the f-th frame;

[0022] The historical sequence divided by the human motion sequence samples is input into the motion feature extraction module to extract historical motion features, and the future sequence divided by the human motion sequence samples is input into the motion feature extraction module to extract future motion features. The motion feature extraction module includes: a velocity calculation module, an acceleration calculation module, a level division module, a fusion module 1, a fusion module 2, a low-level spatiotemporal convolution module 1, a low-level spatiotemporal convolution module 2, a low-level spatiotemporal convolution module 3, a low-level spatiotemporal convolution module 4, a mid-level spatiotemporal convolution module 1, a mid-level spatiotemporal convolution module 2, a mid-level spatiotemporal convolution module 3, a mid-level spatiotemporal convolution module 4, a high-level spatiotemporal convolution module 1, a high-level spatiotemporal convolution module 2, a high-level spatiotemporal convolution module 3, a high-level spatiotemporal convolution module 4, a feature fusion module and a feature cascade module.

[0023] The process of extracting historical motion features from the historical sequence input motion feature extraction module is as follows: Figure 2 As shown, the input of the speed calculation module is the historical sequence V divided by the human motion sequence samples history , the output is the historical speed sequence V history_v . V history_v The formula is as follows:

[0024] V history_v =[vel1,...,vel h ,...,vel 50 ],h∈[1,50],

[0025] Among them, vel h ∈R 20×60 is the historical speed sequence V history_v The matrix expression of the h-th frame skeleton, vel h is calculated as follows:

[0026]

[0027] The input of the acceleration calculation module is the historical velocity sequence V history_v , the output is the historical acceleration sequence V history_a , V history_a The formula is as follows:

[0028] Vhistory_a =[acl1,...,acl h ,...,acl 50 ],h∈[1,50],

[0029] Among them, acl h ∈R 20×60 is the historical acceleration sequence V history_a The matrix expression of the h-th frame skeleton in acl h is calculated as follows:

[0030]

[0031] like Figure 2 As shown in the leftmost column, the input of level division module 1 is V history , which has three outputs, namely V hp_1 、V hp_2 and V hp_3 , V hp_1 Represents the historical sequence of low-level human skeleton, V hp_2 Represents the historical sequence of the middle-level human skeleton, V hp_3 Represents the historical sequence of high-level human skeletons; the level division module consists of pooling layer 1 and pooling layer 2; pooling layer 1 consists of 10 average pooling layers with a pooling size of 2; pooling layer 2 consists of 5 average pooling layers with a pooling size of 4; V hp_1 =V history , which means that the historical sequence is regarded as the historical sequence of the low-level human skeleton. The historical sequence V history Input to pooling layer 1 to obtain the historical sequence V of the mid-level human skeleton hp_2 , V hp_2 The expression is as follows:

[0032] V hp_2 =[mid1,...,mid h ,...,mid 50 ],h∈[1,50],

[0033] in, is the matrix representation of the h-th frame skeleton in the history sequence of the middle-level human skeleton. The human skeleton of this frame is as follows Figure 11 As shown, the historical sequence V history Input to pooling layer 2 to obtain the historical sequence V of the high-level human skeleton hp_3 , V hp_3 The expression is as follows:

[0034] V hp_3 =[high1,...,high h ,...,high 50],h∈[1,50],

[0035] Among them, high h ∈R 5×60 is the matrix representation of the h-th frame skeleton in the history sequence of the human skeleton of a higher level person. The human skeleton of this frame is as follows Figure 12 shown.

[0036] The input of the low-level spatiotemporal convolution module 1 is V hp_1 , the output is F hp_11 The low-level spatiotemporal convolution module 1 consists of 1 spatial graph convolution layer and 1 temporal convolution layer. The spatial graph convolution layer has 64 convolution kernels, each of which is 1×1 in size. The temporal convolution layer has 32 convolution kernels, each of which is 5×1 in size. The structure of the mid-level spatiotemporal convolution module 1 is consistent with that of the low-level spatiotemporal convolution module 1, and its input is V hp_2 , the output is F hp_12 The structure of the high-level spatiotemporal convolution module 1 is consistent with that of the low-level spatiotemporal convolution module 1, and its input is V hp_3 , the output is F hp_13 ;

[0037] The low-level convolution layer 1 contains 32 convolution kernels, each of which is 5×1 in size. The input of the low-level convolution layer 1 is F hp_11 , the output is G 11 ; The low-level linear layer 1 consists of 3 fully connected layers, whose input is G 11 , the output is F 11 ; F 11 ∈R 20 ×256 , F 11 Represents the features of all key points in the extracted low-level human skeleton; F 11_i ∈R 256 represents the feature of the i-th key point in the extracted low-level human skeleton; F 11_j ∈R 256 Represents the features of the jth key point in the extracted low-level human skeleton; for the i-th key point in the low-level human skeleton, the low-level aggregation related feature layer 1 aggregates the features of the remaining 19 key points to the i-th key point, and then passes through 3 layers of fully connected layers. Its formula is expressed as follows:

[0038]

[0039] Among them, [·,·] means connection according to channel, f 11 (·) represents 3 fully connected layers, r 11_i is a column vector. After performing the above operations on all key points in the low-level human skeleton, we get R 11 , R 11The formula is as follows:

[0040] R 11 =(r 11_1 ,...,r 11_i ,...,r 11_20 ), i∈[1,20],

[0041] This formula represents R 11 Is a two-dimensional tensor consisting of 20 column vectors; the input of the low-level update feature layer 1 is R 11 , the output is H 11 ; Low-level update feature layer 1 is used to update features. It consists of three fully connected layers. The formula for feature update is as follows:

[0042] H 11 =g 11 ([F 11 ,R 11 ]),

[0043] Among them, [·,·] means connection according to channel, g 11 (·) represents 3 fully connected layers, H 11 are the features of all key points in the updated low-level human skeleton.

[0044] The structure of the mid-level convolutional layer 1 is the same as that of the low-level convolutional layer 1, and its input is F hp_12 , the output is G 12 The structure of the middle-level linear layer 1 is consistent with that of the low-level linear layer 1, and its input is G 12 , the output is F 12 The structure of the mid-level aggregation-related feature layer 1 is consistent with that of the low-level aggregation-related feature layer 1, and its input is F 12 , the output is R 12 The structure of the mid-level update feature layer 1 is consistent with that of the low-level update feature layer 1, and its input is R 12 , the output is H 12 ;H 12 are the features of all key points in the updated mid-level human skeleton.

[0045] The structure of the high-level convolutional layer 1 is the same as that of the low-level convolutional layer 1, and its input is F hp_13 , the output is G 13 ; The structure of the high-level linear layer 1 is consistent with that of the low-level linear layer 1, and its input is G 13 , the output is F 13 The structure of the high-level aggregation-related feature layer 1 is consistent with that of the low-level aggregation-related feature layer 1, and its input is F 13 , the output is R 13; The structure of the high-level update feature layer 1 is consistent with that of the low-level update feature layer 1, and its input is R 13 , the output is H 13 ;H 13 are the features of all key points in the updated mid-level human skeleton.

[0046] The input of low-level feature fusion layer 1 is H 11 and H 12 , the output is S hp_11 ; In the low-level feature fusion layer 1, H 11 and H 12 First pass through a softmax function, the formula is as follows:

[0047]

[0048] Among them, T represents transpose, A 12 Represents the connection weight matrix; the softmax function is used to normalize the value of each element of the input vector to between (0,1). The calculation formula for each element is as follows:

[0049]

[0050] Among them, e represents the natural base, K represents the number of elements contained in the input vector, and a l Represents the lth element of the input vector, a k represents the kth element of the input vector, S l Represents the result of normalizing the value of the lth element of the input vector; then, H 12 Fusion to H 11 The formula is expressed as follows:

[0051] S hp_11 =A 12 ·H 12 *0.3+H 11 ,

[0052] Among them, S hp_11 Indicates H 12 and H 11 The fused features are also the output of the low-level feature fusion layer 1, and 0.3 represents the weight.

[0053] The input of the mid-level feature fusion layer 1 is H 11 、H 12 and H 13 , the output is S hp_12 ; In the mid-level feature fusion layer 1, H 11 and H 12 First pass through a softmax function, the formula is as follows:

[0054]

[0055] Among them, T represents transpose, A 21 represents the connection weight. Then, H 13 and H 12 After a softmax function, the formula is expressed as follows:

[0056]

[0057] Among them, T represents transpose, A 23 Represents the connection weight. Next, H 11 and H 13 Fusion to H 12 The formula is expressed as follows:

[0058] S hp_12 =(A 21 ·H 11 +A 23 ·H 13 )*0.3+H 12 ,

[0059] Among them, S hp_12 Indicates H 11 、H 12 and H 13 The fused features are also the output of the mid-level feature fusion layer 1, and 0.3 represents the weight.

[0060] The input of high-level feature fusion layer 1 is H 13 and H 12 , the output is S hp_13 In the high-level feature fusion layer 1, H 13 and H 12 First pass through a softmax function, the formula is as follows:

[0061]

[0062] Among them, T represents transpose, A 32 Represents the connection weight matrix. Next, H 12 Fusion to H 13 The formula is expressed as follows:

[0063] S hp_13 =A 32 ·H 12 *0.3+H 13 ,

[0064] Among them, S hp_13 Indicates H 12 and H 13The fused features are also the output of the high-level feature fusion layer 1, and 0.3 represents the weight;

[0065] The input of the low-level spatiotemporal convolution module 2 is S hp_11 , the output is F hp_21 The low-level spatiotemporal convolution module 2 contains 1 spatial graph convolution layer and 1 temporal convolution layer. The spatial graph convolution layer has 128 convolution kernels, each of which is 1×1 in size; the temporal convolution layer has 64 convolution kernels, each of which is 5×1 in size; the input of the mid-level spatiotemporal convolution module 2 is S hp_12 , the output is F hp_22 The structure of the mid-level spatiotemporal convolution module 2 is consistent with that of the low-level spatiotemporal convolution module 2, and its input is S hp_12 , the output is F hp_22 The structure of the high-level spatiotemporal convolution module 2 is consistent with that of the low-level spatiotemporal convolution module 2, and its input is S hp_13 , the output is F hp_23 .

[0066] The structure of fusion module 2 is consistent with that of fusion module 1. Fusion module 2 has three outputs, namely S hp_21 、S hp_22 and S hp_23 .

[0067] The input of the low-level spatiotemporal convolution module 3 is S hp_21 , the output is S hp_31 The low-level spatiotemporal convolution module 3 consists of 1 spatial graph convolution layer and 1 temporal convolution layer. The spatial graph convolution layer has 256 convolution kernels, each of which is 1×1 in size. The temporal convolution layer has 128 convolution kernels, each of which is 5×1 in size. The input is S hp_22 , the output is S hp_3 2 The structure of the mid-level spatiotemporal convolution module 3 is consistent with that of the low-level spatiotemporal convolution module 3, and its input is S hp_22 , the output is S hp_32 The structure of the high-level spatiotemporal convolution module 3 is consistent with that of the low-level spatiotemporal convolution module 3, and its input is S hp_23 , the output is S hp_33 .

[0068] The input of low-level spatiotemporal convolution module 4 is S hp_31 , the output is S hp_41 The low-level spatiotemporal convolution module 4 contains 1 spatial graph convolution layer and 1 temporal convolution layer. The spatial graph convolution layer has 512 convolution kernels, each of which is 1×1 in size. The temporal convolution layer has 256 convolution kernels, each of which is 5×1 in size. The structure of the mid-level spatiotemporal convolution module 4 is consistent with that of the low-level spatiotemporal convolution module 4, and its input is S hp_32, the output is S hp_42 The structure of the high-level spatiotemporal convolution module 4 is consistent with that of the low-level spatiotemporal convolution module 4, and its input is S hp_33 , the output is S hp_43 .

[0069] The input of the feature fusion module is S hp_41 、S hp_42 and S hp_43 , the output is S hp ; In order to make S hp_41 、S hp_42 and S hp_43 The tensor sizes of the three are consistent; the feature fusion module has a large hp_42 Perform 10 upsamplings to get And Assigning a weight of 0.6 yields To S hp_43 Perform 5 upsamplings to get And Assigning a weight of 0.6 yields Then S hp_41 、 and Add; S hp The expression is as follows:

[0070]

[0071] in, and The weights are all 0.6.

[0072] like Figure 2 As shown in the middle column, its structure is the same as Figure 2 The leftmost column has the same structure. Figure 2 The input of the middle column is the historical speed sequence V history_v , the output is S hv .like Figure 2 As shown in the rightmost column, its structure is the same as Figure 1 The structure of the leftmost column is consistent; Figure 2 The input of the rightmost column is the historical acceleration sequence V history_a , the output is S ha .

[0073] The input of the feature cascade module is S hp 、S hv and S ha , the output is H history , H history By S hp 、S hv and S ha The result is obtained by splicing channels. historyRepresents the characteristics of historical movements.

[0074] Future sequence V of human action sequence sample division future Input motion feature extraction module to extract the process of future motion features and the historical sequence V of human action sequence sample division history The process of extracting historical features from the input motion feature extraction module is consistent, such as Figure 3 As shown, H future Represents the characteristics of future sports.

[0075] As an embodiment of this technology, the historical motion features and the future motion features are added together and input into the action prediction module. The historical motion features and the future motion features are added together to obtain the joint motion features, which can be expressed as follows:

[0076] H=H future +H history ,

[0077] Where H represents the joint motion feature.

[0078] The action prediction module is composed of a recurrent neural network with a total of 25 cycles. Its structure is as follows: Figure 3 As shown in Figure 1, it includes a speed calculation module, an acceleration calculation module, a gated recurrent unit (GRU), and a linear layer. The action prediction module has four inputs, namely, the joint motion feature H, the 48th frame human skeleton X in the human action sequence sample V, and the 48th frame human skeleton X in the human action sequence sample V. 48 、49th frame human skeleton X 49 , and the 50th frame human skeleton X 50 .

[0079] like Figure 4 As shown in the leftmost column, it represents the first cycle. The input of the speed calculation module is the 49th frame human skeleton X of the human action sequence sample V. 49 and the 50th frame human skeleton The output is S0, which is calculated as follows:

[0080]

[0081] Where S0 represents the speed of the human skeleton in the 50th frame of the human motion sequence sample V.

[0082] The input of the acceleration calculation module is the 48th frame human skeleton X in the human action sequence sample V 48 、49th frame human skeleton X 49 and the 50th frame human skeleton X 50 , the output is C0, and the calculation method of C0 is as follows:

[0083]

[0084] Where C0 represents the acceleration of the human skeleton in the 50th frame of the human motion sequence sample V.

[0085] Next, S0, C0 and X 50 Connect the channels to get I0. The input of the GRU module is I0 and H, and the output is B0 and H1. The input of the linear layer is B0, and the output is F0. The calculation method of F0 is as follows:

[0086] F0=W0·B0+θ0,

[0087] Where W0 is the weight of the linear layer and θ0 is the bias vector of the linear layer.

[0088] The output F0 of the linear layer and the 50th frame human skeleton X in the human action sequence sample V are combined 50 Add together to get the prediction result of the 51st frame of the human skeleton of the human action sequence sample V

[0089] like Figure 4 As shown in the second column, it represents the second cycle, and its structure is consistent with the structure of the first cycle. The input of the second cycle is the predicted result of the human skeleton of the 51st frame of the human action sequence sample V The 50th frame of the human skeleton X in the human action sequence sample V 50 、49th frame human skeleton X 49 And one of the outputs H0 of GRU in the first cycle. The output of the second cycle is the predicted result of the human skeleton of the 52nd frame of the human action sequence sample V

[0090] like Figure 4 The rightmost column shows the 25th cycle, and its structure is consistent with the structure of the 1st cycle. The input of the 25th cycle is the predicted result of the human skeleton of the 74th frame of the human action sequence sample V. Prediction results of the human skeleton in the 73rd frame Prediction results of the human skeleton in the 72nd frame and one of the outputs H of the GRU in the 24th cycle 24 The output of the 25th cycle is the predicted result of the human skeleton of the 75th frame of the human action sequence sample V

[0091] The output of the action prediction module is the calculated sequence to be predicted The expression is as follows:

[0092]

[0093] in, is the calculated matrix representation of the skeleton of the p-th frame of the sequence to be predicted.

[0094] As an embodiment of the present technology, a dual-stream action prediction network is constructed, such as Figure 5 As shown, the input of the network is the historical sequence and future sequence divided by the human action sequence samples, and the output is the calculated sequence to be predicted, that is, the output of the action prediction module The nth prediction sequence sample output by the action prediction module is And there are 32 samples in total. The network loss function L1 is:

[0095]

[0096] Among them, ||·||1 represents the 1-norm, is the true value of the nth prediction sequence.

[0097] The historical sequence and future sequence divided by the human motion sequence samples used for training are input into the two-stream motion prediction network, and the network is trained until convergence.

[0098] The historical motion features extracted by the trained motion feature extraction module are input into the classification module for action classification. The classification module is trained until convergence. The structure of the classification module is as follows: Figure 6 As shown, the input of linear layer 1 is the historical motion feature H extracted by the trained historical feature extraction module. history , the number of neurons in linear layer 1 is 2560, and the output Q1 of linear layer 1 is calculated as follows:

[0099]

[0100] Where W1 is the weight of linear layer 1, θ1 is the bias vector of linear layer 1, is the relu activation function, and its formula is as follows:

[0101]

[0102] x is the input of the relu activation function. When x is greater than or equal to 0, the output of the relu activation function is x. When x is less than 0, the output of the relu activation function is 0.

[0103] The input of linear layer 2 is Q1. The number of neurons in linear layer 2 is 1280. The output Q2 of linear layer 2 is calculated as follows:

[0104]

[0105] in, is the relu activation function, W2 is the weight of linear layer 2, and θ2 is the bias vector of linear layer 2.

[0106] The input of linear layer 3 is Q2. The number of neurons in linear layer 3 is 15. The output Q3 of linear layer 3 is calculated as follows:

[0107]

[0108] Where W3 is the weight of linear layer 3, θ3 is the bias vector of linear layer 3, is the softmax activation function. Q3 represents the probability that the input historical motion features belong to each action category. The total number of action categories is 15, which is the same as the number of neurons in linear layer 3. The action category corresponding to the largest probability value in Q3 is selected as the output of the classification module.

[0109] The historical sequence divided by the human motion sequence samples used for training is input into the trained motion feature extraction module to obtain historical motion feature samples. The historical motion feature samples are then input into the classification module and trained until convergence. The loss function L2 used in training the classification module is:

[0110]

[0111] Among them, the total number of action categories is 15, and the total number of input historical motion feature samples is 32. is the network output of the a-th historical motion feature sample, I a is the expected output of the ath historical motion feature sample, where I a The g-th dimension data is defined as:

[0112]

[0113] Among them, i a is the label value of the a-th historical motion feature sample.

[0114] Combine the trained dual-stream action prediction network with the trained classification module, and add an empty dictionary module to store information to construct a dual-stream information storage network, such as Figure 7 The input of the motion feature extraction module is the historical sequence of human motion sequence samples used for training, and the output is the historical motion feature H history The input of the future feature extraction module is the future sequence divided by the human action sequence samples used for training, and the output is the future motion feature H future ; The input of the classification module is the historical motion feature H history , the output is the action category to which the historical motion feature belongs. The input of the dictionary module is the action category output by the classification module, the historical motion feature H history and future motion features H future; Historical motion features and future motion features are added together and input into the action prediction module to obtain the action prediction result.

[0115] The historical sequence and future sequence of the human motion sequence samples used for training are input into the dual-stream information storage network, and the corresponding relationship between the historical motion features and future motion features of each action is stored in the dictionary module. The structure of the dictionary module is as follows Figure 8 As shown, the keys in the dictionary module are used to represent action categories: key1 represents the first action category, key 15 Used to represent the 15th action category; the value corresponding to each key in the dictionary module is a sub-dictionary: value1 represents the first sub-dictionary, value 15 Represents the 15th sub-dictionary. The key of the first sub-dictionary value1 is used to store the historical motion features of the first action category, and the value of the first sub-dictionary value1 is used to store the future motion features of the first action category: key Used to store the first historical motion feature of the first action category value Used to store the first future motion feature of the first action category key Used to store the 2000th historical motion feature of the first action category value Used to store the 2000th future motion feature of the first action category The 15th sub-dictionary value 15 The key is used to store the historical motion features of the 15th action category, and the value is used to store the future motion features of the 15th action category. Used to store the first historical motion feature of the 15th action category value Used to store the first future motion feature of the 15th action category key Used to store the 2000th historical motion feature of the 15th action category value Used to store the 2000th future motion feature of the 15th action category

[0116] As an embodiment of the present technical solution, a trained motion feature extraction module, a trained classification module, a trained action prediction module and a dictionary module that stores information are combined to construct an action prediction test network. Figure 9 shown.

[0117] As an embodiment of the present technical solution, the historical sequence divided by the human motion sequence sample under test is input into the motion feature extraction module of the motion prediction test network to extract the historical motion features of the human motion sequence sample under test. Figure 9 As shown, the input of the motion feature extraction module is the historical sequence divided by the human motion sequence sample tested, and the output is the historical motion features of the human motion sequence sample tested.

[0118] As an embodiment of the present technical solution, the historical motion features of the human motion sequence samples tested are input into the classification module of the motion prediction test network to obtain the motion category. Figure 9 As shown, the input of the classification module is the historical motion features of the human action sequence sample tested The output is the action category g.

[0119] As an embodiment of the present technical solution, the future motion features corresponding to the historical motion features are searched in the storage dictionary corresponding to the category. The input of the dictionary module is the action category g and the historical motion features of the human action sequence sample tested. Output is the future motion feature to be found

[0120] The way to find the future motion signature is as follows:

[0121] The historical motion features of the human action sequence samples to be tested Vectorization, coming soon The row vectors are connected into a new row vector is a vector of dimension 2560.

[0122] Select the sub-dictionary value corresponding to the action category g in the dictionary module g . Pair dictionary value g The i-th key stores the historical motion features Vectorization, coming soon The row vectors are connected into a new row vector is a vector of dimension 2560.

[0123] Calculate the cosine similarity between the historical motion features of the human action sequence sample under test and the i-th historical motion features stored in action category g. The calculation method is as follows:

[0124]

[0125] Among them, · represents the dot product operation, express The 2-norm of express 2 norm, simi Represents the cosine similarity between the historical motion features of the tested human action sequence sample and the i-th historical motion features stored in action category g.

[0126] Calculate the similarity between all historical motion features stored in the action category g and the historical motion features of the tested human action sequence samples, and obtain {sim1,...,sim i ,...,sim 2000}.

[0127] Find the maximum value of cosine similarity sim j ,sim j =max{sim1,...,sim i ,...,sim 2000 Next, search for the jth key in the sub-dictionary corresponding to action category g. And find the value corresponding to the jth key Will Stored future motion characteristics As the future motion features found

[0128] As an embodiment of the present technical solution, the historical motion features of the tested human motion sequence samples and the found future motion features are added together and input into the motion prediction module of the motion prediction test network to realize motion prediction.

[0129] The beneficial effects of the present invention are as follows:

[0130] This technical solution uses both future and historical motion features for motion prediction. This approach not only provides context for the sequence being predicted but also for the future, helping to improve the accuracy of motion prediction. Long-term human movement is an activity with clear intentions, which can be distinguished by motion categories. Incorporating motion category information into motion prediction facilitates long-term motion prediction.

[0131] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0132] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0133] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0134] Figure 1 Flowchart of a method for predicting human motion according to an embodiment of the present invention;

[0135] Figure 2 Flowchart of extracting historical motion features of a method for predicting human motion in an embodiment of the present invention;

[0136] Figure 3 Flowchart of extracting future motion features of a method for predicting human motion in an embodiment of the present invention;

[0137] Figure 4 A motion prediction module diagram of a human motion prediction method according to an embodiment of the present invention;

[0138] Figure 5 A dual-stream motion prediction network diagram of a human motion prediction method according to an embodiment of the present invention;

[0139] Figure 6 A classification module diagram of a human motion prediction method according to an embodiment of the present invention;

[0140] Figure 7 A dual-stream information storage network diagram of a human motion prediction method according to an embodiment of the present invention;

[0141] Figure 8 A dictionary module diagram of a human motion prediction method according to an embodiment of the present invention;

[0142] Figure 9 A motion prediction test network diagram containing 20 key points for a human motion prediction method according to an embodiment of the present invention

[0143] Figure 10 A simplified structural diagram of a human skeleton including 20 key points in a method for predicting human motion according to an embodiment of the present invention;

[0144] Figure 11 A schematic diagram of a mid-level human skeleton structure in a method for predicting human motion according to an embodiment of the present invention;

[0145] Figure 12 A high-level schematic diagram of the human skeleton structure in a method for predicting human motion according to an embodiment of the present invention. DETAILED DESCRIPTION

[0146] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0147] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. "Multiple" means two or more, unless otherwise specifically defined. Moreover, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0148] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

[0149] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0150] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0152] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A human motion prediction method, characterized in that: include: Step 1: Divide the preset human motion sequence samples into historical sequences, sequences to be predicted, and future sequences, and input the historical sequences and future sequences into a preset motion feature extraction module to extract historical motion features and future motion features; wherein, The motion feature extraction module includes a speed calculation module, an acceleration calculation module, a level division module, a fusion module, a spatiotemporal convolution module, a feature fusion module and a feature cascade module; Step 2: Input historical motion features and future motion features into the preset motion prediction module for training to construct a dual-stream motion prediction network; Step 3: Input the historical sequence and future sequence into the two-stream action prediction network and train the two-stream action prediction network until convergence; Step 4: Through the trained motion feature extraction module, extract historical motion features and input them into the preset classification module for action classification, and train the classification module until convergence; Step 5: Combine the trained dual-stream action prediction network and classification module and add the preset dictionary module to construct a dual-stream information storage network; The dictionary module of the dual-stream information storage network stores the correspondence between the historical motion features and the future motion features of each action; Step 6: Construct an action prediction test network through the trained motion feature extraction module, classification module, action prediction module and stored dictionary module; Step 7: Input the historical motion features into the classification module of the action prediction test network to obtain the action category, and search the storage dictionary corresponding to the action category for the future motion features corresponding to the historical motion features; Step 8: Add the historical motion features of the human motion sequence sample to the found future motion features, and input them into the motion prediction module of the motion prediction test network to achieve motion prediction; Wherein, the step 1 includes: Step 11: Construct a human action sequence sample using several preset human skeleton samples: V=[X1,…,X t ,…,X N ] Among them, V represents the human action sequence sample, t represents the t-th frame number of the human action sequence sample, t∈[1,N], N represents the total number of frames of the human action sequence sample, X t ∈R M×D ,R M×D The matrix representing the human skeleton of the t-th frame of the human action sequence sample V, M represents the number of key points contained in the human skeleton of the t-th frame; D represents the dimension of the vector representation of each key point in the human skeleton of the t-th frame; Step 12: Divide the human action sequence sample V into a history sequence V history , sequence to be predicted V predict and Future Sequence V future ;in, Among them, V history Represents the historical sequence, h represents the time index of the historical sequence, h∈[1,T history ], T history ∈[1,T predict ], T history Represents the total number of frames in the history sequence; X h ∈R M×D , X h represents the matrix of the human skeleton of the hth frame of the historical sequence, X1 represents the matrix of the human skeleton of the first frame of the historical sequence, X Tjistory The matrix representing the human skeleton of the last frame in the history sequence; V predict Represents the sequence to be predicted, p represents the time index of the sequence to be predicted, p∈[T history +1,T history +T predict ], T predict Represents the total number of frames in the sequence to be predicted, T predict ∈[T predict ,T future ];X p ∈R M×D , X p The matrix representing the human skeleton of the pth frame of the sequence to be predicted; The matrix representing the human skeleton of the first frame of the history sequence; The matrix representing the human skeleton of the last frame in the history sequence; V future represents the future sequence, f represents the time index of the historical sequence, f∈[T history +T predict +1,T history +T predict +T future ], T future Represents the total number of frames in the future sequence, T future ∈[T future ,N];X f ∈R M×D , X f The matrix representing the human skeleton of the fth frame of the future sequence; The matrix representing the human skeleton of the first frame of the future sequence, The matrix representing the human skeleton of the last frame in the future sequence, N = T history +T predict +T future ; Step 13: Use the speed calculation module to extract speed features from the historical sequence and determine the historical speed sequence; Among them, V history_v Represents the historical speed sequence, vel1 represents the matrix of the first frame skeleton in the historical speed sequence, The matrix representing the skeleton of the first frame in the future velocity sequence, is the historical speed sequence V history_v The matrix representation of the h-th frame skeleton in, h represents the time index of the historical sequence, X h The matrix representing the human skeleton of the hth frame of the history sequence, X h-1 The matrix representing the human skeleton of the h-1th frame of the history sequence, Represents the Tth in the historical speed sequence history The matrix of the frame skeleton, T history Represents the total number of frames in the history sequence; Step 14: Calculate the historical velocity sequence through the acceleration calculation module to determine the historical acceleration sequence: Among them, V history_a Represents the historical acceleration sequence, acl1 represents the matrix of the first frame skeleton in the historical acceleration sequence, acl h The matrix representing the h-th frame skeleton in the historical acceleration sequence, Represents the Tth acceleration in the historical acceleration sequence history The matrix of the frame skeleton; Step 15: Using a level division module, the historical sequence is divided into low-level historical sequence, intermediate historical sequence and high-level historical sequence; Among them, V hp_1 Represents the low-level historical sequence, V history Represents the historical sequence, V hp_2 Represents the intermediate historical sequence of the human skeleton obtained by inputting the first pooling layer into the historical sequence, mid h ∈R W×W ,R W×D represents the matrix of the h-th frame human skeleton in the intermediate history sequence about the human skeleton, W and D represent the number of rows and columns of the matrix of the h-th frame skeleton in the intermediate history sequence, V hp_3 Represents the advanced historical sequence, V hp_3 It is the high-level history sequence of the human skeleton obtained by inputting the history sequence into the second pooling layer. h ∈R U×D , U and D correspond to the number of rows and columns of the matrix of the h-th frame skeleton in the high-level history sequence, respectively; Step 16: The low-level history sequence, the intermediate history sequence, and the high-level history sequence are sequentially transmitted to the corresponding spatiotemporal convolution module 1, the fusion module 1, the spatiotemporal convolution module 2, the fusion module 2, the spatiotemporal convolution module 3, and the spatiotemporal convolution module 4 to generate corresponding historical fusion features; Step 17: Transfer the future sequence to steps 13-15 to generate the corresponding future fusion features; Step 18: Generate historical motion features and future motion features through the historical fusion features and the future fusion features; Wherein, the step 2 includes: Step 21: Add the historical motion features and the future motion features to obtain the joint motion features; H=H future +H history Among them, H represents the joint motion characteristics; Step 22: Combine the joint motion feature H and the Tth history -2 human skeleton frames No. T history -1 human skeleton and T history Frame human skeleton Input the preset recurrent neural network to calculate the sequence to be predicted; Step 23: Increment the human skeleton of the human action sequence sample V and repeat step 22 to determine the sequence to be predicted: p∈[T history +1,T history +T predict ], in, is the matrix representation of the skeleton of the p-th frame of the sequence to be predicted; Step 24: Construct a dual-stream motion prediction network through human motion sequence samples, motion feature extraction module and motion prediction module; wherein, The loss function L1 of the dual-stream action prediction network is: Among them, ||||1 represents the 1 norm, is the true value of the nth prediction sequence, is the output of the action prediction module, Represents the nth predicted sequence sample output by the action prediction module.

2. A human motion prediction method according to claim 1, characterized in that: The step 17 comprises: Step 170: Transmit the future sequence to the speed calculation module to obtain the corresponding future speed sequence; Step 171: transmitting the future velocity sequence to an acceleration calculation module to calculate a corresponding future acceleration sequence; Step 172: Transmit the future acceleration sequence to a level division module to determine a low-level future sequence, a mid-level future sequence, and a high-level future sequence; Step 173: The low-level future sequence, the intermediate future sequence, and the high-level future sequence are sequentially transmitted to the corresponding spatiotemporal convolution module 1, the fusion module 1, the spatiotemporal convolution module 2, the fusion module 2, the spatiotemporal convolution module 3, and the spatiotemporal convolution module 4 to generate corresponding future fusion features; wherein, The fusion module 1 includes a low-level convolution layer 1, a low-level linear layer 1, a low-level aggregation-related feature layer 1, a low-level update feature layer 1, a low-level feature fusion layer 1, a medium-level convolution layer 1, a medium-level linear layer 1, a medium-level aggregation-related feature layer 1, a medium-level update feature layer 1, a medium-level feature fusion layer 1, a high-level convolution layer 1, a high-level linear layer 1, a high-level aggregation-related feature layer 1, a high-level update feature layer 1 and a high-level feature fusion layer 1.

3. A human motion prediction method according to claim 1, characterized in that: in, The step 16 comprises: Step 160: Set the low-level history sequence V hp_1 Input low-level spatiotemporal convolution module 1, output F hp_11 ; F hp_11 Representative and low-level history sequence V hp_1 The output of the corresponding low-level spatiotemporal convolution module 1; Step 161: Intermediate History Sequence V hp_2 Input mid-level spatiotemporal convolution module 1, output F hp_12 ; F hp_12 Representative and Intermediate History Sequence V hp_2 The output of the corresponding mid-level spatiotemporal convolution module 1; Step 162: Advanced History Sequence V hp_3 Input high-level spatiotemporal convolution module 1, output F hp_13 ; F hp_13 Representative and Advanced History Sequence V hp_3 The output of the corresponding high-level spatiotemporal convolution module 1; Step 163: F hp_11 、F hp_12 、F hp_13 Input fusion module 1, output the features of all key points in the updated low-level human skeleton, the features of all key points in the updated mid-level human skeleton, and the features of all key points in the updated high-level human skeleton; Step 164: Through H 11 、H 12 and H 13 , determine and generate the corresponding historical fusion features.

4. A human motion prediction method according to claim 3, characterized in that: The step 163 further includes: Step A1: F hp_11 Input low-level convolution layer 1, output G 11 ; G 11 Representative and low-level history sequence V hp_1 The output of the corresponding low-level convolutional layer 1; Step A2: G 11 Input low-level linear layer 1, output is F 11 ; F 11 Represents the features of all key points in the extracted low-level human skeleton; Represents the features of the i-th key point in the extracted low-level human skeleton, Represents the features of the jth key point in the extracted low-level human skeleton; Step A3: For the i-th key point in the low-level human skeleton, the low-level clustered related feature layer 1 aggregates the features of the remaining M-1 key points to the i-th key point, and then passes through the three fully connected layers of the low-level clustered related feature layer 1 to determine the corresponding key point column vector; Among them, [,] represents the channel connection function, f 11 () represents the 3-layer fully connected layer function of the low-level aggregation related feature layer 1, r 11_i is a column vector about the i-th key point, i = 1, 2, ..., M, M represents the total number of key points; Step A4: Execute step A3 for all key points in the low-level human skeleton to obtain the two-dimensional tensor of the key points, R 11 The formula is as follows: R 11 =(r 11_1 ,...,r 11_i ,...,r 11_M ),i∈[1,M], Among them, R 11 It is a two-dimensional tensor consisting of M column vectors; Step A5: R 11 Input low-level updated feature layer 1, output keypoint feature H 11 , the low-level update feature layer 1 is used to update the features. The low-level update feature layer 1 consists of 3 fully connected layers. The formula for updating the features is as follows: H 11 =g 11 ([F 11 ,R 11 ]) Among them, [,] means according to the channel connection function, g 11 () represents the function of updating the low-level feature layer to obtain a 3-layer fully connected layer, H 11 are the features of all key points in the updated low-level human skeleton; Step A6: F hp_12 Input mid-level convolution layer 1, output G 12 ; G 12 Representative and Intermediate History Sequence V hp_2 The output of the corresponding mid-level convolutional layer 1; Step A7: G 12 Input mid-level linear layer 1, output is F 12 ; F 12 Represents the features of all key points in the extracted mid-level human skeleton; represents the feature of the i-th key point in the extracted mid-level human skeleton, Represents the features of the jth key point in the extracted mid-level human skeleton; Step A8: For the i-th key point in the mid-level human skeleton, the mid-level clustered related feature layer 1 aggregates the features of the remaining M-1 key points onto the i-th key point, and then passes through the three fully connected layers of the mid-level clustered related feature layer 1 to determine the corresponding key point column vector; Among them, [,] represents the channel connection function, f 12 () represents the 3-layer fully connected layer function of the mid-level aggregation related feature layer 1, r 12_i is a column vector about the i-th key point, i = 1, 2, ..., M, M represents the total number of key points; Step A9: Execute step A8 for all key points in the mid-level human skeleton to obtain a two-dimensional tensor of the key points of the mid-level human skeleton, R 12 The formula is as follows: R 12 =(r 12_1 ,...,r 12_i ,...,r 12_M ),i∈[1,M], Among them, R 12 It is a two-dimensional tensor consisting of M column vectors; Step A10: R 12 Input mid-level update feature layer 1, output key point feature H 12 , the mid-level update feature layer 1 is used to update features. The mid-level update feature layer 1 consists of 3 fully connected layers. The formula for updating features is as follows: H 12 =g 12 ([F 12 ,R 12 ]) Among them, [,] means according to the channel connection function, g 12 () represents the function of updating the low-level feature layer to obtain a 3-layer fully connected layer, H 12 are the features of all key points in the updated low-level human skeleton; Step A11: F hp_12 Input mid-level convolution layer 1, output G 12 ; G 12 Representative and Intermediate History Sequence V hp_2 The output of the corresponding mid-level convolutional layer 1; Step A12: G 12 Input mid-level linear layer 1, output is F 12 ; F 12 Represents the features of all key points in the extracted mid-level human skeleton; represents the feature of the i-th key point in the extracted mid-level human skeleton, Represents the features of the jth key point in the extracted mid-level human skeleton; Step A13: For the i-th key point in the mid-level human skeleton, the mid-level clustered related feature layer 1 aggregates the features of the remaining M-1 key points onto the i-th key point, and then passes through the three fully connected layers of the mid-level clustered related feature layer 1 to determine the corresponding key point column vector; Among them, [,] represents the channel connection function, f 12 () represents the 3-layer fully connected layer function of the mid-level aggregation related feature layer 1, r 12_i is a column vector about the i-th key point, i = 1, 2, ..., M, M represents the total number of key points; Step A14: Execute step A8 for all key points in the mid-level human skeleton to obtain a two-dimensional tensor of the key points of the mid-level human skeleton, R 12 The formula is as follows: Among them, R 12 It is a two-dimensional tensor consisting of M column vectors; Step A15: R 12 Input mid-level update feature layer 1, output key point feature H 12 , the mid-level update feature layer 1 is used to update features. The mid-level update feature layer 1 consists of 3 fully connected layers. The formula for updating features is as follows: H 12 =g 12 ([F 12 ,R 12 ]) Among them, [,] means according to the channel connection function, g 12 () represents the function of updating the feature layer at the middle level to obtain a 3-layer fully connected layer, H 12 are the features of all key points in the updated mid-level human skeleton; Step A16: F hp_13 Input high-level convolution layer 1, output G 13 ; G 13 Representative and Advanced History Sequence V hp_3 The output of the corresponding high-level convolutional layer 1; Step A17: G 13 Input high-level linear layer 1, output is F 13 ; F 13 Represents the features of all key points in the extracted high-level human skeleton; Represents the features of the i-th key point in the extracted high-level human skeleton, Represents the features of the jth key point in the extracted high-level human skeleton; Step A18: For the i-th key point in the high-level human skeleton, the high-level aggregation-related feature layer 1 aggregates the features of the remaining M-1 key points onto the i-th key point, and then passes through the three fully connected layers of the high-level aggregation-related feature layer 1 to determine the corresponding key point column vector; Among them, [,] represents the channel connection function, f 13 () represents the 3-layer fully connected layer function of high-level aggregation related feature layer 1, r 13_i is a column vector about the i-th key point, i = 1, 2, ..., M, M represents the total number of key points; Step A19: Execute step A8 for all key points in the high-level human skeleton to obtain a two-dimensional tensor of key points of the high-level human skeleton, R 13 The formula is as follows: R 13 =(r 13_1 ,...,r 13_i ,...,r 13_M ),i∈[1,M], Among them, R 13 It is a two-dimensional tensor consisting of M column vectors; Step A15: R 13 Input high-level updated feature layer 1, output keypoint feature H 13 , the high-level update feature layer 1 is used to update the features. The high-level update feature layer 1 consists of 3 layers of fully connected layers. The formula for updating features is as follows: H 13 =g 13 ([F 13 ,R 13 ]) Among them, [,] means according to the channel connection function, g 13 () represents the function of obtaining 3 fully connected layers based on the high-level updated feature layer, H 13 are the features of all key points in the updated mid-level human skeleton.

5. A human motion prediction method according to claim 3, characterized in that: The step 164 includes: Step S10: H 11 and H 12 Input low-level feature fusion layer 1, output S hp_11 ; Step S10 includes: In the low-level feature fusion layer 1, H 11 and H 12 First pass through a softmax function, the formula is as follows: Among them, T represents transpose, A 12 represents the connection weight matrix, H 11 is the feature of all key points in the updated low-level human skeleton, H 12 For the features of all key points in the updated mid-level human skeleton, the softmax function is used to normalize the value of each element of the input vector to between (0, 1). The calculation formula for each element is as follows: Among them, e represents the natural base, K represents the number of elements contained in the input vector, and a l Represents the lth element of the input vector, a k Represents the kth element of the input vector, k=1,2,…,K,S l Represents the normalized value of the lth element of the input vector; Step S11: H 12 Fusion to H 11 The formula is expressed as follows: S hp_11 =A 12 ·H 12 *ω+H 11 Among them, S hp_11 Indicates H 12 and H 11 The fused features are also the output of the low-level feature fusion layer 1, and ω represents the weight; Step S12: 11 、H 12 and H 13 Input mid-level feature fusion layer 1, output is S hp_12 ; Among them, in the mid-level feature fusion layer 1, H 11 and H 12 First pass through a softmax function, the formula is as follows: Among them, T represents transpose, A 21 represents the connection weight; Then, H 13 and H 12 After a softmax function, the formula is expressed as follows: Among them, T represents transpose, A 23 represents the connection weight, H 13 The features of all key points in the updated advanced human skeleton; Next, H 11 and H 13 Fusion to H 12 The formula is expressed as follows: S hp_12 =(A 21 ·H 11 +A 23 ·H 13 )*θ+H 12 , Among them, S hp_12 Indicates H 11 、H 12 and H 13 The fused features are also the output of the mid-level feature fusion layer 1, and θ represents the weight; Step S13: H 13 and H 12 Input high-level feature fusion layer 1, output is S hp_13 ; Among them, in the high-level feature fusion layer 1, H 13 and H 12 First pass through a softmax function, the formula is as follows: Among them, T represents transpose, A 32 represents the connection weight matrix, Next, H 12 Fusion to H 13 The formula is expressed as follows: S hp_13 =A 32 ·H 12 *γ+H 13 , Among them, S hp_13 Indicates H 12 and H 13 The fused features are also the output of the high-level feature fusion layer 1, and γ represents the weight; Step S14: hp_11 Input low-level spatiotemporal convolution module 2, output F hp_21 ; Among them, the low-level spatiotemporal convolution module 2 includes 1 spatial graph convolution layer and 1 temporal convolution layer; the spatial graph convolution layer has 128 convolution kernels, each of which is 1×1 in size; the temporal convolution layer has 64 convolution kernels, each of which is 5×1 in size; Step S15: hp_12 Input mid-level spatiotemporal convolution module 2, output F hp_22 ; Among them, the structure of the mid-level spatiotemporal convolution module 2 is consistent with the structure of the low-level spatiotemporal convolution module 2, and its input is S hp_12 , the output is F hp_22 The structure of the high-level spatiotemporal convolution module 2 is consistent with that of the low-level spatiotemporal convolution module 2, and its input is S hp_13 , the output is F hp_23 ; Step S16: S hp_21 Input low-level spatiotemporal convolution module 3, output S hp_31 ; Among them, the low-level spatiotemporal convolution module 3 includes 1 spatial graph convolution layer and 1 temporal convolution layer; the spatial graph convolution layer has 256 convolution kernels, each of which is 1×1 in size; the temporal convolution layer has 128 convolution kernels, each of which is 5×1 in size; Step S17: S hp_22 Input mid-level spatiotemporal convolution module 3, output S hp_32 ; Wherein, the structure of the mid-level spatiotemporal convolution module 3 is consistent with the structure of the low-level spatiotemporal convolution module 3; Step S18: S hp_23 Input mid-level spatiotemporal convolution module 3, output S hp_33 ; Wherein, the structure of the high-level spatiotemporal convolution module 3 is consistent with the structure of the low-level spatiotemporal convolution module 3; Step S19: S hp_31 Input low-level spatiotemporal convolution module 4, output S hp_41 ; Among them, the low-level spatiotemporal convolution module 4 includes 1 spatial graph convolution layer and 1 temporal convolution layer; the spatial graph convolution layer has 512 convolution kernels, each of which is 1×1 in size; the temporal convolution layer has 256 convolution kernels, each of which is 5×1 in size; Step S20: S hp_32 Input mid-level spatiotemporal convolution module 4, output S hp_42 ; Wherein, the structure of the mid-level spatiotemporal convolution module 4 is consistent with the structure of the low-level spatiotemporal convolution module 4; Step S21: S hp_33 Input high-level spatiotemporal convolution module 4, output S hp_43 ; Wherein, the structure of the high-level spatiotemporal convolution module 4 is consistent with the structure of the low-level spatiotemporal convolution module 3; Step S22: S hp_41 、、S hp_42 and S hp_43 Input the preset feature fusion module and output S hp ; Step S23: Feature fusion module performs S hp_42 Take samples, get And Give weight β1; for S hp_43 Perform 5 upsamplings to get And Give weight β2; hp_41 、 Add, S hp The expression is as follows: Among them, β1 and β2 correspond to weight; Step S24: S hp 、S hv and S ha Input the input of the feature cascade module, and the output is H history Among them, H history By S hp 、S hv and S ha According to the channel splicing, H history Represents the characteristics of historical movements; Step S25: Set the future sequence V future Input motion feature extraction module to obtain future motion features H future .

6. A human motion prediction method according to claim 1, characterized in that: in, The step 22 further includes: Step 220: The Tth history -1 human skeleton and T history Frame human skeleton Input the preset recurrent neural network to calculate the speed S0 of the human skeleton: Among them, S0 represents the Tth human action sequence sample V history Frame speed of human skeleton; Step 221: The Tth history -2 human skeleton frames No. T history -1 human skeleton and T history Frame human skeleton Transmitted to the acceleration calculation module to calculate the acceleration of the human skeleton: Among them, C0 represents the Tth human action sequence sample V history Frame acceleration of the human skeleton; Step 222: The velocity S0, acceleration C0 and the first history Frame human skeleton According to the channel connection, the connection result I0 is determined, and the connection result I0 and the joint motion feature H are transmitted to the gated recurrent unit GRU module to output B0 and H0; Step 223: B0 is transferred to the linear layer and output F0. F0 is calculated as follows: F0=W0·B0+θ0, Where W0 is the weight of the linear layer and θ0 is the bias vector of the linear layer; Step 224: Combine the output F0 of the linear layer and the Tth history Frame human skeleton Add together and get the Tth human action sequence sample V history +1 frame human skeleton prediction result Step 225: The Tth history +1 frame human skeleton prediction result The Tth human action sequence sample V history Frame human skeleton No. T history -1 human skeleton And one of the outputs H0 of GRU in the first cycle is input into the second cycle, and the output is the Tth human action sequence sample V history +2 frames of human skeleton prediction results 7. A human motion prediction method according to claim 1, characterized in that: The step 4 comprises: Step 40: Extract the historical motion features H of the trained motion feature extraction module history , the historical motion feature H history Transmitted to the linear layer 1 of the classification module, output result Q1; Where W1 is the weight of linear layer 1, θ1 is the bias vector of linear layer 1, is the relu activation function, relu stands for linear rectification function, x is the input of the relu activation function. When x is greater than or equal to 0, the output of the relu activation function is x. When x is less than 0, the output of the relu activation function is 0. Q1 represents the output result of linear layer 1. Step 41: Input the output Q1 of linear layer 1 into linear layer 2, and output the result Q2; in, is the relu activation function, W2 is the weight of linear layer 2, and θ2 is the bias vector of linear layer 2; Step 42: Input the output Q1 of linear layer 1 into linear layer 2, and output the result Q2; Where W3 is the weight of linear layer 3, θ3 is the bias vector of linear layer 3, represents the softmax activation function, Q3 represents the probability that the input historical motion feature belongs to each action category; Step 43: Select the action category corresponding to the maximum probability value in Q3, and use the action category as the output of the classification module; Step 44: Input the historical sequence into the trained motion feature extraction module to determine the historical motion feature samples; Step 45: Input the historical motion feature samples into the classification module, and train the classification module until convergence based on the loss function L2 preset in the classification module; wherein the loss function L2 used in training the classification module is: Among them, G is the total number of action categories, A is the total number of input historical motion feature samples, is the probability of each action category of the a-th historical motion feature sample, Represents the network output of the a-th historical motion feature sample of the g-th dimension data, I a is the expected output of the ath historical motion feature sample, where I a The g-th dimension data is defined as: Among them, i a is the label value of the a-th historical motion feature sample.

8. A human motion prediction method according to claim 1, characterized in that: The step 5 includes: step 5: combining the trained dual-stream action prediction network and the classification module and adding a preset dictionary module to construct a dual-stream information storage network; wherein, The dictionary module of the dual-stream information storage network stores the correspondence between the historical motion features and the future motion features of each action; Step 6: Construct an action prediction test network through the trained motion feature extraction module, classification module, action prediction module and stored dictionary module; Step 7: Input the historical motion features into the classification module of the action prediction test network to obtain the action category, and search the storage dictionary corresponding to the action category for the future motion features corresponding to the historical motion features; Step 8: Add the historical motion features of the human motion sequence sample to the found future motion features, and input them into the motion prediction module of the motion prediction test network to achieve motion prediction; Step 50: The action category and historical motion features H output by the classification module history and future motion features H future Input dictionary module; Step 51: Input the historical sequence and the future sequence into the dual-stream information storage network, and store the corresponding relationship between the historical motion features and the future motion features of each action in the dictionary module; wherein, The keys in the dictionary module are used to represent action categories, and the value corresponding to each key in the dictionary module is a sub-dictionary.

9. A human motion prediction method according to claim 1, characterized in that: Step 7 and step 8 include: Step 60: Combine the trained motion feature extraction module, classification module, motion prediction module, and dictionary module that have stored information to construct a motion prediction test network; Step 61: Input the historical sequence into the motion feature extraction module of the action prediction test network, extract the historical motion features of the human motion sequence sample tested, and output the historical motion features of the human motion sequence sample tested. Step 62: The historical motion features of the human action sequence samples tested Input the classification module of the action prediction test network to obtain the action category g; Step 63: Combine the action category g and historical motion features Input dictionary module to find corresponding future motion features Wherein, the step 63 includes: Step 630: The historical motion features of the human motion sequence sample to be tested Vectorization, coming soon The row vectors are connected into a new row vector Step 631: Select the sub-dictionary value corresponding to the action category g in the dictionary module g , for the dictionary value g The i-th key stores the historical motion features Vectorization, coming soon The row vectors are connected into a new row vector Step 632: Calculate the cosine similarity between the historical motion feature of the human motion sequence sample under test and the i-th historical motion feature stored for the motion category g. The calculation method is as follows: Among them, · represents the dot product operation, express The 2-norm of express The 2-norm of sim i Represents the cosine similarity between the historical motion features of the human action sequence sample being tested and the i-th historical motion features stored for action category g; Step 633: Calculate the similarity between all historical motion features stored in the action category g and the historical motion features of the tested human action sequence sample, and obtain {sim1,...,sim i ,...,sim E }; Step 634: Find the maximum value of cosine similarity sim j ,sim j =max{sim1,...,sim i ,...,sim E }, then, look for the jth key in the sub-dictionary corresponding to action category g And find the value corresponding to the jth key Will Stored future motion characteristics As the future motion features found 10. The method for predicting human motion based on continuous monitoring according to claim 5, wherein: The steps S15 and S16 also include: The low-level history sequence, the intermediate history sequence and the high-level history sequence are sequentially transmitted to the corresponding spatiotemporal convolution module 1, the fusion module 1 and the spatiotemporal convolution module 2, and the low-level convolution result F is output. hp_21 , intermediate convolution result F hp_22 And the advanced convolution result F hp_23 ; Step S151: The low-level convolution result F hp_21 Input the low-level convolution layer 2 of the fusion module 2, and output the result G 21 , the result G 21 Input low-level linear layer 2, output result F 21 , the result F 21 Input low-level aggregation related feature layer 2, output result H 21 , the result H 21 Input low-level feature fusion layer 2 and determine the result S hp_21 ; Step S152: The intermediate convolution result F hp_22 Input the mid-level convolution layer 2 of the fusion module 2, and output the result G 22 , the result G 22 Input mid-level linear layer 2, output result F 22 , the result F 22 Input mid-level aggregation related feature layer 2, output result H 22 , the result H 22 Input mid-level feature fusion layer 2 and determine the result S hp_22 ; Step S153: The advanced convolution result F hp_23 Input the high-level convolution layer 2 of the fusion module 2, and output the result G 23 , the result G 23 Input high-level linear layer 2, output result F 23 , the result F 23 Input high-level aggregation related feature layer 2, output result H 23 , the result H 23 Input high-level feature fusion layer 2 and determine the result S hp_23 .

Images