A motion prediction method based on continuous monitoring

Abstract

The present invention provides a motion prediction method based on continuous monitoring, comprising: dividing a preset human motion sequence sample into a historical sequence, a sequence to be predicted, and a future sequence, and inputting the historical sequence and the future sequence into a preset motion feature extraction module to extract historical motion features and future motion features; adding the historical motion features and the future motion features, and then inputting the added historical motion features into a preset motion prediction module for training to construct a dual-stream motion prediction network; inputting the historical sequence and the future sequence into the dual-stream motion prediction network, and training the dual-stream motion prediction network and the training classification module until convergence; adding the dual-stream motion prediction network and the classification module into a dictionary module to construct a dual-stream information storage network; combining the motion feature extraction module, the classification module, the motion prediction module, and the dictionary module to construct a self-updating motion prediction model; and inputting the historical sequence divided by the human motion sequence sample into the self-updating motion prediction model to determine the motion prediction result.

Description

Technical Field

[0001] The present invention relates to the fields of big data centers and motion prediction technology, and in particular to a motion prediction method based on continuous monitoring. Background Art

[0002] Human motion prediction is currently a key topic in computer vision. It has widespread applications in areas such as human-computer interaction and intelligent video surveillance. With the rapid development of electronic devices like the Microsoft Kinect and the rapid advancement of various hardware, human skeleton-based motion prediction methods have attracted increasing attention. Compared to video-based methods, skeleton-based methods are more robust and unaffected by factors such as lighting and background color.

[0003] Most existing methods extract historical motion features from the initial stages of human motion sequences and analyze the body's inertial motion to predict human motion. These methods neglect the study of human motion intentions, resulting in better performance in short-term motion prediction but poorer results in long-term motion prediction. Furthermore, these methods are more suitable for motion prediction in limited-length video surveillance and are not applicable to continuous video surveillance.

[0004] Therefore, to address the above motion prediction algorithm problems, a motion prediction method in continuous monitoring is proposed.

[0005] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0007] Figure 1 This is a flow chart of a method for motion prediction based on continuous monitoring in an embodiment of the present invention;

[0008] Figure 2 Flowchart of a process for extracting future motion features in a motion prediction method based on continuous monitoring in an embodiment of the present invention;

[0009] Figure 3 This is a flow chart of an action prediction module of an action prediction method based on continuous monitoring in an embodiment of the present invention;

[0010] Figure 4 1 is a dual-stream action prediction network flow chart of an action prediction method based on continuous monitoring in an embodiment of the present invention;

[0011] Figure 5A classification module diagram of an action prediction method based on continuous monitoring in an embodiment of the present invention;

[0012] Figure 6 This is a dual-stream information storage network flow chart of an action prediction method based on continuous monitoring in an embodiment of the present invention;

[0013] Figure 7 Flowchart of a dictionary module of a method for predicting an action based on continuous monitoring according to an embodiment of the present invention;

[0014] Figure 8 Flowchart of a self-updating motion prediction model of a motion prediction method based on continuous monitoring in an embodiment of the present invention;

[0015] Figure 9 A human skeleton flow chart of 20 key points of a motion prediction method based on continuous monitoring in an embodiment of the present invention;

[0016] Figure 10 A flow chart of a mid-level human skeleton in a method for motion prediction based on continuous monitoring according to an embodiment of the present invention;

[0017] Figure 11 This is a high-level human skeleton flow chart of a motion prediction method based on continuous monitoring in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0019] It should be noted that when a component is referred to as being “fixed to” or “disposed on” another component, it can be directly on the other component or indirectly on the other component. When a component is referred to as being “connected to” another component, it can be directly or indirectly connected to the other component.

[0020] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the present invention.

[0021] In addition, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. "Multiple" means two or more, unless otherwise specifically limited. Moreover, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or apparatus that includes a list of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0022] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

[0023] As an embodiment of the present technical solution, a human motion sequence sample includes 100 frames of human skeletons. Taking any human motion sequence sample V as an example:

[0024] V=[X1,...,X t ,...,X 100 ],t∈[1,100],

[0025] Among them, t represents the time index, and the human action sequence sample V has a total of 100 frames. is the matrix representation of the human skeleton of the t-th frame of the human action sequence sample V. The number of rows and columns of the matrix representation of the human skeleton of this frame are 20 and 60 respectively. R indicates that the matrix is a real matrix. The human skeleton of this frame is represented by 20 key points, and each key point is represented by a 60-dimensional vector. The human skeleton of this frame is as follows Figure 9 shown.

[0026] The human motion sequence sample V is divided into a historical sequence, a sequence to be predicted, and a future sequence. The historical sequence has 50 frames, the sequence to be predicted has 25 frames, and the future sequence has 25 frames. At this time, the human motion sequence sample can be represented as a set consisting of a historical sequence, a sequence to be predicted, and a future sequence, which can be expressed as follows:

[0027] V={V history ,V predict ,V future},

[0028] Among them, V history Represents the historical sequence, V predict represents the sequence to be predicted, Vfuture represents the future sequence, the historical sequence V history The formula is as follows:

[0029] V history =[X1,...,X h ,...,X 50 ], h∈[1,50],

[0030] Where h represents the time index and the history sequence has 50 frames. The history sequence consists of the human skeletons from the first frame to the 50th frame of the human action sequence sample V. For historical sequence V history The matrix representation of the human skeleton of the hth frame; the sequence to be predicted V predict The formula is as follows:

[0031] V predict =[X 51 ,...,X p ,...,X 75 ],p∈[51,75],

[0032] Among them, p represents the time index, and the sequence to be predicted has a total of 25 frames. The sequence to be predicted consists of the human skeleton from the 51st frame to the 75th frame of the human action sequence sample V. is the sequence to be predicted V predict The matrix representation of the human skeleton of the p-th frame.

[0033] Future Sequence V future The formula is as follows:

[0034] V future =[X 76 ,…,X f ,…,X 100 ],f∈[76,100],

[0035] Where, f represents the time index, and the future sequence has a total of 25 frames. The future sequence consists of the human skeletons from the 76th frame to the 100th frame of the human action sequence sample V. is the sequence to be predicted V predict The matrix representation of the human skeleton of the f-th frame.

[0036] As an embodiment of the present technical solution, the historical sequence divided by the human motion sequence samples is input into the motion feature extraction module to extract historical motion features, and the future sequence divided by the human motion sequence samples is input into the motion feature extraction module to extract future motion features. The motion feature extraction module includes: a speed calculation module, an acceleration calculation module, a level division module, a fusion module 1, a fusion module 2, a low-level spatiotemporal convolution module 1, a low-level spatiotemporal convolution module 2, a low-level spatiotemporal convolution module 3, a low-level spatiotemporal convolution module 4, a medium-level spatiotemporal convolution module 1, a medium-level spatiotemporal convolution module 2, a medium-level spatiotemporal convolution module 3, a medium-level spatiotemporal convolution module 4, a high-level spatiotemporal convolution module 1, a high-level spatiotemporal convolution module 2, a high-level spatiotemporal convolution module 3, a high-level spatiotemporal convolution module 4, a feature fusion module and a feature cascade module.

[0037] As an embodiment of the present technical solution, the process of inputting the historical sequence divided by the human body action sequence sample into the motion feature extraction module to extract the historical motion features is as follows: Figure 1 As shown, the input of the speed calculation module is the historical sequence V divided by the human motion sequence samples history , the output is the historical speed sequence V history_v . V history_v The formula is as follows:

[0038] V history_v =[vel1,...,vel h ,...,vel 50 ],h∈[1,50], where is the historical speed sequence V history_v The matrix expression of the skeleton of the hth frame in vel h is calculated as follows:

[0039]

[0040] The input of the acceleration calculation module is the historical velocity sequence V history_v , the output is the historical acceleration sequence V history_a . V history_a The formula is as follows:

[0041] V history_a =[acl1,...,acl h ,...,acl 50 ],h∈[1,50], where is the historical acceleration sequence V history_a The matrix expression of the skeleton of the hth frame in acl. h is calculated as follows:

[0042]

[0043] like Figure 1 As shown in the leftmost column, the input of level division module 1 is V history , which has three outputs, namely V hp_1 、V hp_2 and V hp_3 . V hp_1 Represents the historical sequence of low-level human skeleton, V hp_2 Represents the historical sequence of the middle-level human skeleton, V hp_3 Represents the historical sequence of the human skeleton at a high level. The level division module consists of pooling layer 1 and pooling layer 2. Pooling layer 1 consists of 10 average pooling layers with a pooling size of 2. Pooling layer 2 consists of 5 average pooling layers with a pooling size of 4. V hp_1 =V history , which means that the historical sequence is regarded as the historical sequence of low-level human skeleton. Historical sequence V history Input to pooling layer 1 to obtain the historical sequence V of the mid-level human skeleton hp_2 . V hp_2 The expression is as follows:

[0044] V hp_2 =[mid1,...,mid h ,...,mid 50 ],h∈[1,50],

[0045] in, is the matrix representation of the h-th frame skeleton in the history sequence of the middle-level human skeleton. The human skeleton of this frame is as follows Figure 10 As shown. Historical sequence V history Input to pooling layer 2 to obtain the historical sequence V of the high-level human skeleton hp_3 . V hp_3 The expression is as follows:

[0046] V hp_3 =[high1,...,high h ,...,high 50 ],h∈[1,50], where is the matrix representation of the h-th frame skeleton in the history sequence of the human skeleton of a higher level person. The human skeleton of this frame is as follows Figure 11 shown.

[0047] The input of the low-level spatiotemporal convolution module 1 is V hp_1 , the output is F hp_11The low-level spatiotemporal convolution module 1 contains 1 spatial graph convolution layer and 1 temporal convolution layer. The spatial graph convolution layer has 64 convolution kernels, each of which is 1×1 in size. The temporal convolution layer has 32 convolution kernels, each of which is 5×1 in size. The structure of the mid-level spatiotemporal convolution module 1 is consistent with that of the low-level spatiotemporal convolution module 1, and its input is V hp_2 , the output is F hp_12 The structure of the high-level spatiotemporal convolution module 1 is consistent with that of the low-level spatiotemporal convolution module 1, and its input is V hp_3 , the output is F hp_13 .

[0048] Fusion module 1 includes low-level convolution layer 1, low-level linear layer 1, low-level aggregation-related feature layer 1, low-level update feature layer 1, low-level feature fusion layer 1, medium-level convolution layer 1, medium-level linear layer 1, medium-level aggregation-related feature layer 1, medium-level update feature layer 1, medium-level feature fusion layer 1, high-level convolution layer 1, high-level linear layer 1, high-level aggregation-related feature layer 1, high-level update feature layer 1 and high-level feature fusion layer 1.

[0049] The low-level convolution layer 1 contains 32 convolution kernels, each of which is 5×1 in size. The input of the low-level convolution layer 1 is F hp_11 , the output is G 11 The low-level linear layer 1 consists of three fully connected layers, whose input is G 11 , the output is F 11 . F 11 Represents the features of all key points in the extracted low-level human skeleton. Represents the features of the i-th key point in the extracted low-level human skeleton, Represents the features of the jth key point in the extracted low-level human skeleton. For the i-th key point in the low-level human skeleton, the low-level aggregation related feature layer 1 aggregates the features of the remaining 19 key points to the i-th key point, and then passes through 3 layers of fully connected layers. Its formula is expressed as follows:

[0050]

[0051] Among them, [·,·] means connection according to channel, f 11 (·) represents 3 fully connected layers, r 11_i is a column vector. After performing the above operations on all key points in the low-level human skeleton, we get R 11 . R 11 The formula is as follows:

[0052] R 11 =(r 11_1 ,...,r 11_i,...,r 11_20 ), i∈[1,20],

[0053] This formula represents R 11 It is a two-dimensional tensor consisting of 20 column vectors. The input of low-level update feature layer 1 is R 11 , the output is H 11 The low-level update feature layer 1 is used to update features. It consists of three fully connected layers. The feature update formula is expressed as follows:

[0054] H 11 =g 11 ([F 11 ,R 11 ]),

[0055] Among them, [·,·] means connection according to channel, g 11 (·) represents 3 fully connected layers, H 11 are the features of all key points in the updated low-level human skeleton.

[0056] The structure of the mid-level convolutional layer 1 is the same as that of the low-level convolutional layer 1, and its input is F hp_12 , the output is G 12 The structure of the middle-level linear layer 1 is the same as that of the low-level linear layer 1, and its input is G 12 , the output is F 12 The structure of the mid-level clustering related feature layer 1 is consistent with that of the low-level clustering related feature layer 1, and its input is F 12 , the output is R 12 The structure of the mid-level update feature layer 1 is consistent with that of the low-level update feature layer 1, and its input is R 12 , the output is H 12 . H 12 are the features of all key points in the updated mid-level human skeleton.

[0057] The structure of the high-level convolutional layer 1 is the same as that of the low-level convolutional layer 1, and its input is F hp_13 , the output is G 13 The structure of the high-level linear layer 1 is the same as that of the low-level linear layer 1, and its input is G 13 , the output is F 13 The structure of the high-level aggregation-related feature layer 1 is consistent with that of the low-level aggregation-related feature layer 1, and its input is F 13 , the output is R 13 The structure of the high-level update feature layer 1 is consistent with that of the low-level update feature layer 1, and its input is R 13 , the output is H 13 . H 13are the features of all key points in the updated mid-level human skeleton.

[0058] The input of low-level feature fusion layer 1 is H 11 and H 12 , the output is S hp_11 In the low-level feature fusion layer 1, H 11 and H 12 First pass through a softmax function, the formula is as follows:

[0059]

[0060] Among them, T represents transpose, A 12 Represents the connection weight matrix. The softmax function is used to normalize the value of each element of the input vector to between (0,1). The calculation formula for each element is as follows:

[0061]

[0062] Among them, e represents the natural base, K represents the number of elements contained in the input vector, and a l Represents the lth element of the input vector, a k represents the kth element of the input vector, S l represents the result of normalizing the value of the lth element of the input vector. 12 Fusion to H 11 The formula is expressed as follows:

[0063] S hp_11 =A 12 ·H 12 *0.3+H 11 ,

[0064] Among them, S hp_11 Indicates H 12 and H 11 The fused features are also the output of the low-level feature fusion layer 1, and 0.3 represents the weight.

[0065] The input of the mid-level feature fusion layer 1 is H 11 、H 12 and H 13 , the output is S hp_12 In the mid-level feature fusion layer 1, H 11 and H 12 First pass through a softmax function, the formula is as follows:

[0066]

[0067] Among them, T represents transpose, A 21represents the connection weight. Then, H 13 and H 12 After a softmax function, the formula is expressed as follows:

[0068]

[0069] Among them, T represents transpose, A 23 Represents the connection weight. Next, H 11 and H 13 Fusion to H 12 The formula is expressed as follows:

[0070] S hp_12 =(A 21 ·H 11 +A 23 ·H 13 )*0.3+H 12 , where S hp_12 Indicates H 11 、H 12 and H 13 The fused features are also the output of the mid-level feature fusion layer 1, and 0.3 represents the weight.

[0071] The input of high-level feature fusion layer 1 is H 13 and H 12 , the output is S hp_13 In the high-level feature fusion layer 1, H 13 and H 12 First pass through a softmax function, the formula is as follows:

[0072]

[0073] Among them, T represents transpose, A 32 Represents the connection weight matrix. Next, H 12 Fusion to H 13 The formula is expressed as follows:

[0074] S hp_13 =A 32 ·H 12 *0.3+H 13 ,

[0075] Among them, S hp_13 Indicates H 12 and H 13 The fused features are also the output of the high-level feature fusion layer 1, and 0.3 represents the weight.

[0076] The input of the low-level spatiotemporal convolution module 2 is S hp_11 , the output is F hp_21The low-level spatiotemporal convolution module 2 contains 1 spatial graph convolution layer and 1 temporal convolution layer. The spatial graph convolution layer has 128 convolution kernels, each of which is 1×1 in size. The temporal convolution layer has 64 convolution kernels, each of which is 5×1 in size. The input of the mid-level spatiotemporal convolution module 2 is S hp_12 , the output is F hp_22 The structure of the mid-level spatiotemporal convolution module 2 is consistent with that of the low-level spatiotemporal convolution module 2, and its input is S hp_12 , the output is F hp_22 The structure of the high-level spatiotemporal convolution module 2 is consistent with that of the low-level spatiotemporal convolution module 2, and its input is S hp_13 , the output is F hp_23 .

[0077] The structure of fusion module 2 is consistent with that of fusion module 1. Fusion module 2 has three outputs, namely S hp_21 、S hp_22 and S hp_23 .

[0078] The input of low-level spatiotemporal convolution module 3 is S hp_21 , the output is S hp_31 The low-level spatiotemporal convolution module 3 consists of 1 spatial graph convolution layer and 1 temporal convolution layer. The spatial graph convolution layer has 256 convolution kernels, each of which is 1×1 in size. The temporal convolution layer has 128 convolution kernels, each of which is 5×1 in size. The input is S hp_22 , the output is S hp_32 The structure of the mid-level spatiotemporal convolution module 3 is consistent with that of the low-level spatiotemporal convolution module 3, and its input is S hp_22 , the output is S hp_32 The structure of the high-level spatiotemporal convolution module 3 is consistent with that of the low-level spatiotemporal convolution module 3, and its input is S hp_23 , the output is S hp_33 .

[0079] The input of low-level spatiotemporal convolution module 4 is S hp_31 , the output is S hp_41 The low-level spatiotemporal convolution module 4 contains 1 spatial graph convolution layer and 1 temporal convolution layer. The spatial graph convolution layer has 512 convolution kernels, each of which is 1×1 in size. The temporal convolution layer has 256 convolution kernels, each of which is 5×1 in size. The structure of the mid-level spatiotemporal convolution module 4 is consistent with that of the low-level spatiotemporal convolution module 4, and its input is S hp_32 , the output is S hp_42 The structure of the high-level spatiotemporal convolution module 4 is consistent with that of the low-level spatiotemporal convolution module 4, and its input is S hp_33 , the output is S hp_43 .

[0080] The input of the feature fusion module is S hp_41 、S hp_42 and S hp_43 , the output is S hp In order to make S hp_41 、S hp_42 and S hp_43 The tensor sizes of the three are consistent. The feature fusion module has the same effect on S hp_42 Perform 10 upsamplings to get And Assigning a weight of 0.6 yields To S hp_43 Perform 5 upsamplings to get And Assigning a weight of 0.6 yields Then S hp_41 、 and Add. S hp The expression is as follows:

[0081]

[0082] in, and The weights are all 0.6.

[0083] like Figure 1 As shown in the middle column, its structure is the same as Figure 1 The leftmost column has the same structure. Figure 1 The input of the middle column is the historical speed sequence V history_v , the output is S hv .like Figure 1 As shown in the rightmost column, its structure is the same as Figure 1 The leftmost column has the same structure. Figure 1 The input of the rightmost column is the historical acceleration sequence V history_a , the output is S ha .

[0084] The input of the feature cascade module is S hp 、S hv and S ha , the output is H history . H history By S hp 、S hv and S ha The result is obtained by splicing by channel. history Represents the characteristics of historical movements.

[0085] Future sequence V of human action sequence sample division futureInput motion feature extraction module to extract the process of future motion features and the historical sequence V of human action sequence sample division history The process of extracting historical features from the input motion feature extraction module is consistent, such as Figure 2 As shown, H future Represents the characteristics of future sports.

[0086] As an embodiment of this technical solution, the historical motion features and the future motion features are added together and input into the action prediction module. The historical motion features and the future motion features are added together to obtain the joint motion features, which can be expressed as follows:

[0087] H=H future +H history ,

[0088] Where H represents the joint motion feature.

[0089] The action prediction module is composed of a recurrent neural network with a total of 25 cycles. Its structure is as follows: Figure 3 As shown in Figure 1, it includes a speed calculation module, an acceleration calculation module, a gated recurrent unit (GRU), and a linear layer. The action prediction module has four inputs, namely, the joint motion feature H, the 48th frame human skeleton X in the human action sequence sample V, and the 48th frame human skeleton X in the human action sequence sample V. 48 、49th frame human skeleton X 49 , and the 50th frame human skeleton X 50 .

[0090] like Figure 3 As shown in the leftmost column, it represents the first cycle. The input of the speed calculation module is the 49th frame human skeleton X of the human action sequence sample V 49 and the 50th frame human skeleton X T50 , the output is S0, and the calculation method of S0 is as follows:

[0091]

[0092] Where S0 represents the speed of the human skeleton in the 50th frame of the human motion sequence sample V.

[0093] The input of the acceleration calculation module is the 48th frame human skeleton X in the human action sequence sample V 48 、49th frame human skeleton X 49 , and the 50th frame human skeleton X 50 , the output is C0, and the calculation method of C0 is as follows:

[0094]

[0095] Where C0 represents the acceleration of the human skeleton in the 50th frame of the human motion sequence sample V.

[0096] Next, S0, C0 and X 50 Connect the channels to get I0. The input of the GRU module is I0 and H, and the output is B0 and H1. The input of the linear layer is B0, and the output is F0. F0 is calculated as follows:

[0097] F0=W0·B0+θ0,

[0098] Where W0 is the weight of the linear layer and θ0 is the bias vector of the linear layer.

[0099] The output F0 of the linear layer and the 50th frame human skeleton X in the human action sequence sample V are combined 50 Add together to get the prediction result of the 51st frame of the human skeleton of the human action sequence sample V

[0100] like Figure 3 As shown in the second column, it represents the second cycle, and its structure is consistent with the structure of the first cycle. The input of the second cycle is the predicted result of the human skeleton of the 51st frame of the human action sequence sample V The 50th frame of the human skeleton X in the human action sequence sample V 50 、49th frame human skeleton X 49 And one of the outputs H0 of GRU in the first cycle. The output of the second cycle is the predicted result of the human skeleton of the 52nd frame of the human action sequence sample V

[0101] like Figure 3 The rightmost column shows the 25th cycle, and its structure is consistent with the structure of the 1st cycle. The input of the 25th cycle is the predicted result of the human skeleton of the 74th frame of the human action sequence sample V. Prediction results of the human skeleton in the 73rd frame Prediction results of the human skeleton in the 72nd frame and one of the outputs H of the GRU in the 24th cycle 24 The output of the 25th cycle is the predicted result of the human skeleton of the 75th frame of the human action sequence sample V

[0102] The output of the action prediction module is the calculated sequence to be predicted The expression is as follows:

[0103]

[0104] in, is the calculated matrix representation of the skeleton of the p-th frame of the sequence to be predicted.

[0105] As an embodiment of this technical solution, a dual-stream action prediction network is constructed, such as Figure 4 As shown, the input of the network is the historical sequence and future sequence divided by the human action sequence samples, and the output is the calculated sequence to be predicted, that is, the output of the action prediction module The nth prediction sequence sample output by the action prediction module is And there are 32 samples in total. The network loss function L1 is:

[0106]

[0107] Among them, ||·||1 represents the 1-norm, is the true value of the nth prediction sequence.

[0108] As an embodiment of this technical solution, the historical and future sequences of human motion sequence samples used for training are input into a dual-stream motion prediction network, and the network is trained until convergence. The human motion sequence samples used for training are taken from the field of continuous monitoring.

[0109] As an embodiment of the present technical solution, the historical motion features extracted by the trained motion feature extraction module are input into the classification module to perform motion classification, and the classification module is trained until convergence. The structure of the classification module is as follows Figure 5 As shown, the input of linear layer 1 is the historical motion feature H extracted by the trained motion feature extraction module. history , the number of neurons in linear layer 1 is 2560, and the output Q1 of linear layer 1 is calculated as follows:

[0110]

[0111] Where W1 is the weight of linear layer 1, θ1 is the bias vector of linear layer 1, is the relu activation function, and its formula is as follows:

[0112]

[0113] x is the input of the relu activation function. When x is greater than or equal to 0, the output of the relu activation function is x. When x is less than 0, the output of the relu activation function is 0.

[0114] The input of linear layer 2 is Q1. The number of neurons in linear layer 2 is 1280. The output Q2 of linear layer 2 is calculated as follows:

[0115]

[0116] in, is the relu activation function, W2 is the weight of linear layer 2, and θ2 is the bias vector of linear layer 2.

[0117] The input of linear layer 3 is Q2. The number of neurons in linear layer 3 is 15. The output Q3 of linear layer 3 is calculated as follows:

[0118]

[0119] Where W3 is the weight of linear layer 3, θ3 is the bias vector of linear layer 3, is a softmax activation function. The output of Q3 represents the probability that the input historical motion feature belongs to each action category. The total number of action categories is 15, which is the same as the number of neurons in linear layer 3. The action category corresponding to the largest probability value in Q3 is selected as the output of the classification module.

[0120] The historical sequence divided by the human motion sequence samples used for training is input into the trained motion feature extraction module to obtain historical motion feature samples. The historical motion feature samples are then input into the classification module and trained until convergence. The loss function L2 used in training the classification module is:

[0121]

[0122] Among them, the total number of action categories is 15, and the total number of input historical motion feature samples is 32. is the network output of the a-th historical motion feature sample, I a is the expected output of the ath historical motion feature sample, where I a The g-th dimension data is defined as:

[0123]

[0124] Among them, i a is the label value of the a-th historical motion feature sample.

[0125] As an embodiment of the present technical solution, the trained dual-stream action prediction network and the trained classification module are combined, and an empty dictionary module for storing information is added to construct a dual-stream information storage network, such as Figure 6 shown. Figure 6 The input of the motion feature extraction module in the first row is the historical sequence divided by the human action sequence samples used for training, and the output is the historical motion feature H history . Figure 6 The input of the motion feature extraction module in the second row is the future sequence divided by the human action sequence samples used for training, and the output is the future motion feature H future The input of the classification module is the historical motion feature H history, the output is the action category to which the historical motion feature belongs. The input of the dictionary module is the action category output by the classification module, the historical motion feature H history and future motion features H future The historical motion features and future motion features are added together and input into the action prediction module to obtain the action prediction result.

[0126] As an embodiment of this technical solution, the historical sequence and future sequence of the human motion sequence samples used for training are input into the dual-stream information storage network, and the corresponding relationship between the historical motion features and the future motion features of each action is stored in the dictionary module. The structure of the dictionary module is as follows: Figure 7 As shown, the keys in the dictionary module are used to represent action categories: key1 represents the first action category, key 15 Used to represent the 15th action category. The value corresponding to each key in the dictionary module is a sub-dictionary: value1 represents the first sub-dictionary, value 15 Represents the 15th sub-dictionary. The key of the first sub-dictionary value1 is used to store the historical motion features of the first action category, and the value of the first sub-dictionary value1 is used to store the future motion features of the first action category: key Used to store the first historical motion feature of the first action category value Used to store the first future motion feature of the first action category key Used to store the 2000th historical motion feature of the first action category value Used to store the 2000th future motion feature of the first action category The 15th sub-dictionary value 15 The key is used to store the historical motion features of the 15th action category, and the value is used to store the future motion features of the 15th action category. Used to store the first historical motion feature of the 15th action category value Used to store the first future motion feature of the 15th action category key Used to store the 2000th historical motion feature of the 15th action category value Used to store the 2000th future motion feature of the 15th action category

[0127] As an embodiment of the present technical solution, a trained motion feature extraction module, a trained classification module, a trained action prediction module and a dictionary module with stored information are combined to construct a self-updating action prediction model. The self-updating action prediction model is as follows: Figure 8 As shown. The self-updating action prediction model performs 25 action predictions in total, and its input is the historical sequence of the human action sequence samples tested The output is the final action prediction result Among them, the human motion sequence samples tested are taken from the field of continuous monitoring.

[0128] like Figure 8 As shown in the first column, it includes motion feature extraction module, classification module, dictionary module and action prediction module, which represents the first iteration process, used to perform the first action prediction. The input of the motion feature extraction module is the historical sequence divided by the human action sequence sample tested. The output is the historical motion features of the human action sequence sample tested The input of the classification module is the historical motion features of the human action sequence sample tested The output is action category g1. The input of the dictionary module is the action category g1 and the historical motion features of the tested human action sequence sample. The output is the future motion features found The method for finding future motion features is as follows:

[0129] (1) The historical motion features of the human motion sequence samples tested Vectorization, coming soon All row vectors of are concatenated into a new row vector is a vector of dimension 2560.

[0130] (2) Select the sub-dictionary corresponding to action category g1 in the dictionary module Pair Dictionary The i-th key stores the historical motion features Vectorization, coming soon All row vectors of are concatenated into a new row vector is a vector of dimension 2560.

[0131] (3) Calculate the cosine similarity between the historical motion features of the human action sequence sample under test and the i-th historical motion features stored in action category g1. The calculation method is as follows:

[0132]

[0133] Among them, · represents the dot product operation, express The 2-norm of express The 2-norm of sim i Represents the cosine similarity between the historical motion features of the tested human action sequence sample and the i-th historical motion features stored in action category g1.

[0134] (4) Calculate the similarity between all historical motion features stored in action category g1 and the historical motion features of the tested human action sequence samples, and obtain {sim1,...,sim i ,...,sim 2000}.

[0135] (5) Find the maximum value of cosine similarity sim j ,sim j =max{sim1,...,sim i ,...,sim 2000}.

[0136] (6) Find the jth key in the sub-dictionary corresponding to action category g1 And find the value corresponding to the jth key Will Stored future motion characteristics As the future motion features found

[0137] Next, the historical motion features of the human motion sequence samples tested are and the future motion features found Add them together and input them into the action prediction module to get the first action prediction result The formula is as follows:

[0138]

[0139] in, is the matrix expression of the p-th frame skeleton of the first action prediction result, This is the prediction result of the human skeleton of the 51st frame of the tested human action sequence sample V.

[0140] like Figure 8 As shown in the second column, it represents the first update process, which is used to predict the second action. Its structure is the same as Figure 8 The structure of the first column is the same. The input of the motion feature extraction module is the first reconstructed historical sequence The output is the historical motion features after the first update The first reconstruction of the historical sequence History sequence divided by the tested human action sequence samples The 2nd frame human skeleton X2 to the 50th frame human skeleton X 50And the first frame human skeleton in the first action prediction result The composition is expressed as follows:

[0141]

[0142] The input of the classification module is the historical motion features after the first update The output is the action category g2 after the first update. The input of the dictionary module is the historical motion features after the first update. And the action category g2 after the first update, the output is the future motion feature obtained by querying after the first update The historical movement characteristics after the first update And the future motion features obtained after the first update Add them together and input them into the action prediction module to get the second action prediction result Second action prediction result The formula is as follows:

[0143]

[0144] in, is the matrix expression of the p-th frame skeleton of the second action prediction result, This is the prediction result of the human skeleton of the 52nd frame of the tested human action sequence sample V.

[0145] like Figure 8 As shown in the third column, it represents the second update process, which is used to predict the third action. Its structure is the same as Figure 8 The structure of the first column is the same. The input of the motion feature extraction module is the second reconstructed history sequence The output is the historical motion features after the second update The second reconstruction of the historical sequence History sequence divided by the tested human action sequence samples The 3rd frame human skeleton X3 to the 50th frame human skeleton X 50 , the first frame human skeleton in the first action prediction result And the first frame human skeleton in the second action prediction result The composition is expressed as follows:

[0146]

[0147] The input of the classification module is the historical motion features after the second update The output is the action category g3 after the second update. The input of the dictionary module is the historical motion features after the second update. And the action category g3 after the second update, the output is the future motion feature obtained by querying after the second update The historical movement characteristics after the second update And the future motion features obtained after the second update Add them together and input them into the action prediction module to get the third action prediction result The third action prediction result The formula is as follows:

[0148]

[0149] in, is the matrix expression of the p-th frame skeleton of the third action prediction result, This is the prediction result of the human skeleton of the 53rd frame of the tested human action sequence sample V.

[0150] Figure 8 The last column is the 24th update process, which is used to predict the 25th action. Its structure is the same as Figure 8 The structure of the first column is the same, and its final output is the 25th action prediction result. The formula is as follows:

[0151]

[0152] in, is the matrix expression of the skeleton of the p-th frame of the 25th action prediction result, This is the prediction result of the human skeleton of the 75th frame of the tested human action sequence sample V.

[0153] Final action prediction results It is composed of the first frame human skeleton in each action prediction result mentioned above, and its formula is expressed as follows:

[0154]

[0155] in, The first frame of the human skeleton representing the first action prediction result, The first frame of the human skeleton representing the q-th action prediction result, The first frame of the human skeleton representing the 25th action prediction result.

[0156] As an embodiment of the present technical solution, a historical sequence divided by human motion sequence samples used for testing is input into a self-updating motion prediction model to achieve motion prediction.

[0157] The beneficial effect of this technical solution is that the proposed self-updating action prediction model can update the historical motion characteristics of human action sequence samples, making the human action intention more clear and helping to improve the accuracy of long-term action prediction. In addition, the self-updating action prediction model is more suitable for action prediction in continuous video surveillance.

[0158] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0159] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0160] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0161] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0162] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0163] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A motion prediction method based on continuous monitoring, characterized in that: include: Step 1: Divide the preset human motion sequence samples into historical sequences, sequences to be predicted, and future sequences, and input the historical sequences and future sequences into a preset motion feature extraction module to extract historical motion features and future motion features; wherein, The motion feature extraction module includes a speed calculation module, an acceleration calculation module, a level division module, a fusion module, a spatiotemporal convolution module, a feature fusion module and a feature cascade module; Step 2: After adding the historical motion features and the future motion features, the features are input into the preset motion prediction module for training to construct a dual-stream motion prediction network. Step 3: Input the historical sequence and future sequence into the two-stream action prediction network and train the two-stream action prediction network until convergence; Step 4: Through the trained motion feature extraction module, extract historical motion features and input them into the preset classification module for action classification, and train the classification module until convergence; Step 5: Combine the trained dual-stream action prediction network and classification module and add the preset dictionary module to construct a dual-stream information storage network; The dictionary module of the dual-stream information storage network stores the correspondence between the historical motion features and the future motion features of each action; Step 6: Combine the trained motion feature extraction module, the trained classification module, the trained action prediction module and the dictionary module that has stored the information to construct a self-updating action prediction model; Step 7: Input the historical sequence divided by the human motion sequence samples used for testing into the self-updating motion prediction model to determine the motion prediction result; Wherein, the step 1 includes: Step 11: Construct a human action sequence sample using several preset human skeleton samples: V=[X1,…,X t ,…,X N ] Among them, V represents the human action sequence sample, t represents the t-th frame number of the human action sequence sample, t∈[1,N], N represents the total number of frames of the human action sequence sample, X t ∈R M×D ,R M×D The matrix representing the human skeleton of the t-th frame of the human action sequence sample V, M represents the number of key points contained in the human skeleton of the t-th frame; D represents the dimension of the vector representation of each key point in the human skeleton of the t-th frame; Step 12: Divide the human action sequence sample V into a history sequence V history , sequence to be predicted V predict and Future Sequence V future ;in, Among them, V history Represents the historical sequence, h represents the time index of the historical sequence, h∈[1,T history ], T history ∈[1,T predict ], T history Represents the total number of frames in the history sequence; X h ∈R M×D , X h represents the matrix of the human skeleton of the hth frame of the historical sequence, X1 represents the matrix of the human skeleton of the first frame of the historical sequence, X Thistory The matrix representing the human skeleton of the last frame in the history sequence; V predict Represents the sequence to be predicted, p represents the time index of the sequence to be predicted, p∈[T history +1,T history +T predict ], T predict Represents the total number of frames in the sequence to be predicted, T predict ∈[T predict ,T future ];X p ∈R M×D , X p The matrix representing the human skeleton of the pth frame of the sequence to be predicted; The matrix representing the human skeleton of the first frame of the history sequence; The matrix representing the human skeleton of the last frame in the history sequence; V future represents the future sequence, f represents the time index of the historical sequence, f∈[T history +T predict +1,T history +T predict +T future ], T future Represents the total number of frames in the future sequence, T future ∈[T future ,N];X f ∈R M×D , X f The matrix representing the human skeleton of the fth frame of the future sequence; The matrix representing the human skeleton of the first frame of the future sequence, The matrix representing the human skeleton of the last frame in the future sequence, N = T history +T predict +T future ; Step 13: Use the speed calculation module to extract speed features from the historical sequence and determine the historical speed sequence; Among them, V history_v Represents the historical speed sequence, vel1 represents the matrix of the first frame skeleton in the historical speed sequence, The matrix representing the skeleton of the first frame in the future velocity sequence, is the historical speed sequence V history_v The matrix representation of the h-th frame skeleton in, h represents the time index of the historical sequence, X h The matrix representing the human skeleton of the hth frame of the history sequence, X h-1 The matrix representing the human skeleton of the h-1th frame of the history sequence, Represents the Tth in the historical speed sequence history The matrix of the frame skeleton, T history Represents the total number of frames in the history sequence; Step 14: Calculate the historical velocity sequence through the acceleration calculation module to determine the historical acceleration sequence: Among them, V history_a Represents the historical acceleration sequence, acl1 represents the matrix of the first frame skeleton in the historical acceleration sequence, acl h The matrix representing the h-th frame skeleton in the historical acceleration sequence, Represents the Tth acceleration in the historical acceleration sequence history The matrix of the frame skeleton; Step 15: Using a level division module, the historical sequence is divided into low-level historical sequence, intermediate historical sequence and high-level historical sequence; Among them, V hp_1 Represents the low-level historical sequence, V history Represents the historical sequence, V hp_2 Represents the intermediate historical sequence of the human skeleton obtained by inputting the first pooling layer into the historical sequence, mid h ∈R W×D ,R W×D represents the matrix of the h-th frame human skeleton in the intermediate history sequence about the human skeleton, W and D represent the number of rows and columns of the matrix of the h-th frame skeleton in the intermediate history sequence, V hp_3 Represents the advanced historical sequence, V hp_3 It is the high-level history sequence of the human skeleton obtained by inputting the history sequence into the second pooling layer. h ∈R U×D , U and D correspond to the number of rows and columns of the matrix of the h-th frame skeleton in the high-level history sequence, respectively; Step 16: The low-level history sequence, the intermediate history sequence, and the high-level history sequence are sequentially transmitted to the corresponding spatiotemporal convolution module 1, the fusion module 1, the spatiotemporal convolution module 2, the fusion module 2, the spatiotemporal convolution module 3, and the spatiotemporal convolution module 4 to generate corresponding historical fusion features; Step 17: Transfer the future sequence to steps 13-15 to generate the corresponding future fusion features; Step 18: Generate historical motion features and future motion features through the historical fusion features and the future fusion features; Wherein, the step 2 includes: Step 21: Add the historical motion features and the future motion features to obtain the joint motion features; H=H future +H history Among them, H represents the joint motion characteristics; Step 22: Combine the joint motion feature H and the Tth history -2 human skeleton frames No. T history -1 human skeleton and T history Frame human skeleton Input the preset recurrent neural network to calculate the sequence to be predicted; Step 23: Increment the human skeleton of the human action sequence sample V and repeat step 22 to determine the sequence to be predicted: p∈[T history +1,T history +T predict ], in, is the matrix representation of the skeleton of the p-th frame of the sequence to be predicted; Step 24: Construct a dual-stream motion prediction network through human motion sequence samples, motion feature extraction module and motion prediction module; wherein, The loss function L1 of the dual-stream action prediction network is: Among them, ||||1 represents the 1 norm, is the true value of the nth prediction sequence, is the output of the action prediction module, Represents the nth predicted sequence sample output by the action prediction module.

2. The method for motion prediction based on continuous monitoring according to claim 1, wherein: The step 17 comprises: Step 170: Transmit the future sequence to the speed calculation module to obtain the corresponding future speed sequence; Step 171: transmitting the future velocity sequence to an acceleration calculation module to calculate a corresponding future acceleration sequence; Step 172: Transmit the future acceleration sequence to a level division module to determine a low-level future sequence, a mid-level future sequence, and a high-level future sequence; Step 173: The low-level future sequence, the intermediate future sequence, and the high-level future sequence are sequentially transmitted to the corresponding spatiotemporal convolution module 1, the fusion module 1, the spatiotemporal convolution module 2, the fusion module 2, the spatiotemporal convolution module 3, and the spatiotemporal convolution module 4 to generate corresponding future fusion features; wherein, The fusion module 1 includes a low-level convolution layer 1, a low-level linear layer 1, a low-level aggregation-related feature layer 1, a low-level update feature layer 1, a low-level feature fusion layer 1, a medium-level convolution layer 1, a medium-level linear layer 1, a medium-level aggregation-related feature layer 1, a medium-level update feature layer 1, a medium-level feature fusion layer 1, a high-level convolution layer 1, a high-level linear layer 1, a high-level aggregation-related feature layer 1, a high-level update feature layer 1 and a high-level feature fusion layer 1.

3. The method for motion prediction based on continuous monitoring according to claim 1, wherein: The step 16 comprises: Step 160: Set the low-level history sequence V hp_1 Input low-level spatiotemporal convolution module 1, output F hp_11 ; F hp_11 Representative and low-level history sequence V hp_1 The output of the corresponding low-level spatiotemporal convolution module 1; Step 161: Intermediate History Sequence V hp_2 Input mid-level spatiotemporal convolution module 1, output F hp_12 ; F hp_12 Representative and Intermediate History Sequence V hp_2 The output of the corresponding mid-level spatiotemporal convolution module 1; Step 162: Advanced History Sequence V hp_3 Input high-level spatiotemporal convolution module 1, output F hp_13 ; F hp_13 Representative and Advanced History Sequence V hp_3 The output of the corresponding high-level spatiotemporal convolution module 1; Step 163: F hp_11 、F hp_12 、F hp_13 Input fusion module 1, output the features of all key points in the updated low-level human skeleton, the features of all key points in the updated mid-level human skeleton, and the features of all key points in the updated high-level human skeleton; Step 164: Through H 11 、H 12 and H 13 , determine and generate the corresponding historical fusion features.

4. The method for motion prediction based on continuous monitoring according to claim 3, wherein: The step 163 further includes: Step A1: F hp_11 Input low-level convolution layer 1, output G 11 ; G 11 Representative and low-level history sequence V hp_1 The output of the corresponding low-level convolutional layer 1; Step A2: G 11 Input low-level linear layer 1, output is F 11 ; F 11 Represents the features of all key points in the extracted low-level human skeleton; Represents the features of the i-th key point in the extracted low-level human skeleton, Represents the features of the jth key point in the extracted low-level human skeleton; Step A3: For the i-th key point in the low-level human skeleton, the low-level clustered related feature layer 1 aggregates the features of the remaining M-1 key points to the i-th key point, and then passes through the three fully connected layers of the low-level clustered related feature layer 1 to determine the corresponding key point column vector; Among them, [,] represents the channel connection function, f 11 () represents the 3-layer fully connected layer function of the low-level aggregation related feature layer 1, r 11_i is a column vector about the i-th key point, i = 1, 2, ..., M, M represents the total number of key points; Step A4: Execute step A3 for all key points in the low-level human skeleton to obtain the two-dimensional tensor of the key points, R 11 The formula is as follows: R 11 =(r 11_1 ,...,r 11_i ,...,r 11_M ),i∈[1,M], Among them, R 11 It is a two-dimensional tensor consisting of M column vectors; Step A5: R 11 Input low-level updated feature layer 1, output keypoint feature H 11 , the low-level update feature layer 1 is used to update the features. The low-level update feature layer 1 consists of 3 fully connected layers. The formula for updating the features is as follows: H 11 =g 11 ([F 11 ,R 11 ]) Among them, [,] means according to the channel connection function, g 11 () represents the function of updating the low-level feature layer to obtain a 3-layer fully connected layer, H 11 are the features of all key points in the updated low-level human skeleton; Step A6: F hp_12 Input mid-level convolution layer 1, output G 12 ; G 12 Representative and Intermediate History Sequence V hp_2 The output of the corresponding mid-level convolutional layer 1; Step A7: G 12 Input mid-level linear layer 1, output is F 12 ; F 12 Represents the features of all key points in the extracted mid-level human skeleton; represents the feature of the i-th key point in the extracted mid-level human skeleton, Represents the features of the jth key point in the extracted mid-level human skeleton; Step A8: For the i-th key point in the mid-level human skeleton, the mid-level clustered related feature layer 1 aggregates the features of the remaining M-1 key points onto the i-th key point, and then passes through the three fully connected layers of the mid-level clustered related feature layer 1 to determine the corresponding key point column vector; Among them, [,] represents the channel connection function, f 12 () represents the 3-layer fully connected layer function of the mid-level aggregation related feature layer 1, r 12_i is a column vector about the i-th key point, i = 1, 2, ..., M, M represents the total number of key points; Step A9: Execute step A8 for all key points in the mid-level human skeleton to obtain a two-dimensional tensor of the key points of the mid-level human skeleton, R 12 The formula is as follows: R 12 =(r 12_1 ,...,r 12_i ,...,r 12_M ),i∈[1,M], Among them, R 12 It is a two-dimensional tensor consisting of M column vectors; Step A10: R 12 Input mid-level update feature layer 1, output key point feature H 12 , the mid-level update feature layer 1 is used to update features. The mid-level update feature layer 1 consists of 3 fully connected layers. The formula for updating features is as follows: H 12 =g 12 ([F 12 ,R 12 ]) Among them, [,] means according to the channel connection function, g 12 () represents the function of updating the low-level feature layer to obtain a 3-layer fully connected layer, H 12 are the features of all key points in the updated low-level human skeleton; Step A11: F hp_12 Input mid-level convolution layer 1, output G 12 ; G 12 Representative and Intermediate History Sequence V hp_2 The output of the corresponding mid-level convolutional layer 1; Step A12: G 12 Input mid-level linear layer 1, output is F 12 ; F 12 Represents the features of all key points in the extracted mid-level human skeleton; represents the feature of the i-th key point in the extracted mid-level human skeleton, Represents the features of the jth key point in the extracted mid-level human skeleton; Step A13: For the i-th key point in the mid-level human skeleton, the mid-level clustered related feature layer 1 aggregates the features of the remaining M-1 key points onto the i-th key point, and then passes through the three fully connected layers of the mid-level clustered related feature layer 1 to determine the corresponding key point column vector; Among them, [,] represents the channel connection function, f 12 () represents the 3-layer fully connected layer function of the mid-level aggregation related feature layer 1, r 12_i is a column vector about the i-th key point, i = 1, 2, ..., M, M represents the total number of key points; Step A14: Execute step A8 for all key points in the mid-level human skeleton to obtain a two-dimensional tensor of the key points of the mid-level human skeleton, R 12 The formula is as follows: R 12 =(r 12_1 ,...,r 12_i ,...,r 12_M ),i∈[1,M], Among them, R 12 It is a two-dimensional tensor consisting of M column vectors; Step A15: R 12 Input mid-level update feature layer 1, output key point feature H 12 , the mid-level update feature layer 1 is used to update features. The mid-level update feature layer 1 consists of 3 fully connected layers. The formula for updating features is as follows: H 12 =g 12 ([F 12 ,R 12 ]) Among them, [,] means according to the channel connection function, g 12 () represents the function of updating the feature layer at the middle level to obtain a 3-layer fully connected layer, H 12 are the features of all key points in the updated mid-level human skeleton; Step A16: F hp_13 Input high-level convolution layer 1, output G 13 ; G 13 Representative and Advanced History Sequence V hp_3 The output of the corresponding high-level convolutional layer 1; Step A17: G 13 Input high-level linear layer 1, output is F 13 ; F 13 Represents the features of all key points in the extracted high-level human skeleton; Represents the features of the i-th key point in the extracted high-level human skeleton, Represents the features of the jth key point in the extracted high-level human skeleton; Step A18: For the i-th key point in the high-level human skeleton, the high-level aggregation-related feature layer 1 aggregates the features of the remaining M-1 key points onto the i-th key point, and then passes through the three fully connected layers of the high-level aggregation-related feature layer 1 to determine the corresponding key point column vector; Among them, [,] represents the channel connection function, f 13 () represents the 3-layer fully connected layer function of high-level aggregation related feature layer 1, r 13_i is a column vector about the i-th key point, i = 1, 2, ..., M, M represents the total number of key points; Step A19: Execute step A8 for all key points in the high-level human skeleton to obtain a two-dimensional tensor of key points of the high-level human skeleton, R 13 The formula is as follows: R 13 =(r 13_1 ,...,r 13_i ,...,r 13_M ),i∈[1,M], Among them, R 13 It is a two-dimensional tensor consisting of M column vectors; Step A15: R 13 Input high-level updated feature layer 1, output keypoint feature H 13 , the high-level update feature layer 1 is used to update the features. The high-level update feature layer 1 consists of 3 layers of fully connected layers. The formula for updating features is as follows: H 13 =g 13 ([F 13 ,R 13 ]) Among them, [,] means according to the channel connection function, g 13 () represents the function of obtaining 3 fully connected layers based on the high-level updated feature layer, H 13 are the features of all key points in the updated mid-level human skeleton.

5. The method for motion prediction based on continuous monitoring according to claim 3, wherein: The step 164 includes: Step S10: H 11 and H 12 Input low-level feature fusion layer 1, output S hp_11 ; Step S10 includes: In the low-level feature fusion layer 1, H 11 and H 12 First pass through a softmax function, the formula is as follows: Among them, T represents transpose, A 12 represents the connection weight matrix, H 11 is the feature of all key points in the updated low-level human skeleton, H 12 For the features of all key points in the updated mid-level human skeleton, the softmax function is used to normalize the value of each element of the input vector to between (0, 1). The calculation formula for each element is as follows: Among them, e represents the natural base, K represents the number of elements contained in the input vector, and a l Represents the lth element of the input vector, a k Represents the kth element of the input vector, k=1,2,…,K,S l Represents the normalized value of the lth element of the input vector; Step S11: H 12 Fusion to H 11 The formula is expressed as follows: S hp_11 =A 12 ·H 12 *ω+H 11 Among them, S hp_11 Indicates H 12 and H 11 The fused features are also the output of the low-level feature fusion layer 1, and ω represents the weight; Step S12: 11 、H 12 and H 13 Input mid-level feature fusion layer 1, output is S hp_12 ; Among them, in the mid-level feature fusion layer 1, H 11 and H 12 First pass through a softmax function, the formula is as follows: Among them, T represents transpose, A 21 represents the connection weight; Then, H 13 and H 12 After a softmax function, the formula is expressed as follows: Among them, T represents transpose, A 23 represents the connection weight, H 13 The features of all key points in the updated advanced human skeleton; Next, H 11 and H 13 Fusion to H 12 The formula is expressed as follows: S hp_12 =(A 21 ·H 11 +A 23 ·H 13 )*θ+H 12 , Among them, S hp_12 Indicates H 11 、H 12 and H 13 The fused features are also the output of the mid-level feature fusion layer 1, and θ represents the weight; Step S13: H 13 and H 12 Input high-level feature fusion layer 1, output is S hp_13 ; Among them, in the high-level feature fusion layer 1, H 13 and H 12 First pass through a softmax function, the formula is as follows: Among them, T represents transpose, A 32 represents the connection weight matrix, Next, H 12 Fusion to H 13 The formula is expressed as follows: S hp_13 =A 32 ·H 12 *γ+H 13 , Among them, S hp_13 Indicates H 12 and H 13 The fused features are also the output of the high-level feature fusion layer 1, and γ represents the weight; Step S14: hp_11 Input low-level spatiotemporal convolution module 2, output F hp_21 ; Among them, the low-level spatiotemporal convolution module 2 includes 1 spatial graph convolution layer and 1 temporal convolution layer; the spatial graph convolution layer has 128 convolution kernels, each of which is 1×1 in size; the temporal convolution layer has 64 convolution kernels, each of which is 5×1 in size; Step S15: hp_12 Input mid-level spatiotemporal convolution module 2, output F hp_22 ; Among them, the structure of the mid-level spatiotemporal convolution module 2 is consistent with the structure of the low-level spatiotemporal convolution module 2, and its input is S hp_12 , the output is F hp_22 The structure of the high-level spatiotemporal convolution module 2 is consistent with that of the low-level spatiotemporal convolution module 2, and its input is S hp_13 , the output is F hp_23 ; Step S16: S hp_21 Input low-level spatiotemporal convolution module 3, output S hp_31 ; Among them, the low-level spatiotemporal convolution module 3 includes 1 spatial graph convolution layer and 1 temporal convolution layer; the spatial graph convolution layer has 256 convolution kernels, each of which is 1×1 in size; the temporal convolution layer has 128 convolution kernels, each of which is 5×1 in size; Step S17: S hp_22 Input mid-level spatiotemporal convolution module 3, output S hp_32 ; Wherein, the structure of the mid-level spatiotemporal convolution module 3 is consistent with the structure of the low-level spatiotemporal convolution module 3; Step S18: S hp_23 Input mid-level spatiotemporal convolution module 3, output S hp_33 ; Wherein, the structure of the high-level spatiotemporal convolution module 3 is consistent with the structure of the low-level spatiotemporal convolution module 3; Step S19: S hp_31 Input low-level spatiotemporal convolution module 4, output S hp_41 ; Among them, the low-level spatiotemporal convolution module 4 includes 1 spatial graph convolution layer and 1 temporal convolution layer; the spatial graph convolution layer has 512 convolution kernels, each of which is 1×1 in size; the temporal convolution layer has 256 convolution kernels, each of which is 5×1 in size; Step S20: S hp_32 Input mid-level spatiotemporal convolution module 4, output S hp_42 ; Wherein, the structure of the mid-level spatiotemporal convolution module 4 is consistent with the structure of the low-level spatiotemporal convolution module 4; Step S21: S hp_33 Input high-level spatiotemporal convolution module 4, output S hp_43 ; Wherein, the structure of the high-level spatiotemporal convolution module 4 is consistent with the structure of the low-level spatiotemporal convolution module 3; Step S22: S hp_41 、S hp_42 and S hp_43 Input the preset feature fusion module and output S hp ; Step S23: Feature fusion module performs S hp_42 Take samples, get And Assign weight β1; for S hp_43 Perform 5 upsamplings to get And Weight β2; S hp_41 、 S hp The expression is as follows: Among them, β1 and β2 correspond to weight; Step S24: S hp 、S hv and S ha Input the input of the feature cascade module, and the output is H history Among them, H history By S hp 、S hv and S ha According to the channel splicing, H history Represents the characteristics of historical movements; Step S25: Set the future sequence V future Input motion feature extraction module to obtain future motion features H future .

6. The method for motion prediction based on continuous monitoring according to claim 1, characterized in that: in, The step 22 further includes: Step 220: The Tth history -1 human skeleton and T history Frame human skeleton Input the preset recurrent neural network to calculate the speed S0 of the human skeleton: Among them, S0 represents the Tth human action sequence sample V history Frame speed of human skeleton; Step 221: The Tth history -2 human skeleton frames No. T history -1 human skeleton and T history Frame human skeleton Transmitted to the acceleration calculation module to calculate the acceleration of the human skeleton: Among them, C0 represents the Tth human action sequence sample V history Frame acceleration of the human skeleton; Step 222: The velocity S0, acceleration C0 and the first history Frame human skeleton According to the channel connection, the connection result I0 is determined, and the connection result I0 and the joint motion feature H are transmitted to the gated recurrent unit GRU module to output B0 and H0; Step 223: B0 is transferred to the linear layer and output F0. F0 is calculated as follows: F0=W0·B0+θ0, Where W0 is the weight of the linear layer and θ0 is the bias vector of the linear layer; Step 224: Combine the output F0 of the linear layer and the Tth history Frame human skeleton Add together and get the Tth human action sequence sample V history +1 frame human skeleton prediction result Step 225: The Tth history +1 frame human skeleton prediction result The Tth human action sequence sample V history Frame human skeleton No. T history -1 human skeleton And one of the outputs H0 of GRU in the first cycle is input into the second cycle, and the output is the Tth human action sequence sample V history +2 frames of human skeleton prediction results 7. The method for motion prediction based on continuous monitoring according to claim 1, wherein: The step 4 comprises: Step 40: Extract the historical motion features H of the trained motion feature extraction module history , the historical motion feature H history Transmitted to the linear layer 1 of the classification module, output result Q1; Where W1 is the weight of linear layer 1, θ1 is the bias vector of linear layer 1, is the relu activation function, relu stands for linear rectification function, x is the input of the relu activation function. When x is greater than or equal to 0, the output of the relu activation function is x. When x is less than 0, the output of the relu activation function is 0. Q1 represents the output result of linear layer 1. Step 41: Input the output Q1 of linear layer 1 into linear layer 2, and output the result Q2; in, is the relu activation function, W2 is the weight of linear layer 2, and θ2 is the bias vector of linear layer 2; Step 42: Input the output Q1 of linear layer 1 into linear layer 2, and output the result Q2; Where W3 is the weight of linear layer 3, θ3 is the bias vector of linear layer 3, represents the softmax activation function, Q3 represents the probability that the input historical motion feature belongs to each action category; Step 43: Select the action category corresponding to the maximum probability value in Q3, and use the action category as the output of the classification module; Step 44: Input the historical sequence into the trained motion feature extraction module to determine the historical motion feature samples; Step 45: Input the historical motion feature samples into the classification module, and train the classification module until convergence based on the loss function L2 preset in the classification module; wherein the loss function L2 used in training the classification module is: Among them, G is the total number of action categories, A is the total number of input historical motion feature samples, is the probability of each action category of the a-th historical motion feature sample, Represents the network output of the a-th historical motion feature sample of the g-th dimension data, I a is the expected output of the ath historical motion feature sample, where I a The g-th dimension data is defined as: Among them, i a is the label value of the a-th historical motion feature sample.

8. The method for motion prediction based on continuous monitoring according to claim 1, wherein: The self-updating motion prediction model includes a motion feature extraction module, a classification module, a dictionary module and a motion prediction module; wherein, The input of the motion feature extraction module is the historical sequence of the human motion sequence samples tested The output is the historical motion features of the human action sequence sample tested The input of the classification module is the historical motion features of the human action sequence sample tested The output is action category g1; The input of the dictionary module is the historical motion features of the action category g1 and the tested human action sequence sample Finding future motion features Finding future motion features The following steps are involved: Step 81: The historical motion features of the human action sequence samples tested Vectorization, determine the new row vector in, described is a vector of dimension 2560; Step 82: Select the sub-dictionary value corresponding to the action category g1 in the dictionary module g1 , pair dictionary The i-th key stores the historical motion features Vectorize the historical motion features Connect all row vectors in to determine the new row vector in, described is a vector of dimension 2560; Step 83: Calculate the cosine similarity between the historical motion feature of the human motion sequence sample and the i-th historical motion feature stored in the motion category g1: Among them, · represents the dot product operation, express The 2-norm of express The 2-norm of sim i Represents the cosine similarity between the historical motion features of the human action sequence sample being tested and the i-th historical motion features stored in action category g1; Step 84: Calculate the similarity between all historical motion features stored in the action category g1 and the historical motion features of the tested human action sequence sample, and obtain {sim1,...,sim i ,...,sim 2000 }; Step 85: Find the maximum value of cosine similarity sim j ,sim j =max{sim1,...,sim i ,...,sim 2000 };in, The sim j is the maximum value of cosine similarity; Step 86: Find the jth key in the sub-dictionary corresponding to action category g1 And find the value corresponding to the jth key Will Stored future motion characteristics As the future motion features found 9. The method for motion prediction based on continuous monitoring according to claim 5, characterized in that: The steps S15 and S16 also include: The low-level history sequence, the intermediate history sequence and the high-level history sequence are sequentially transmitted to the corresponding spatiotemporal convolution module 1, the fusion module 1 and the spatiotemporal convolution module 2, and the low-level convolution result F is output. hp_21 , intermediate convolution result F hp_22 And the advanced convolution result F hp_23 ; Step S151: The low-level convolution result F hp_21 Input the low-level convolution layer 2 of the fusion module 2, and output the result G 21 , the result G 21 Input low-level linear layer 2, output result F 21 , the result F 21 Input low-level aggregation related feature layer 2, output result H 21 , the result H 21 Input low-level feature fusion layer 2 and determine the result S hp_21 ; Step S152: The intermediate convolution result F hp_22 Input the mid-level convolution layer 2 of the fusion module 2, and output the result G 22 , the result G 22 Input mid-level linear layer 2, output result F 22 , the result F 22 Input mid-level aggregation related feature layer 2, output result H 22 , the result H 22 Input mid-level feature fusion layer 2 and determine the result S hp_22 ; Step S153: The advanced convolution result F hp_23 Input the high-level convolution layer 2 of the fusion module 2, and output the result G 23 , the result G 23 Input high-level linear layer 2, output result F 23 , the result F 23 Input high-level aggregation related feature layer 2, output result H 23 , the result H 23 Input high-level feature fusion layer 2 and determine the result S hp_23 .

10. The method for motion prediction based on continuous monitoring according to claim 1, wherein: The step 6 comprises: Step A1: The historical motion features of the human action sequence samples tested and the future motion features found Add them together and input them into the action prediction module to get the first action prediction result The formula is as follows: in, The matrix expression of the skeleton of the p-th frame of the first action prediction result, The prediction result of the human skeleton of the 51st frame of the human action sequence sample V tested, is the prediction result of the human skeleton of the p-th frame of the human action sequence sample V being tested, The predicted result of the human skeleton of the 75th frame of the human action sequence sample V tested; Step A2: Get the first frame of human skeleton in the first frame human skeleton matrix Predict the result through the first action The first frame of the human skeleton The human skeleton from the second frame to the T frame in the human action sequence sample history Frame human skeleton, forming the historical sequence of the first reconstruction The first reconstruction of the historical sequence Input motion feature extraction module, output is the historical motion feature after the first update Step A3: Update the historical motion features after the first update Input classification module, output action category g2; Step A4: Combine the action category g2 and the historical motion features after the first update Input the dictionary module, return to the dictionary module of the self-updating motion prediction model, repeat steps 81-86 to find future motion features Step A5: Repeat step A1 to get the second action prediction result Step A6: Predict the result through the first action The first frame of the human skeleton, the second action prediction result The first frame of the human skeleton in the human action sequence sample and the third frame of the human skeleton to the T frame history Frame human skeleton, forming the second reconstruction history sequence The historical sequence Input the motion feature extraction module, repeat steps A2-A5 to a preset number of repetitions, and determine the final motion prediction result based on the number of repetitions.

Images