Human 3D skeleton motion prediction method based on multi-layer perceptron

Abstract

The present invention discloses a method for predicting 3D human skeleton motion based on a multi-layer perceptron. 3D skeleton joint point sequence data is input into a first spatiotemporal separable graph convolutional network, and a predicted first joint coordinate sequence is output; the first joint coordinate sequence is linearly projected to a hidden dimension through a first fully connected layer to generate a two-dimensional real-valued input table; the two-dimensional real-valued input table is input into at least one multi-layer perceptron mixer block, and a human skeleton spatiotemporal dependency information matrix is output; the human skeleton spatiotemporal dependency information matrix is input into a second spatiotemporal separable graph convolutional network, and a predicted second joint coordinate sequence is output; the second joint coordinate sequence is sequentially fed into a second fully connected layer and a third fully connected layer, and a predicted 3D skeleton joint point sequence is output. The present invention improves the state-of-the-art in short-term and long-term predictions, with low network computational complexity, short time consumption, and high efficiency, and has strong robustness to difficulties such as scale differences and motion complexity differences of different human bodies.

Description

Technical Field

[0001] The present invention relates to an intelligent robot navigation technology and an intelligent car unmanned driving technology, and specifically to a human 3D skeleton motion prediction method based on a multi-layer perceptron, belonging to the field of computer vision. Background Art

[0002] The goal of human motion prediction is to predict the motion trend of a skeleton-based human body in the future based on a given historical motion sequence. This is an important computer vision task with many potential applications such as autonomous driving, human-computer interaction, target tracking, and motion planning.

[0003] Given a few time steps of human motion, we can predict how that person will continue to move and imagine the complex dynamics of their future motion. The ability to make such predictions allows us to react to and plan our own actions. Similarly, predictive models of human motion are an essential component of many real-world computer vision applications, such as surveillance systems and collision avoidance for robots and self-driving cars. In recent years, research on 3D human motion prediction has received considerable attention. Deep models are designed to take a few time steps of 3D motion trajectories as input and predict long-term future 3D motion trajectories as output. Summary of the Invention

[0004] The purpose of the present invention is to provide a human 3D skeleton motion prediction method based on a multi-layer perceptron to achieve state-of-the-art performance using far fewer parameters than existing methods.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] A method for predicting 3D human skeleton motion based on a multi-layer perceptron includes the following steps:

[0007] Input the 3D skeleton joint point sequence into the first spatiotemporal separable graph convolutional network and output the predicted first joint coordinate sequence;

[0008] Linearly project the first joint coordinate sequence to the hidden dimension through the first fully connected layer to generate a two-dimensional real-valued input table;

[0009] Input the two-dimensional real-valued input table into at least one multi-layer perceptron mixer block, and output a human skeleton spatiotemporal dependency information matrix;

[0010] Input the human skeleton spatiotemporal dependency information matrix into the second spatiotemporal separable graph convolutional network and output the predicted second joint coordinate sequence;

[0011] The second joint coordinate sequence is fed into the second fully connected layer and the third fully connected layer in sequence, and the predicted 3D skeleton joint point sequence is output.

[0012] Furthermore, the first spatiotemporal separable graph convolutional network includes a first residual block, a first graph convolutional network, and a first temporal convolutional network. Inputting the 3D skeleton joint point sequence into the first spatiotemporal separable graph convolutional network and outputting the predicted first joint coordinate sequence includes:

[0013] The joint coordinates in the input 3D skeleton joint point sequence are encoded through the convolution layer and normalization layer in the first residual block to obtain the residual;

[0014] The 3D skeleton joint point sequence is input into the first graph convolutional network, and the Einstein summation of the temporal adjacency matrix and the spatial adjacency matrix is performed in turn;

[0015] The summation result is input into the first time convolutional network, and the future joint coordinates are predicted using the spatiotemporal representation.

[0016] Output the first joint coordinate sequence

[0017] Furthermore, the 3D skeleton joint point sequence is T h is the time step, 3D human posture x t ∈R 3×J , J is the number of joint points in the human skeleton, and the predicted first joint coordinate sequence is Where K is the spatial dimension, K = 3*J, and each time step x t The frame is flattened into a vector of length K = 3*J; the first joint coordinate sequence is linearly projected to the hidden dimension through the first fully connected layer to generate a two-dimensional real-valued input table, including:

[0018] Each body skeleton x t ∈R K Through the first fully connected layer linear projection to the hidden dimension C, a two-dimensional real-valued input table is obtained

[0019] Furthermore, the multilayer perceptron mixer block includes a channel mixing MLP, a label mixing MLP, a first jump connection layer, a second jump connection layer, a first SE block and a second SE block, and the two-dimensional real-valued input table is input into at least one multilayer perceptron mixer block to output a human skeleton spatiotemporal dependency information matrix, including:

[0020] The two-dimensional real-valued input table is affine transformed and then input into the channel mixing MLP to extract the spatial dependencies in the skeleton sequence;

[0021] The extracted skeleton sequence spatial dependency is input into the first SE block, the influence of time steps is automatically weighted, and the two-dimensional real-valued input table is combined with the output of the first SE block through the first jump connection layer to obtain the human skeleton spatial dependency matrix;

[0022] The spatial dependency matrix of the human skeleton is transposed and affine transformed in sequence, and then input into the labeled hybrid MLP to learn the temporal information of the skeleton sequence. The output of the labeled hybrid MLP is transposed to obtain the temporal dependency of the skeleton sequence.

[0023] The obtained skeleton sequence temporal dependency is input into the second SE block, the influence of the time step is automatically weighted, and the human skeleton spatial dependency matrix is combined with the output of the second SE block through the second jump connection layer to obtain the human skeleton spatiotemporal dependency information matrix;

[0024] If there are multiple multi-layer perceptron mixer blocks, the obtained human skeleton spatiotemporal dependency information matrix is input into the next multi-layer perceptron mixer block, and the above steps are repeated until the final human skeleton spatiotemporal dependency information matrix is output.

[0025] Furthermore, the channel-mixed MLP includes a fourth fully connected layer, a first GELU activation function, a first Dropout layer, and a fifth fully connected layer connected in sequence. The two-dimensional real-valued input table is affine transformed and then input into the channel-mixed MLP to extract spatial dependencies in the skeleton sequence, including:

[0026] The fine-grained spatial dependencies between body joints are learned by acting on the columns of a two-dimensional real-valued input table, where each column encodes the spatial information of one time step. The computational flow is:

[0027] Y1=W2σ1(W1Aff(Y))

[0028] Where Y1 is the spatial dependency of the skeleton sequence, W1∈R C×C , W2∈R C×C , C is the hidden dimension, σ1(·) is the first GELU activation function, Aff is the affine transformation, and Y is a two-dimensional real-valued input table.

[0029] Furthermore, the first SE block includes a first global maximum pooling layer, a sixth fully connected layer, a first ReLU activation function, a seventh fully connected layer, and a first Sigmoid activation function. The extracted skeleton sequence spatial dependency is input into the first SE block, and the influence of the time step is automatically weighted. The calculation process is:

[0030]

[0031] in is the output of the first SE block, δ1(·) and σ R1(·) are the first Sigmoid activation function and the first ReLU activation function, respectively, and the weight W s ∈R s×e and W e ∈R e×s Shared between spatial and temporal hybrid units, s and e are the number of joints and frames respectively;

[0032] The two-dimensional real-valued input table is combined with the output of the first SE block through the first jump connection layer to obtain the human skeleton spatial dependency matrix. The calculation process is:

[0033]

[0034] where Y s is the human skeleton spatial dependency matrix.

[0035] Furthermore, the labeled hybrid MLP includes an eighth fully connected layer, a second GELU activation function, a second Dropout layer, and a ninth fully connected layer. The human skeleton spatial dependency matrix is sequentially transposed and affine transformed and then input into the labeled hybrid MLP to learn the skeleton sequence time information, and the output of the labeled hybrid MLP is transposed. The calculation process is as follows:

[0036] Y2=(W4σ2(W3Aff(Y s T ))) T

[0037] Where Y2 is the time dependency of the skeleton sequence, T h is the time step, Y s T Y s The transpose of , σ2(v) is the second GELU activation function.

[0038] Furthermore, the second SE block includes a second global maximum pooling layer, a tenth fully connected layer, a second ReLU activation function, an eleventh fully connected layer, and a second Sigmoid activation function. The obtained skeleton sequence time dependency is input into the second SE block, and the influence of the time step is automatically weighted. The calculation process is:

[0039]

[0040] in is the output of the second SE block, δ2(·) and σ R2 (v) are the second Sigmoid activation function and the second ReLU activation function, respectively, with weights W s ∈R s×e and W e ∈R e×sShared between spatial and temporal hybrid units, s and e are the number of joints and frames respectively;

[0041] The human skeleton spatial dependency matrix is combined with the output of the second SE block through the second jump connection layer to obtain the human skeleton spatiotemporal dependency information matrix. The calculation process is:

[0042]

[0043] where Y t is the spatiotemporal dependency information matrix of the human skeleton.

[0044] Furthermore, the second spatiotemporal separable graph convolutional network includes a second residual block, a second graph convolutional network, and a second temporal convolutional network. Inputting the human skeleton spatiotemporal dependency information matrix into the second spatiotemporal separable graph convolutional network and outputting a predicted second joint coordinate sequence includes:

[0045] The input human skeleton spatiotemporal dependency information matrix Y is obtained through the convolutional layer and normalization layer in the second residual block. t Encode to get residual (n) ;

[0046] The human skeleton spatiotemporal dependency information matrix Y t At the same time, the data are input into the second graph convolutional network, and the Einstein summation is performed on the temporal adjacency matrix and the spatial adjacency matrix in turn;

[0047] The summation result is input into the second time convolutional network, and the future joint coordinates X are predicted using the spatiotemporal representation. (n) ;

[0048] Output the second joint coordinate sequence Y (n) =X (n) +residual (n) .

[0049] Furthermore, the second joint coordinate sequence is sequentially fed into the second fully connected layer and the third fully connected layer to output a predicted 3D skeleton joint point sequence, including:

[0050] The second joint coordinate sequence Y is transformed into (n) The hidden dimension C is transformed into K = 3*J, and Y is transformed into (n) Each Projected to a time length T f The future 3D skeleton joint point sequence is obtained based on the vector:

[0051]

[0052] in bp1 ∈R 3×J ,b p2 ∈R C is the weight of the fully connected layer.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] The present invention learns the spatiotemporal 3D body posture dependencies by sequentially mixing two modes. This is a simple and effective human motion prediction network that contains only three components: a fully connected layer, an affine transformation, and a transposition operation, achieving state-of-the-art performance with far fewer parameters than existing methods. The multi-layer perceptron mixer is a sequence-to-sequence model with three main modules: posture embedding, spatiotemporal mixing, and posture prediction. Pose embedding and spatiotemporal mixing are coupled together to encode the spatiotemporal dependencies of human joints. Pose embedding consists of a spatiotemporal separable graph convolution and a fully connected layer to encode a given historical 3D posture and pre-learn the spatiotemporal relationship of a given historical sequence. In the spatiotemporal mixing, each posture is first characterized by an affine transformation. Linear transformation is performed and repeated multi-layer perceptron mixer blocks are given, each block contains two multi-layer perceptrons with skip connections; in a single spatiotemporal MLP, two mixing operations are used to simulate the interaction of body joints over time, spatial mixing allows the interaction between joint spatial positions, while temporal mixing allows the long-term interaction of observed motions; in posture prediction, the output of the mixture is finally aggregated into a global vector and input into the MLP to predict future motion; the human 3D skeleton motion prediction method based on multi-layer perceptron proposed in this invention can be modeled in a simple way without explicitly fusing spatial and temporal information. As an extreme example, a single fully connected layer can already achieve reasonable performance.

[0055] The present invention improves the state-of-the-art in short-term and long-term prediction by introducing a multi-layer perceptron network and a spatiotemporal separable graph convolutional network when capturing time-space domain features. The network has low computational complexity, short computation time, and high efficiency. It is highly robust to difficulties such as scale differences and motion complexity differences among different human bodies, and has strong generalization capabilities for similar motion categories. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 This is a flow chart of the method for predicting 3D human skeleton motion based on a multi-layer perceptron of the present invention;

[0057] Figure 2 Schematic diagram of the first spatiotemporal separable graph convolutional network in the present invention;

[0058] Figure 3 Schematic diagram of the first squeeze-and-excitation (SE) block in the present invention;

[0059] Figure 4 This is a 3D skeleton motion prediction map of the human body. DETAILED DESCRIPTION

[0060] The present invention will be further described below in conjunction with specific examples. The following examples are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0061] The present invention provides a method for predicting 3D human skeleton motion based on a multi-layer perceptron. Figure 1 As shown, the following steps are included:

[0062] Step 1: Input the 3D skeleton joint point sequence data into the first spatiotemporal separable graph convolutional network and output the predicted first joint coordinate sequence;

[0063] The first spatiotemporal separable graph convolutional network (STS-GCN) is to transform the adjacency matrix A st Decomposed into the time adjacency matrix A t and the spatial adjacency matrix A s , calculated as:

[0064] H (l+1) =σ(A s-(l) A t-(l) H (l) W (l) )

[0065] Where A st-(l) ∈R VT×VT is the spatiotemporal adjacency matrix of layer l, A s-(l) is the spatial adjacency matrix of layer l, A t-(l) is the temporal adjacency matrix of layer l, is the convolution weight of the trainable graph of layer l, which transforms each graph node from C (l) Dimension projection to C (l+1) dimension, σ is an activation function such as ReLU, PReLU or tanh; H (l) It is equal to the output of the l-1th layer graph convolution, V is the number of joint points, and T is the number of input frames.

[0066] like Figure 2 As shown, the first spatiotemporal separable graph convolutional network includes a first residual block, a first graph convolutional network (GCN), and a first temporal convolutional network (TCN). The first residual block contains a two-dimensional convolutional layer and a BatchNorm normalization layer, and the first temporal convolutional network includes a convolutional layer, a normalization layer, and a dropout layer.

[0067] The human skeleton motion information exists in the form of a sequence, and the input and output of the network are both human skeleton coordinate sequence information.

[0068] Human motion is defined as T h +T f A sequence of consecutive frames, where each frame parameterizes the angle or 3D coordinate of a human joint. To the current time step T h Historical motion sequence, 3D human body posture x t ∈R 3×J , J is the number of joints in the human skeleton, T f The length of the predicted future time.

[0069] 3D skeleton joint point sequence data Input into the first spatiotemporal separable graph convolutional network, firstly, the body joint coordinates observed in the given input sequence are encoded by the convolutional layer and normalization layer in the first residual block to obtain the residual; at the same time, the posture sequence is input into the first graph convolutional network, and the temporal adjacency matrix A is sequentially t and the spatial adjacency matrix A s Perform Einstein summation. Finally, the summation result is input into the first time convolution network, and the spatiotemporal representation is used to predict the future joint coordinates. In this way, the temporal and spatial relationship of human body motion posture is pre-learned, and the output result is the first joint coordinate sequence

[0070] Step 2: Linearly project the first joint coordinate sequence to the hidden dimension through the first fully connected layer to generate a two-dimensional real-valued input table;

[0071] For the first joint coordinate sequence X obtained in step 1, each time step x t The frame is flattened into a vector of length K = 3*J, which gives a two-dimensional tensor One of the time dimensions T h and a spatial dimension K. For simplicity, the subscript T is omitted h , thus replacing X with Each body skeleton x t ∈R K Through a single fully connected layer, linear projection is performed to the hidden dimension C to learn the pose embedding, generating a two-dimensional real-valued input table.

[0072] Among them, the first fully connected layer is used to extract the predicted skeleton coordinate two-dimensional real-valued input table for posture embedding, and the calculation method is:

[0073] Y=W0X+b0

[0074] in is the weight of the fully connected layer.

[0075] Step 3: Input the two-dimensional real-valued input table into at least one multi-layer perceptron (MLP) mixer block, and output the human skeleton spatiotemporal dependency information matrix;

[0076] There may be one or more multilayer perceptron mixer blocks, each of which includes a channel mixing MLP, a label mixing MLP, a first jump connection layer, a second jump connection layer, a first SE block, and a second SE block.

[0077] If there is only one multilayer perceptron mixer block, such as Figure 1 As shown, this step specifically includes:

[0078] Step 301: Affine transform the two-dimensional real-valued input table and input it into the channel mixing MLP to extract the spatial dependency in the skeleton sequence;

[0079] Perform affine transformation on the two-dimensional real-value input table, perform data normalization, and output the normalized skeleton coordinate information Aff(Y). The calculation method is:

[0080] Aff α,β (x) = Diag(α)x + β

[0081] Where α and β are learnable weight vectors, Aff is the affine transformation, and Diag is the diagonal matrix.

[0082] The skeleton coordinate information Aff(Y) after affine transformation is input to the channel mixing MLP of the multi-layer perceptron mixer block to extract the spatial dependency in the skeleton sequence, such as Figure 1 As shown, the channel mixing MLP includes the fourth fully connected layer, the first GELU activation function layer, the first Dropout layer, and the fifth fully connected layer. The calculation process is

[0083] Y1=W2σ1(W1Aff(Y))

[0084] Where Y1 is the spatial dependency of the skeleton sequence, W1∈R C×C , W2∈R C×C , σ1(·) is the first GELU activation function, and Aff is the affine transformation.

[0085] That is, fine-grained spatial dependencies between body joints are learned by operating on the columns of Aff(Y) after affine transformation, where each column encodes the spatial information of one time step.

[0086] Step 302: Input the extracted skeleton sequence spatial dependency into the first squeeze-and-excitation (SE) block, automatically weight the influence of time steps, and combine the two-dimensional real-valued input table with the output of the first SE block through the first jump connection layer to obtain the human skeleton spatial dependency matrix;

[0087] like Figure 3 As shown, the first SE block includes the first global maximum pooling layer (Globalpooling), the sixth fully connected layer, the first ReLU activation function, the seventh fully connected layer, and the first Sigmoid activation function.

[0088] The skeleton sequence spatial dependency Y1 extracted based on the channel mixing MLP is input into the first SE block, that is, the first global maximum pooling, the sixth fully connected layer, the first ReLU activation function, the seventh fully connected layer, and the first Sigmoid activation function are performed in sequence to automatically adjust the input importance and help the network reweight the influence of each time step. The calculation process is:

[0089]

[0090] in is the output of the first SE block, δ1(·) and σ R1 (·) are the first Sigmoid activation function and the first ReLU activation function, respectively, and the weight W s ∈R s×e and W e ∈R e×s Shared between spatial and temporal hybrid units, s and e are the number of joints and frames respectively.

[0091] After step 2, the first hop connection layer is introduced, such as Figure 1 As shown, the two-dimensional real-valued input table Y is connected to the output of the first SE block through the first jump connection layer Combined, we get the human skeleton spatial dependency matrix Y s , the calculation process is:

[0092]

[0093] Step 303: The spatial dependency matrix of the human skeleton is sequentially transposed and affine transformed, and then input into the labeled hybrid MLP to learn the temporal information of the skeleton sequence. The output of the labeled hybrid MLP is transposed to obtain the temporal dependency of the skeleton sequence.

[0094] like Figure 1 As shown, the labeled hybrid MLP includes the eighth fully connected layer, the second activation function layer GELU, the second Dropout layer and the ninth fully connected layer.

[0095] First, the human skeleton spatial dependency matrix Y s Perform a transpose operation, then an affine transformation, and then input it into the labeled hybrid MLP to learn the skeleton sequence time information. Then, perform a transpose operation on the output of the labeled hybrid MLP. The calculation process is as follows:

[0096] Y2=(W4σ2(W3Aff(Y s T ))) T

[0097] Where Y2 is the time dependency of the skeleton sequence, Y s T Y s The transpose of , σ2(·) is the second GELU activation function.

[0098] By acting on the affine transformation Aff(Y s T ) columns to learn fine-grained temporal dependencies between body joints. Each linear operator in temporal mixing assigns each time step to a linear combination of all frames, where the linear weights depend on the frame position. Therefore, temporal information is maintained in each mixing step, allowing the model to capture long-term dependencies by exploiting long-term interactions between frames.

[0099] Step 304: Input the obtained skeleton sequence temporal dependency into the second SE block, automatically weight the influence of the time step, and combine the human skeleton spatial dependency matrix with the output of the second SE block through the second jump connection layer to obtain the human skeleton spatiotemporal dependency information matrix;

[0100] The structure of the second SE block is the same as that of the first SE block. The second SE block includes a second global maximum pooling layer, a tenth fully connected layer, a second ReLU activation function, an eleventh fully connected layer, and a second Sigmoid activation function.

[0101] The output Y2 of step 303 is input to the second SE block (i.e., the second global maximum pooling, the tenth fully connected layer, the second ReLU activation function, the eleventh fully connected layer, and the second Sigmoid activation function are performed in sequence) to automatically weight the influence of the time step. The calculation process is:

[0102]

[0103] in is the output of the second SE block, δ2(·) and σ R2 (·) are the second Sigmoid activation function and the second ReLU activation function, respectively, and the weight W s ∈R s×e and W e ∈R e×s Shared between spatial and temporal hybrid units, s and e are the number of joints and frames respectively.

[0104] In step 302, a second hop connection layer is introduced, such as Figure 1 As shown, the human skeleton spatial dependency matrix Y is transformed intos With the output of the second SE block Combined, we get the human skeleton spatiotemporal dependency information matrix Y t , the calculation process is:

[0105]

[0106] If there are multiple multilayer perceptron mixer blocks, the obtained human skeleton spatiotemporal dependency information matrix Y t Input the next multi-layer perceptron mixer block, repeat steps 301 to 304, and continue to input the obtained new human skeleton spatiotemporal dependency information matrix into the next multi-layer perceptron mixer block until the final human skeleton spatiotemporal dependency information matrix is output.

[0107] Step 4: Input the human skeleton spatiotemporal dependency information matrix into the second spatiotemporal separable graph convolutional network and output the predicted second joint coordinate sequence;

[0108] The second spatiotemporal separable graph convolutional network is the same as the first spatiotemporal separable graph convolutional network, which also converts the adjacency matrix A st Decomposed into the time adjacency matrix A t and the spatial adjacency matrix A s Its structure is the same as the first spatiotemporal separable graph convolutional network, including a second residual block, a second graph convolutional network, and a second temporal convolutional network. The second residual block contains a two-dimensional convolutional layer and a BatchNorm normalization layer, and the second temporal convolutional network includes a convolutional layer, a normalization layer, and a Dropout layer.

[0109] The human skeleton spatiotemporal dependency information matrix Y t Input into the second spatiotemporal separable graph convolution network, perform spatial convolution calculation, extract high-order features in the spatial domain, and then perform temporal graph convolution operation on the output features of the spatial graph convolution. Specifically, first, the convolution layer and normalization layer in the second residual block are used to calculate the given input Y. t Encode to get residual (n) , and at the same time, Y t Input into the second graph convolutional network and sequentially calculate the time adjacency matrix A t and the spatial adjacency matrix A s Perform Einstein summation, and then input the summation result into the second time convolution network, using the spatiotemporal representation to predict the future joint coordinates X (n) , get the output second joint coordinate sequence Y (n) =X (n) +residual (n) .

[0110] Step 5: Send the second joint coordinate sequence to the second fully connected layer and the third fully connected layer in sequence, and output the predicted 3D skeleton joint point sequence.

[0111] Based on a two-layer nonlinear feedforward network, the second joint coordinate sequence Y is transformed into (n) The hidden dimension C is transformed into K = 3*J, and Y is transformed into (n) Each Projected to a time length T f The mixed features are further propagated on the vector to predict the future 3D skeleton joint point sequence:

[0112]

[0113] in b p1 ∈R 3×J ,b p2 ∈R C is the weight of the fully connected layer.

[0114] Example

[0115] a) Human 3D skeleton pose embedding

[0116] set up To the current time step T h Historical motion sequence, 3D human body posture x t ∈R 3×J , J is the number of joints in the human skeleton. The goal of this application is to learn the historical sequence and future sequences 's mapping.

[0117] Generally speaking, the tensor format of 3D human skeleton data is (B, C, T, N), where B represents the batch input size, C represents the channel dimension, T represents the number of time frames in the human skeleton sequence, and N represents the number of human skeleton joints. Therefore, the input data tensor format is (50, 3, 10, 22).

[0118] Input the 3D skeleton joint sequence data into the first spatiotemporal separable graph convolutional network, as Figure 2 As shown, the body joint coordinates observed in a given input frame are first encoded by the two-dimensional convolution layer and batch normalization layer in the first residual block to obtain residual(50,3,10,22), and the input frame is input into the first graph convolution network to sequentially transform the temporal adjacency matrix A t (22,10,10) and the spatial adjacency matrix A s(10,22,22) performs Einstein summation (einsum), and then inputs the summation result into the first time convolutional network to use the spatiotemporal representation to predict the future joint coordinates Finally, the future first joint coordinate sequence is obtained

[0119] Given the observed first joint coordinate sequence The tensor form is (50,10,66), each time step x t The frame is flattened into a vector of length K = 3*J, J = 22. This results in a two-dimensional tensor One of the time dimensions T h and a spatial dimension K. Replace X with The tensor form is (50,10,66). Each body skeleton x t ∈R K Through a single fully connected layer linear projection to the hidden dimension C = 50, the two-dimensional real-valued input table of the pose embedding is obtained The tensor is in the form of (50,10,50).

[0120] b) Spatial dependency learning of human 3D skeleton

[0121] After obtaining the two-dimensional real-valued input table of the posture embedding, perform the affine transformation Aff α,β (x) = Diag(α)x + β, perform data normalization operation, and output the normalized skeleton coordinate information Aff(Y) in the tensor form of (50, 10, 50).

[0122] The skeleton coordinate information Aff(Y) after affine transformation is input to the channel mixing MLP of the multi-layer perceptron mixer block (such as Figure 1 As shown in the figure, the spatial dependencies in the skeleton sequence are extracted. The channel-mixed MLP contains the fourth fully connected layer, the first GELU activation function layer, the first Dropout layer, and the fifth fully connected layer. The calculation process is Y1=W2σ1(W1Aff(Y)), and the tensor form is (50,10,50). That is, the fine-grained spatial dependencies between body joints are learned by acting on the columns of Aff(Y) after affine transformation. Each column encodes the spatial information of one time step.

[0123] The skeleton sequence spatial dependency Y1 extracted based on the channel mixing MLP is input into the first SE block (e.g. Figure 3 The calculation process is as follows: To automatically adjust the importance of inputs and help the network reweight the impact of each time step.

[0124] After step 2, the first hop connection layer is introduced (such as Figure 1As shown), the human skeleton spatial dependence matrix is obtained The tensor is in the form of (50,10,50).

[0125] c) Human 3D skeleton temporal information learning

[0126] After obtaining the human skeleton space dependency matrix Y s After that, first change Y s Perform a transpose operation, the tensor form is (50,50,10), and then perform an affine transformation Aff α,β (x) = Diag(α)x + β operation, and then input to the label mixing MLP (such as Figure 1 As shown in the figure, the skeleton sequence time information is learned, and then the output of the labeled hybrid MLP is transposed. The tensor form is (50,10,50), and the calculation process is Y2=(W4σ2(W3Aff(Y s T ))) T , that is, by acting on the normalization operation Aff(Y s T ) columns to learn fine-grained temporal dependencies between body joints, maintain temporal information in each mixing step, allow the model to capture long-term dependencies by applying long-term interactions between frames, and finally input into the second SE block for automatic weighting of the influence of time steps. The calculation process is

[0127] After step b), a second hop connection layer (such as Figure 1 As shown), the spatiotemporal dependency information matrix of the human skeleton is obtained The tensor is in the form of (50,10,50).

[0128] d) Human 3D skeleton pose prediction

[0129] After obtaining the human skeleton spatiotemporal dependency information matrix Y t Afterwards, Y t Input into the second spatiotemporal separable graph convolutional network, first pass the two-dimensional convolution layer and BatchNorm normalization layer in the second residual block to the given input Y t Encode to get residual (n) , the data format is (50,10,50), and Y t Input into the second graph convolutional network and sequentially calculate the time adjacency matrix A t (22,10,10) and the spatial adjacency matrix A s (10,22,22) performs Einstein summation (einsum), and then inputs the summation result into the second time convolution network to use the spatiotemporal representation to predict the future joint coordinates X (n)(50,10,50), and get the output result Y (n) =X (n) +residual (n) .

[0130] Based on a two-layer nonlinear feedforward network, the second fully connected layer transforms the hidden dimension C = 50 to 66 = 3 * 22, and the third fully connected layer transforms Y (n) Each Projected to a length T f The mixed features are further propagated on the vector, and the data tensor form changes from (50, 10, 66) to (50, 25, 66), and the future posture prediction sequence is obtained like Figure 4 shown.

[0131] Through the above embodiments, for a given observed motion sequence, the present invention uses a spatiotemporal separable graph convolutional network to pre-learn the time-space relationship of the given human skeleton motion information to obtain a time adjacency matrix and a spatial adjacency matrix, and then linearly projects each body skeleton to the hidden dimension through a single fully connected layer, thereby realizing posture embedding to obtain a two-dimensional real-valued input table; the channel mixing MLP in the multi-layer perceptron mixer first acts on the columns of the two-dimensional real-valued input table, learns the spatial dependency of the human skeleton motion information, and shares it in all columns, and then the label mixing MLP acts on the rows of the two-dimensional real-valued input table, learns the temporal information of the human skeleton motion information, and shares it in all rows; the two-dimensional real-valued input table is used as a class adjacency matrix to perform spatial domain graph convolution calculation, extract high-order features of the spatial domain, and then the output features of the spatial domain graph convolution are subjected to a temporal domain graph convolution operation to obtain the output result; the MLP decoder further propagates the mixed features for future posture prediction. The present invention introduces a multi-layer perceptron network and a spatiotemporal separable graph convolutional network when capturing time-space domain features, which improves the state-of-the-art in short-term and long-term prediction. The network has low computational complexity, short computation time, and high efficiency. It is highly robust to difficulties such as scale differences and motion complexity differences among different human bodies, and has strong generalization capabilities for similar motion categories.

[0132] In another embodiment, the present invention provides a computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions that, when executed by a computing device, cause the computing device to execute the aforementioned human 3D skeleton motion prediction method based on a multi-layer perceptron.

[0133] In another embodiment, the present invention provides a computing device comprising: one or more processors, one or more memories, and one or more programs, wherein the one or more programs are stored in the one or more memories and are configured to be executed by the one or more processors, and the one or more programs include a method for executing the aforementioned human 3D skeleton motion prediction method based on a multi-layer perceptron.

[0134] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0135] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0136] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0138] The present invention has been disclosed above with preferred embodiments, which are not intended to limit the present invention. Any technical solutions obtained by adopting equivalent replacement or equivalent transformation solutions fall within the protection scope of the present invention.

Claims

1. A method for predicting human 3D skeleton motion based on a multi-layer perceptron, characterized in that: The following steps are involved: Input the 3D skeleton joint point sequence into the first spatiotemporal separable graph convolutional network and output the predicted first joint coordinate sequence; Linearly project the first joint coordinate sequence to the hidden dimension through the first fully connected layer to generate a two-dimensional real-valued input table; Input the two-dimensional real-valued input table into at least one multi-layer perceptron mixer block, and output a human skeleton spatiotemporal dependency information matrix; Input the human skeleton spatiotemporal dependency information matrix into the second spatiotemporal separable graph convolutional network and output the predicted second joint coordinate sequence; The second joint coordinate sequence is fed into the second fully connected layer and the third fully connected layer in sequence, and the predicted 3D skeleton joint point sequence is output; The multi-layer perceptron mixer block includes a channel mixing MLP, a label mixing MLP, a first jump connection layer, a second jump connection layer, a first SE block and a second SE block. The two-dimensional real-valued input table is input into at least one multi-layer perceptron mixer block to output a human skeleton spatiotemporal dependency information matrix, including: The two-dimensional real-valued input table is affine transformed and then input into the channel mixing MLP to extract the spatial dependencies in the skeleton sequence; The extracted skeleton sequence spatial dependency is input into the first SE block, the influence of time steps is automatically weighted, and the two-dimensional real-valued input table is combined with the output of the first SE block through the first jump connection layer to obtain the human skeleton spatial dependency matrix; The spatial dependency matrix of the human skeleton is transposed and affine transformed in sequence, and then input into the labeled hybrid MLP to learn the temporal information of the skeleton sequence. The output of the labeled hybrid MLP is transposed to obtain the temporal dependency of the skeleton sequence. The obtained skeleton sequence temporal dependency is input into the second SE block, the influence of the time step is automatically weighted, and the human skeleton spatial dependency matrix is combined with the output of the second SE block through the second jump connection layer to obtain the human skeleton spatiotemporal dependency information matrix; If there are multiple multi-layer perceptron mixer blocks, the obtained human skeleton spatiotemporal dependency information matrix is input into the next multi-layer perceptron mixer block, and the above steps are repeated until the final human skeleton spatiotemporal dependency information matrix is output.

2. The method for predicting 3D human skeleton motion based on a multi-layer perceptron according to claim 1, wherein: The first spatiotemporal separable graph convolutional network includes a first residual block, a first graph convolutional network, and a first temporal convolutional network. Inputting a 3D skeleton joint point sequence into the first spatiotemporal separable graph convolutional network and outputting a predicted first joint coordinate sequence includes: The joint coordinates in the input 3D skeleton joint point sequence are encoded through the convolution layer and normalization layer in the first residual block to obtain the residual; The 3D skeleton joint point sequence is input into the first graph convolutional network, and the Einstein summation of the temporal adjacency matrix and the spatial adjacency matrix is performed in turn; The summation result is input into the first time convolutional network, and the future joint coordinates are predicted using the spatiotemporal representation. ; Output the first joint coordinate sequence .

3. The method for predicting 3D human skeleton motion based on a multi-layer perceptron according to claim 1, wherein: The 3D skeleton joint point sequence is , is the time step, 3D human posture , is the number of joint points of the human skeleton, and the predicted first joint coordinate sequence is X= ,in is the spatial dimension, , each time step The frame is flattened to length The first joint coordinate sequence is linearly projected to the hidden dimension through the first fully connected layer to generate a two-dimensional real-valued input table, including: Each body skeleton Through the first fully connected layer linear projection to the hidden dimension C, we get a two-dimensional real-valued input table .

4. The method for predicting 3D human skeleton motion based on a multi-layer perceptron according to claim 1, wherein: The channel-mixed MLP includes a fourth fully connected layer, a first GELU activation function, a first Dropout layer, and a fifth fully connected layer connected in sequence. The two-dimensional real-valued input table is affine transformed and then input into the channel-mixed MLP to extract spatial dependencies in the skeleton sequence, including: The fine-grained spatial dependencies between body joints are learned by acting on the columns of a two-dimensional real-valued input table, where each column encodes the spatial information of one time step. The computational flow is: ； Where, is the spatial dependence of the backbone sequence, , , C is the hidden dimension, is the first GELU activation function, is an affine transformation, is a two-dimensional real-valued input table.

5. The method for predicting 3D human skeleton motion based on a multi-layer perceptron according to claim 4, wherein: The first SE block includes a first global maximum pooling layer, a sixth fully connected layer, a first ReLU activation function, a seventh fully connected layer, and a first Sigmoid activation function. The extracted skeleton sequence spatial dependency is input into the first SE block, and the influence of the time step is automatically weighted. The calculation process is: ； in is the output of the first SE block, and They are the first Sigmoid activation function and the first ReLU activation function, weights and Shared between spatial and temporal hybrid units, s and e are the number of joints and frames respectively; The two-dimensional real-valued input table is combined with the output of the first SE block through the first jump connection layer to obtain the human skeleton spatial dependency matrix. The calculation process is: ； in is the human skeleton spatial dependency matrix.

6. The method for predicting 3D human skeleton motion based on a multi-layer perceptron according to claim 5, characterized in that: The labeled hybrid MLP includes an eighth fully connected layer, a second GELU activation function, a second Dropout layer, and a ninth fully connected layer. The human skeleton spatial dependency matrix is sequentially transposed and affine transformed and then input into the labeled hybrid MLP to learn the skeleton sequence time information and perform a transposition operation on the output of the labeled hybrid MLP. The calculation process is as follows: ； in is the skeleton sequence time dependency, , , is the time step, for The transpose of is the second GELU activation function.

7. The method for predicting 3D human skeleton motion based on a multi-layer perceptron according to claim 6, characterized in that: The second SE block includes a second global maximum pooling layer, a tenth fully connected layer, a second ReLU activation function, an eleventh fully connected layer, and a second Sigmoid activation function. The obtained skeleton sequence time dependency is input into the second SE block, and the influence of the time step is automatically weighted. The calculation process is: ； in is the output of the second SE block, and are the second Sigmoid activation function and the second ReLU activation function, respectively, and the weights and Shared between spatial and temporal hybrid units, s and e are the number of joints and frames respectively; The human skeleton spatial dependency matrix is combined with the output of the second SE block through the second jump connection layer to obtain the human skeleton spatiotemporal dependency information matrix. The calculation process is: ； in is the spatiotemporal dependency information matrix of the human skeleton.

8. The method for predicting 3D human skeleton motion based on a multi-layer perceptron according to claim 7, characterized in that: The second spatiotemporal separable graph convolutional network includes a second residual block, a second graph convolutional network, and a second temporal convolutional network. Inputting the human skeleton spatiotemporal dependency information matrix into the second spatiotemporal separable graph convolutional network and outputting a predicted second joint coordinate sequence includes: The spatiotemporal dependency information matrix of the input human skeleton is obtained through the convolutional layer and normalization layer in the second residual block. Encode to get ; The human skeleton spatiotemporal dependency information matrix At the same time, the data are input into the second graph convolutional network, and the Einstein summation is performed on the temporal adjacency matrix and the spatial adjacency matrix in turn; The summation result is input into the second temporal convolutional network to predict future joint coordinates using the spatiotemporal representation. ; Output the second joint coordinate sequence .

9. The method for predicting 3D human skeleton motion based on a multi-layer perceptron according to claim 8, characterized in that: The second joint coordinate sequence is sequentially fed into the second fully connected layer and the third fully connected layer to output a predicted 3D skeleton joint point sequence, including: The second joint coordinate sequence is transformed into The hidden dimension C is transformed into K=3*J, and the third fully connected layer is used to transform Each Projected to a time length of The future 3D skeleton joint point sequence is obtained based on the vector: ； in is the weight of the fully connected layer.

Images