Human pose prediction method based on transformer and graph convolution network

Abstract

A human pose prediction method based on a Transformer and a graph convolution network, which comprises the following steps: obtaining a three-dimensional human pose dataset, dividing the dataset, constructing a human pose prediction network, training the human pose prediction network, and testing the human pose prediction network. Since the human pose prediction network based on the Transformer and the graph convolution network is adopted, the technical problem that the existing human pose prediction method depends on a hidden variable model and a probabilistic modeling to synthesize possible human motion, the complexity and high dimensionality of the human pose sequence, and the human pose sequence cannot be accurately predicted are solved. The joint use of the Transformer network and the graph convolution network can take into account the local and global spatiotemporal dependency of the human pose sequence, and more effectively extract motion features. The long-range dependency feature extraction capability of the Transformer and the spatial feature modeling capability of the graph convolution are combined, and the accuracy of human pose prediction is improved.

Description

Technical Field

[0001] This invention belongs to the fields of computer vision and deep learning, and specifically relates to a method for predicting human pose. Background Technology

[0002] Human pose prediction refers to predicting future human pose sequences based on previously observed human pose sequences. It has broad application prospects in various fields such as autonomous driving, virtual reality, and intelligent robotics. Traditional human pose prediction methods typically rely on latent variable models and probabilistic modeling to synthesize possible human movements. However, due to the complexity and high dimensionality of human pose sequences, these methods cannot accurately predict them. In recent years, with the rapid development of deep learning, various neural network models such as recurrent neural networks, generative adversarial networks, and graph convolutional networks have been widely applied to human pose prediction.

[0003] Graph convolutional networks (GCNNs) extract features from graph-structured data through convolution operations, effectively capturing dependencies between nodes. In human pose prediction, keypoints in the human skeleton can be modeled as nodes in a graph, and connections between joints are modeled as edges. This modeling approach allows GCNNs to fully utilize the spatial topology of the human skeleton, extracting complex dependencies and spatial features between joints. However, GCNNs only extract spatial features of human joints at each time step, failing to dynamically capture pose changes across different time steps. Transformer networks, on the other hand, utilize self-attention mechanisms to capture long-term dependencies in pose sequences. The combined use of GCNNs and Transformer networks can address both local and global spatiotemporal dependencies in human pose sequences, extracting motion features more effectively.

[0004] In the field of human posture prediction technology, one urgent technical problem to be solved is to provide a human posture prediction method with high accuracy and recognition rate. Summary of the Invention

[0005] The main technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide a human pose prediction method based on Transformer and graph convolutional networks that has high accuracy and recognition rate.

[0006] The technical solution adopted to solve the above technical problems consists of the following steps:

[0007] (1) Obtain a 3D human pose dataset

[0008] Download the Human3.6m dataset from the official website: http: / / vision.imar.ro / human3.6m / description.php

[0009] (2) Split the dataset

[0010] The Human 3.6m dataset contains seven sub-datasets: Action Set S1, Action Set S5, Action Set S6, Action Set S7, Action Set S8, Action Set S9, and Action Set S11. Each sub-dataset contains 15 actions: walking (f1), eating (f2), smoking (f3), discussing (f4), giving directions (f5), greeting (f6), filming (f7), posing (f8), buying (f9), sitting (f11), etc. 10 Sit down 11 Taking photos 12 Waiting for f 13 Walking the dog 14 Let's go together 15 Use action S1, action set S6, action set S7, action set S8, and action set S9 as the training set, and use action set S5 and action set S11 as the test set.

[0011] (3) Constructing a human pose prediction network

[0012] The human pose prediction network consists of an encoder and a decoder connected in series.

[0013] The encoder is composed of a Transformer encoder and a spatial encoder connected in parallel.

[0014] The decoder is composed of a learnable positional coding layer 3, a Transformer coding layer 3, a convolutional layer 1, and a convolutional layer 2 connected in series.

[0015] (4) Training the human posture prediction network

[0016] 1) Constructing the loss function

[0017] The loss function L is constructed as follows: MPJPE :

[0018]

[0019] Where J represents the number of human joints, with a value ranging from 16 to 32, and T represents the number of frames, with a value ranging from 10 to 25. Let x represent the predicted 3D coordinates of joint j in frame t within the real number range. jt This represents the three-dimensional true coordinates of joint j in the t-th frame within the real number range.

[0020] 2) Training the human posture prediction network

[0021] The training set was input into the human pose prediction network for training. The training parameters were: batch size of 16, number of training iterations of 60, training server of GTX 1080, and training until the loss function L was reached. MPJPE convergence.

[0022] (5) Testing the human pose prediction network

[0023] The test set is input into the trained human pose prediction network for testing, and the predicted values ​​of action set S5 and action set S11 are output.

[0024] In step (3) of the present invention, the human pose prediction network is constructed by connecting a learnable position coding layer 1 and a Transformer coding layer 1 in series.

[0025] In step (3) of the present invention, the spatial encoder is composed of a learnable position coding layer 2, a Transformer coding layer 2, and a graph convolutional network connected in series.

[0026] In step (3) of the present invention, the human pose prediction network is constructed. The kernel size of the convolutional layer 1 is 3×3 and the stride is 1. The kernel size of the convolutional layer 2 is 1×1 and the stride is 1.

[0027] The learnable position encoding layer 1 of the present invention includes an embedding layer P e The construction method of the learnable positional coding layer 1 is as follows:

[0028] P e =E[P]

[0029] Where E is the weight matrix, which is initialized with a normal distribution with a mean of -0.05 to 0.05 and a standard deviation of 0.01 to 0.05, P is the position index, and [·] represents the vector mapping.

[0030] The structure and construction method of the learnable position coding layer 2 of the present invention are the same as those of the learnable position coding layer 1.

[0031] The structure and construction method of the learnable position coding layer 3 of the present invention are the same as those of the learnable position coding layer 1.

[0032] The beneficial effects of this invention are as follows:

[0033] This invention employs a human pose prediction network combining Transformer and graph convolutional networks, addressing the technical problem of existing human pose prediction methods relying on latent variable models and probabilistic modeling to synthesize possible human movements. The complexity and high dimensionality of human pose sequences prevent accurate prediction. The combined use of Transformer and graph convolutional networks can consider both local and global spatiotemporal dependencies in human pose sequences, more effectively extracting motion features. By combining the long-range dependency feature extraction capability of Transformer with the spatial feature modeling capability of graph convolution, this invention improves the accuracy of human pose prediction. Attached Figure Description

[0034] Figure 1 This is a flowchart of Embodiment 1 of the present invention.

[0035] Figure 2 This is a schematic diagram of the structure of a human pose prediction network.

[0036] Figure 3 yes Figure 2 A schematic diagram of the encoder structure.

[0037] Figure 4 yes Figure 2 A schematic diagram of the decoder structure. Detailed Implementation

[0038] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the following embodiments.

[0039] Example 1

[0040] like Figure 1 As shown, the human pose prediction method based on Transformer and graph convolutional networks in this embodiment consists of the following steps:

[0041] (1) Obtain a 3D human pose dataset

[0042] Download the Human3.6m dataset from the official website: http: / / vision.imar.ro / human3.6m / description.php

[0043] (2) Split the dataset

[0044] The Human 3.6m dataset contains seven sub-datasets: Action Set S1, Action Set S5, Action Set S6, Action Set S7, Action Set S8, Action Set S9, and Action Set S11. Each sub-dataset contains 15 actions: walking (f1), eating (f2), smoking (f3), discussing (f4), giving directions (f5), greeting (f6), filming (f7), posing (f8), buying (f9), sitting (f11), etc.10 Sit down 11 Taking photos 12 Waiting for f 13 Walking the dog 14 Let's go together 15 Use action S1, action set S6, action set S7, action set S8, and action set S9 as the training set, and use action set S5 and action set S11 as the test set.

[0045] (3) Constructing a human pose prediction network

[0046] Figure 2 A schematic diagram of the human pose prediction network is given. Figure 2 In this embodiment, the human pose prediction network consists of an encoder and a decoder connected in series.

[0047] Figure 3 Given Figure 2 A schematic diagram of the encoder structure is shown in the figure. Figure 3 In this embodiment, the encoder is composed of a Transformer encoder and a spatial encoder connected in parallel.

[0048] The Transformer encoder in this embodiment is composed of a learnable position encoding layer 1 and a Transformer encoding layer 1 connected in series; the spatial encoder is composed of a learnable position encoding layer 2, a Transformer encoding layer 2, and a graph convolutional network connected in series in sequence.

[0049] The learnable position encoding layer 1 in this embodiment includes one embedding layer P. e The construction method of the learnable positional coding layer 1 is as follows:

[0050] P e =E[P]

[0051] Where E is the weight matrix, which is initialized by a normal distribution with a mean of -0.05 to 0.05 and a standard deviation of 0.01 to 0.05. In this embodiment, the mean is 0 and the standard deviation is 0.02. P is the position index, and [·] represents the vector mapping.

[0052] The structure and construction method of the learnable position coding layer 2 and learnable position coding layer 3 in this embodiment are the same as those of the learnable position coding layer 1.

[0053] The spatial encoder in this embodiment is composed of a learnable position coding layer 2, a Transformer coding layer 2, and a graph convolutional network connected in series.

[0054] Figure 4 Given Figure 2 A schematic diagram of the decoder structure is shown in the diagram. Figure 4 In this embodiment, the decoder is composed of a learnable positional coding layer 3, a Transformer coding layer 3, a convolutional layer 1, and a convolutional layer 2 connected in series. The convolutional kernel size of the convolutional layer 1 is 3×3 with a stride of 1, and the convolutional kernel size of the convolutional layer 2 is 1×1 with a stride of 1.

[0055] This invention employs a human pose prediction network combining Transformer and graph convolutional networks, addressing the technical problem of existing human pose prediction methods relying on latent variable models and probabilistic modeling to synthesize possible human movements. The complexity and high dimensionality of human pose sequences prevent accurate prediction. The combined use of Transformer and graph convolutional networks can consider both local and global spatiotemporal dependencies in human pose sequences, more effectively extracting motion features. By combining the long-range dependency feature extraction capability of Transformer with the spatial feature modeling capability of graph convolution, this invention improves the accuracy of human pose prediction.

[0056] (4) Training the human posture prediction network

[0057] 1) Constructing the loss function

[0058] The loss function L is constructed as follows: MPJPE :

[0059]

[0060] in, Let x represent the predicted 3D coordinates of joint j in frame t within the real number range. jt Let J represent the three-dimensional real coordinates of joint j in frame t within the real number range, where J is the number of human joints and takes values ​​from 16 to 32. In this embodiment, J takes a value of 22. T is the number of frames and takes values ​​from 10 to 25. In this example, T takes a value of 15.

[0061] 2) Training the human posture prediction network

[0062] The training set was input into the human pose prediction network for training. The training parameters were: batch size of 16, number of training iterations of 60, training server of GTX 1080, and training until the loss function L was reached. MPJPE convergence.

[0063] (5) Testing the human pose prediction network

[0064] The test set is input into the trained human pose prediction network for testing, and the predicted values ​​of action set S5 and action set S11 are output.

[0065] Complete a human pose prediction method based on Transformer and graph convolutional networks.

[0066] Example 2

[0067] The human pose prediction method based on Transformer and graph convolutional networks in this embodiment consists of the following steps:

[0068] (1) Obtain a 3D human pose dataset

[0069] The steps are the same as in Example 1.

[0070] (2) Split the dataset

[0071] The steps are the same as in Example 1.

[0072] (3) Constructing a human pose prediction network

[0073] The structure of the human pose prediction network is the same as that in Example 1.

[0074] The learnable position encoding layer 1 in this embodiment includes one embedding layer P. e The construction method of the learnable positional coding layer 1 is as follows:

[0075] P e =E[P]

[0076] Where E is the weight matrix, which is initialized by a normal distribution with a mean of -0.05 to 0.05 and a standard deviation of 0.01 to 0.05. In this embodiment, the mean is -0.05 and the standard deviation is 0.01. P is the position index, and [·] represents the vector mapping.

[0077] (4) Training the human posture prediction network

[0078] 1) Constructing the loss function

[0079] The expression for the loss function is the same as in Example 1.

[0080] In the expression of the loss function, Let x represent the predicted 3D coordinates of joint j in frame t within the real number range. jt Let J represent the three-dimensional real coordinates of joint j in frame t within the real number range, where J is the number of human joints and takes values ​​from 16 to 32. In this embodiment, J takes the value of 16. T is the number of frames and takes values ​​from 10 to 25. In this example, T takes the value of 10.

[0081] The other steps in this procedure are the same as in Example 1.

[0082] The other steps are the same as in Example 1. This completes the human pose prediction method based on Transformer and graph convolutional networks.

[0083] Example 3

[0084] The human pose prediction method based on Transformer and graph convolutional networks in this embodiment consists of the following steps:

[0085] (1) Obtain a 3D human pose dataset

[0086] The steps are the same as in Example 1.

[0087] (2) Split the dataset

[0088] The steps are the same as in Example 1.

[0089] (3) Constructing a human pose prediction network

[0090] The structure of the human pose prediction network is the same as that in Example 1.

[0091] The learnable position encoding layer 1 in this embodiment includes one embedding layer P. e The construction method of the learnable positional coding layer 1 is as follows:

[0092] P e =E[P]

[0093] Where E is the weight matrix, which is initialized by a normal distribution with a mean of -0.05 to 0.05 and a standard deviation of 0.01 to 0.05. In this embodiment, the mean is 0.05 and the standard deviation is 0.05. P is the position index, and [·] represents the vector mapping.

[0094] (4) Training the human posture prediction network

[0095] 1) Constructing the loss function

[0096] The expression for the loss function is the same as in Example 1.

[0097] In the expression of the loss function, Let x represent the predicted 3D coordinates of joint j in frame t within the real number range. jt Let J represent the three-dimensional real coordinates of joint j in frame t within the real number range, where J is the number of human joints and takes values ​​from 16 to 32. In this embodiment, J takes a value of 32. T is the number of frames and takes values ​​from 10 to 25. In this example, T takes a value of 25.

[0098] The other steps in this procedure are the same as in Example 1.

[0099] The other steps are the same as in Example 1. This completes the human pose prediction method based on Transformer and graph convolutional networks.

[0100] To verify the beneficial effects of the present invention, the inventors conducted comparative experiments using the human pose prediction method based on Transformer and graph convolutional networks from Embodiment 1 of the present invention, a pose prediction method based on spatiotemporally separable graph convolutional networks (hereinafter referred to as STS-GCN), and a human motion prediction method based on 3D skeleton aided by auxiliary tasks (hereinafter referred to as Auxformer). The various experimental results are shown in Table 1 below:

[0101] The evaluation index is determined to be the average position error ε:

[0102]

[0103] in, For the predicted three-dimensional coordinates in the real number range, The value of ε represents the three-dimensional true coordinates within the real number range. The smaller the value of ε, the higher the accuracy of the prediction.

[0104] Table 1. Experimental results of the method in Example 1 and the comparative experimental method.

[0105]

[0106] As shown in Table 1, the method in Example 1 has an average position error of 8.0 when the prediction time is 80 milliseconds, which is lower than that of the STS-GCN and Auxformer methods. The method in Example 1 has an average position error of 30.2 when the prediction time is 320 milliseconds, which is also lower than that of the STS-GCN and Auxformer methods. The method in Example 1 has an average position error of 49.5 when the prediction time is 560 milliseconds, which is also lower than that of the STS-GCN and Auxformer methods. The method in Example 1 has an average position error of 74.9 when the prediction time is 1000 milliseconds, which is also lower than that of the STS-GCN and Auxformer methods. Experiments show that the method of the present invention has higher accuracy in predicting human pose.

Claims

1. A human pose prediction method based on Transformer and graph convolutional networks, characterized in that... It consists of the following steps: (1) Obtain a 3D human pose dataset Download the Human 3.6m dataset from the official website: http: / / vision.imar.ro / human3.6m / description.php; (2) Split the dataset The Human 3.6m dataset contains seven sub-datasets: action set S1, action set S5, action set S6, action set S7, action set S8, action set S9, and action set S11. Each sub-dataset contains 15 actions, including walking. ,Have a meal Smoking ,discuss Directions ,greet Camera posing ,Buy ,sit ,sit down ,Photograph ,wait Walking the dog Let's go together Use action S1, action set S6, action set S7, action set S8, and action set S9 as the training set, and use action set S5 and action set S11 as the test set. (3) Constructing a human pose prediction network Human pose prediction networks consist of encoders and decoders connected in series. The encoder is composed of a Transformer encoder and a spatial encoder connected in parallel; The Transformer encoder is composed of a learnable position coding layer 1 and a Transformer coding layer 1 connected in series; The spatial encoder is composed of a learnable position coding layer 2, a Transformer coding layer 2, and a graph convolutional network connected in series. The learnable position encoding layer 1 includes an embedding layer. The construction method of the learnable positional coding layer 1 is as follows: in, This is the weight matrix. The data were obtained from a normal distribution with a mean of -0.05 to 0.05 and a standard deviation of 0.01 to 0.

05. Initialize, where P is the position index. Represents a vector mapping; The decoder is composed of a learnable positional coding layer 3, a Transformer coding layer 3, a convolutional layer 1, and a convolutional layer 2 connected in series. (4) Training the human posture prediction network 1) Constructing the loss function Construct the loss function as follows : Where J represents the number of human joints, with a value ranging from 16 to 32, and T represents the number of frames, with a value ranging from 10 to 25. This represents the predicted 3D coordinates of joint j in frame t within the real number range. Represents the true three-dimensional coordinates of joint j in the t-th frame within the real number range; 2) Training the human posture prediction network The training set was input into the human pose prediction network for training. The training parameters were: batch size of 16, number of training iterations of 60, training server of GTX 1080, and training until the loss function was defined. convergence; (5) Testing the human pose prediction network The test set is input into the trained human pose prediction network for testing, and the predicted values ​​of action set S5 and action set S11 are output.

2. The human pose prediction method based on Transformer and graph convolutional networks according to claim 1, characterized in that: In step (3) of constructing the human pose prediction network, the kernel size of the convolutional layer 1 is 3×3 and the stride is 1, and the kernel size of the convolutional layer 2 is 1×1 and the stride is 1.

3. The human pose prediction method based on Transformer and graph convolutional networks according to claim 1, characterized in that: The structure and construction method of the learnable position coding layer 2 are the same as those of the learnable position coding layer 1.

4. The human pose prediction method based on Transformer and graph convolutional networks according to claim 1, characterized in that: The structure and construction method of the learnable position coding layer 3 are the same as those of the learnable position coding layer 1.

Images