Human body posture prediction method based on Transform and graph convolutional network
By combining the human posture prediction method of Transformer and graph convolutional network, the problem of low prediction accuracy and recognition rate caused by the complexity and high dimensionality of human posture sequence in the prior art is solved, and a higher pose prediction accuracy is achieved and recognition rate.
Patent Information
- Application Number
- CN202411067398.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-06
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-06
AI Technical Summary
The prior art cannot accurately predict human posture sequences, especially due to the complexity and high dimensionality of human posture sequences, resulting in low prediction accuracy and recognition rate.
Using the combined method of Transformer and graph convolution network, the human posture prediction network formed by an encoder and decoder is extracted by combining Transformer's long-range dependency feature extraction capability and spatial feature modeling capability of graph convolution to extract the local and local posture sequences of human posture sequences. Global time and space dependency.
It improves the accuracy of human posture prediction, can extract motion characteristics more effectively, and achieve high accuracy and high recognition rate prediction.
Smart Images

Figure CN119091499A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer vision and deep learning, and specifically relates to a method for predicting human body posture. Background Art
[0002] Human posture prediction refers to predicting the human posture sequence in the future based on the human posture sequence observed in the past. It has broad application prospects in many fields such as autonomous driving, virtual reality and intelligent robots. Traditional human posture prediction methods usually rely on latent variable models and probabilistic modeling to synthesize possible human movements, but due to the complexity and high dimensionality of human posture sequences, these methods cannot accurately predict human posture sequences. In recent years, with the rapid development of deep learning, various neural network models such as recurrent neural networks, generative adversarial networks and graph convolutional networks have been widely used in human posture prediction.
[0003] Graph convolutional networks extract features of graph structured data through convolution operations and effectively capture dependencies between nodes. In human posture prediction, key points of the human skeleton can be modeled as nodes on the graph, and connections between joints are modeled as edges on the graph. This modeling method enables graph convolutional networks to fully utilize the spatial topological relationship of the human skeleton and extract complex dependencies and spatial features between joints. However, graph convolutional networks only extract spatial features of human joints at each time step and cannot dynamically capture posture changes at different time steps. Transformer networks can use self-attention mechanisms to capture long-term dependencies of posture sequences. The joint use of graph convolutional networks and transformer networks can take into account local and global spatiotemporal dependencies of human posture sequences and extract motion features more effectively.
[0004] In the field of human posture prediction technology, a technical problem that needs to be urgently solved is to provide a human posture prediction method with accurate prediction and high recognition rate. Summary of the invention
[0005] The main technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a human posture prediction method based on Transformer and graph convolutional network with accurate prediction and high recognition rate.
[0006] The technical solution adopted to solve the above technical problems consists of the following steps:
[0007] (1) Obtaining a 3D human posture dataset
[0008] Download the Human3.6m dataset from the official website http: / / vision.imar.ro / human3.6m / description.php.
[0009] (2) Dividing the Dataset
[0010] The Human 3.6m dataset contains 7 sub-datasets: Action Set S1, Action Set S5, Action Set S6, Action Set S7, Action Set S8, Action Set S9, Action Set S11. Each sub-dataset contains 15 actions: walking f1, eating f2, smoking f3, discussing f4, pointing f5, greeting f6, taking pictures f7, posing f8, buying f9, sitting f10, etc. 10 , sit down 11 , take photos 12 , Wait for 13 , Walking the dog 14 Let's go together 15 ; Use action S1, action set S6, action set S7, action set S8, and action set S9 as training sets, and use action set S5 and action set S11 as test sets.
[0011] (3) Constructing a human posture prediction network
[0012] The human posture prediction network consists of an encoder and a decoder connected in series.
[0013] The encoder is composed of a Transformer encoder and a spatial encoder in parallel.
[0014] The decoder is composed of a learnable position encoding layer 3 and a Transformer encoding layer 3, a convolutional layer 1, and a convolutional layer 2 connected in series in sequence.
[0015] (4) Training the human posture prediction network
[0016] 1) Construct loss function
[0017] The loss function L is constructed as follows MPJPE :
[0018]
[0019] Among them, J is the number of human joints, J value ranges from 16 to 32, T is the number of frames, T value ranges from 10 to 25, represents the three-dimensional predicted coordinates of joint j in the tth frame in the real number range, x jt Represents the three-dimensional real coordinates of joint j in the t-th frame in the real number range.
[0020] 2) Training human posture prediction network
[0021] The training set is input into the human posture prediction network for training. The training parameters are: training batch is 16, training iteration number is 60, training server is GTX 1080, training to loss function L MPJPE convergence.
[0022] (5) Test the human posture prediction network
[0023] The test set is input into the trained human posture prediction network for testing, and the predicted values of action set S5 and action set S11 are output.
[0024] In step (3) of constructing a human posture prediction network of the present invention, the Transformer encoder is composed of a learnable position encoding layer 1 and a Transformer encoding layer 1 connected in series.
[0025] In step (3) of constructing a human posture prediction network of the present invention, the spatial encoder is composed of a learnable position encoding layer 2, a Transformer encoding layer 2, and a graph convolutional network connected in series in sequence.
[0026] In the step (3) of constructing the human posture prediction network of the present invention, the convolution kernel size of the convolution layer 1 is 3×3, the step size is 1, and the convolution kernel size of the convolution layer 2 is 1×1, the step size is 1.
[0027] The learnable position encoding layer 1 of the present invention comprises an embedding layer P e , the construction method of the learnable position encoding layer 1 is as follows:
[0028] P e =E[P]
[0029] Where E is the weight matrix, E is initialized by a normal distribution with a mean of -0.05 to 0.05 and a standard deviation of 0.01 to 0.05, P is the position index, and [·] represents vector mapping.
[0030] The structure and construction method of the learnable position encoding layer 2 of the present invention are the same as those of the learnable position encoding layer 1.
[0031] The structure and construction method of the learnable position encoding layer 3 of the present invention are the same as those of the learnable position encoding layer 1.
[0032] The beneficial effects of the present invention are as follows:
[0033] Since the present invention adopts the human posture prediction network of Transformer and graph convolutional network, it solves the technical problem that the existing human posture prediction method relies on latent variable model and probabilistic modeling to synthesize possible human body movements, the complexity and high dimensionality of human posture sequence, and cannot accurately predict the human posture sequence. The combined use of Transformer network and graph convolutional network can take into account the local and global spatiotemporal dependency of human posture sequence and extract motion features more effectively. The present invention improves the accuracy of human posture prediction by combining the long-range dependency feature extraction capability of Transformer and the spatial feature modeling capability of graph convolution. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 It is a flow chart of embodiment 1 of the present invention.
[0035] Figure 2 It is a structural diagram of the human posture prediction network.
[0036] Figure 3 yes Figure 2 Schematic diagram of the encoder structure.
[0037] Figure 4 yes Figure 2 Schematic diagram of the decoder structure. DETAILED DESCRIPTION
[0038] The present invention will be further described in detail with reference to the accompanying drawings and embodiments, but the present invention is not limited to the following embodiments.
[0039] Example 1
[0040] like Figure 1 As shown, the human posture prediction method based on Transformer and graph convolutional network of this embodiment consists of the following steps:
[0041] (1) Obtaining a 3D human posture dataset
[0042] Download the Human3.6m dataset from the official website http: / / vision.imar.ro / human3.6m / description.php.
[0043] (2) Dividing the Dataset
[0044] The Human 3.6m dataset contains 7 sub-datasets: Action Set S1, Action Set S5, Action Set S6, Action Set S7, Action Set S8, Action Set S9, Action Set S11. Each sub-dataset contains 15 actions: walking f1, eating f2, smoking f3, discussing f4, pointing f5, greeting f6, taking pictures f7, posing f8, buying f9, sitting f10, etc.10 , sit down 11 , take photos 12 , Wait for 13 , Walking the dog 14 Let's go together 15 ; Use action S1, action set S6, action set S7, action set S8, and action set S9 as training sets, and use action set S5 and action set S11 as test sets.
[0045] (3) Constructing a human posture prediction network
[0046] Figure 2 The structural diagram of the human posture prediction network is given. Figure 2 In the embodiment, the human body posture prediction network is composed of an encoder and a decoder connected in series.
[0047] Figure 3 Given Figure 2 The structural diagram of the encoder is shown in Figure 3 In the embodiment, the encoder is composed of a Transformer encoder and a spatial encoder in parallel.
[0048] The Transformer encoder of this embodiment is composed of a learnable position encoding layer 1 and a Transformer encoding layer 1 connected in series; the spatial encoder is composed of a learnable position encoding layer 2, a Transformer encoding layer 2, and a graph convolutional network connected in series in sequence.
[0049] The learnable position encoding layer 1 in this embodiment includes an embedding layer P e , the construction method of the learnable position encoding layer 1 is as follows:
[0050] P e =E[P]
[0051] Wherein, E is a weight matrix, E is initialized by a normal distribution with a mean of -0.05 to 0.05 and a standard deviation of 0.01 to 0.05. In this embodiment, the mean is 0 and the standard deviation is 0.02. P is a position index, and [·] represents vector mapping.
[0052] The structures and construction methods of the learnable position coding layer 2 and the learnable position coding layer 3 of this embodiment are the same as those of the learnable position coding layer 1.
[0053] The spatial encoder of this embodiment is composed of a learnable position encoding layer 2, a Transformer encoding layer 2, and a graph convolutional network connected in series.
[0054] Figure 4 Given Figure 2 The schematic diagram of the decoder structure in Figure 4 In the decoder of this embodiment, a learnable position encoding layer 3 and a Transformer encoding layer 3, a convolutional layer 1, and a convolutional layer 2 are sequentially connected in series. The convolution kernel size of the convolutional layer 1 is 3×3, and the step size is 1. The convolution kernel size of the convolutional layer 2 is 1×1, and the step size is 1.
[0055] Since the present invention adopts the human posture prediction network of Transformer and graph convolutional network, it solves the technical problem that the existing human posture prediction method relies on latent variable model and probabilistic modeling to synthesize possible human body movements, the complexity and high dimensionality of human posture sequence, and cannot accurately predict the human posture sequence. The combined use of Transformer network and graph convolutional network can take into account the local and global spatiotemporal dependency of human posture sequence and extract motion features more effectively. The present invention improves the accuracy of human posture prediction by combining the long-range dependency feature extraction capability of Transformer and the spatial feature modeling capability of graph convolution.
[0056] (4) Training the human posture prediction network
[0057] 1) Construct loss function
[0058] The loss function L is constructed as follows MPJPE :
[0059]
[0060] in, represents the three-dimensional predicted coordinates of joint j in the tth frame in the real number range, x jt It represents the three-dimensional real coordinates of joint j in the t-th frame within the real number range, J is the number of human joints, J takes a value of 16 to 32, and the value of J in this embodiment is 22, T is the number of frames, T takes a value of 10 to 25, and the value of T in this example is 15.
[0061] 2) Training human posture prediction network
[0062] The training set is input into the human posture prediction network for training. The training parameters are: training batch is 16, training iteration number is 60, training server is GTX 1080, training to loss function L MPJPE convergence.
[0063] (5) Test the human posture prediction network
[0064] The test set is input into the trained human posture prediction network for testing, and the predicted values of action set S5 and action set S11 are output.
[0065] Completed the human posture prediction method based on Transformer and graph convolutional network.
[0066] Example 2
[0067] The human body posture prediction method based on Transformer and graph convolutional network in this embodiment consists of the following steps:
[0068] (1) Obtaining a 3D human posture dataset
[0069] This step is the same as in Example 1.
[0070] (2) Dividing the Dataset
[0071] This step is the same as in Example 1.
[0072] (3) Constructing a human posture prediction network
[0073] The structure of the human posture prediction network is the same as that of Example 1.
[0074] The learnable position encoding layer 1 in this embodiment includes an embedding layer P e , the construction method of the learnable position encoding layer 1 is as follows:
[0075] P e =E[P]
[0076] Wherein, E is a weight matrix, E is initialized by a normal distribution with a mean of -0.05 to 0.05 and a standard deviation of 0.01 to 0.05. In this embodiment, the mean is -0.05 and the standard deviation is 0.01. P is a position index, and [·] represents vector mapping.
[0077] (4) Training the human posture prediction network
[0078] 1) Construct loss function
[0079] The expression of the loss function is the same as that in Example 1.
[0080] In the expression of the loss function, represents the three-dimensional predicted coordinates of joint j in the tth frame in the real number range, x jt It represents the three-dimensional real coordinates of joint j in the t-th frame within the real number range, J is the number of human joints, J takes a value of 16 to 32, and the value of J in this embodiment is 16, T is the number of frames, T takes a value of 10 to 25, and the value of T in this example is 10.
[0081] The other steps of this step are the same as those in Example 1.
[0082] The other steps are the same as those in Example 1. A human posture prediction method based on Transformer and graph convolutional network is completed.
[0083] Example 3
[0084] The human body posture prediction method based on Transformer and graph convolutional network in this embodiment consists of the following steps:
[0085] (1) Obtaining a 3D human posture dataset
[0086] This step is the same as in Example 1.
[0087] (2) Dividing the Dataset
[0088] This step is the same as in Example 1.
[0089] (3) Constructing a human posture prediction network
[0090] The structure of the human posture prediction network is the same as that of Example 1.
[0091] The learnable position encoding layer 1 in this embodiment includes an embedding layer P e , the construction method of the learnable position encoding layer 1 is as follows:
[0092] P e =E[P]
[0093] Wherein, E is a weight matrix, E is initialized by a normal distribution with a mean of -0.05 to 0.05 and a standard deviation of 0.01 to 0.05. In this embodiment, the mean is 0.05 and the standard deviation is 0.05. P is a position index, and [·] represents vector mapping.
[0094] (4) Training the human posture prediction network
[0095] 1) Construct loss function
[0096] The expression of the loss function is the same as that in Example 1.
[0097] In the expression of the loss function, represents the three-dimensional predicted coordinates of joint j in the tth frame in the real number range, x jt It represents the three-dimensional real coordinates of joint j in the t-th frame within the real number range, J is the number of human joints, J takes a value of 16 to 32, and the value of J in this embodiment is 32, T is the number of frames, T takes a value of 10 to 25, and the value of T in this example is 25.
[0098] The other steps of this step are the same as those in Example 1.
[0099] The other steps are the same as those in Example 1. A human posture prediction method based on Transformer and graph convolutional network is completed.
[0100] In order to verify the beneficial effects of the present invention, the inventors conducted comparative experiments using the human posture prediction method based on Transformer and graph convolutional network of Example 1 of the present invention, the posture prediction method based on spatiotemporal separable graph convolutional network (hereinafter referred to as STS-GCN), and the human motion prediction method based on 3D skeleton with auxiliary tasks (hereinafter referred to as Auxformer). Various experimental conditions are shown in Table 1 below:
[0101] The evaluation index is determined as the average position error ε:
[0102]
[0103] in, are the three-dimensional prediction coordinates in the real number range, is the three-dimensional true coordinate in the real number range. The smaller the value of ε, the higher the prediction accuracy.
[0104] Table 1 Experimental results of the method of Example 1 and the comparative experimental method
[0105]
[0106] As can be seen from Table 1, the average position error of the method in Example 1 is 8.0 when the prediction time is 80 milliseconds, which is lower than the STS-GCN method and the Auxformer method; the average position error of the method in Example 1 is 30.2 when the prediction time is 320 milliseconds, which is lower than the STS-GCN method and the Auxformer method; the average position error of the method in Example 1 is 49.5 when the prediction time is 560 milliseconds, which is lower than the STS-GCN method and the Auxformer method; the average position error of the method in Example 1 is 74.9 when the prediction time is 1000 milliseconds, which is lower than the STS-GCN method and the Auxformer method. Experiments show that the method of the present invention has higher accuracy in human body posture prediction.
Claims
1. A human posture prediction method based on Transformer and graph convolutional network, Features It consists of the following steps: (1) Obtaining a 3D human posture dataset Download the Human 3.6m dataset from the official website http: / / vision.imar.ro / human3.6m / description.php; (2) Dividing the Dataset The Human 3.6m dataset contains 7 sub-datasets: Action Set S1, Action Set S5, Action Set S6, Action Set S7, Action Set S8, Action Set S9, Action Set S11. Each sub-dataset contains 15 actions: walking, 1 , eat 2 , smoking 3 、Discussion 4 、Guide 5 , say hello 6 , Camera 7 , pose 8 、Purchase 9 、Sit 10 , sit down 11 , take photos 12 , Wait for 13 , Walking the dog 14 Let's go together 15 ; Use action S1, action set S6, action set S7, action set S8, action set S9 as training sets, and action set S5 and action set S11 as test sets; (3) Constructing a human posture prediction network The human posture prediction network is composed of an encoder and a decoder in series; The encoder is composed of a Transformer encoder and a spatial encoder in parallel; The decoder is composed of a learnable position encoding layer 3 and a Transformer encoding layer 3, a convolutional layer 1, and a convolutional layer 2 connected in series in sequence; (4) Training the human posture prediction network 1) Construct loss function The loss function L is constructed as follows MPJPE : Among them, J is the number of human joints, J value ranges from 16 to 32, T is the number of frames, T value ranges from 10 to 25, represents the three-dimensional predicted coordinates of joint j in the tth frame in the real number range, x jt represents the three-dimensional real coordinates of joint j in the t-th frame within the real number range; 2) Training human posture prediction network The training set is input into the human posture prediction network for training. The training parameters are: training batch is 16, training iteration number is 60, training server is GTX 1080, training to loss function L MPJPE convergence; (5) Test the human posture prediction network The test set is input into the trained human posture prediction network for testing, and the predicted values of action set S5 and action set S11 are output.
2. According to claim 1, the human posture prediction method based on Transformer and graph convolutional network, Features: In step (3) of constructing a human posture prediction network, the Transformer encoder is composed of a learnable position encoding layer 1 and a Transformer encoding layer 1 connected in series.
3. The human body posture prediction method based on Transformer and graph convolutional network according to claim 1, Features: In step (3) of constructing a human posture prediction network, the spatial encoder is composed of a learnable position encoding layer 2, a Transformer encoding layer 2, and a graph convolutional network connected in series.
4. The human body posture prediction method based on Transformer and graph convolutional network according to claim 1, Features: In step (3) of constructing a human posture prediction network, the convolution kernel size of the convolution layer 1 is 3×3, the step size is 1, and the convolution kernel size of the convolution layer 2 is 1×1, the step size is 1.
5. The human body posture prediction method based on Transformer and graph convolutional network according to claim 2, Features The learnable position encoding layer 1 includes an embedding layer P e , the construction method of the learnable position encoding layer 1 is as follows: P e =E[P] Where E is the weight matrix, E is initialized by a normal distribution with a mean of -0.05 to 0.05 and a standard deviation of 0.01 to 0.05, P is the position index, and [·] represents vector mapping.
6. The human body posture prediction method based on Transformer and graph convolutional network according to claim 3, Features: The structure and construction method of the learnable position encoding layer 2 are the same as those of the learnable position encoding layer 1.
7. The human body posture prediction method based on Transformer and graph convolutional network according to claim 1, Features: The structure of the learnable position encoding layer 3 and the construction method thereof are the same as those of the learnable position encoding layer 1 .
Citation Information
Patent Citations
Three-dimensional human body posture estimation method based on graph convolution network
CN112712019A
Transformer based neural network for 3D human pose estimation
EP4310783A1
System and method for performing unified human motion prediction (HMP) for 2d and 3D poses
EP4358042A1
Sparse code multiple access encoding and decoding system based on generative adversarial network
WO2024016424A1
