Human body posture prediction method based on Transform and graph convolutional network

By combining the human posture prediction method of Transformer and graph convolutional network, the problem of low prediction accuracy and recognition rate caused by the complexity and high dimensionality of human posture sequence in the prior art is solved, and a higher pose prediction accuracy is achieved and recognition rate.

CN119091499AActive Publication Date: 2024-12-06SHAANXI NORMAL UNIV
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202411067398.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-06
Publication Date
2024-12-06
Estimated Expiration
2044-08-06

AI Technical Summary

Technical Problem

The prior art cannot accurately predict human posture sequences, especially due to the complexity and high dimensionality of human posture sequences, resulting in low prediction accuracy and recognition rate.

Method used

Using the combined method of Transformer and graph convolution network, the human posture prediction network formed by an encoder and decoder is extracted by combining Transformer's long-range dependency feature extraction capability and spatial feature modeling capability of graph convolution to extract the local and local posture sequences of human posture sequences. Global time and space dependency.

Benefits of technology

It improves the accuracy of human posture prediction, can extract motion characteristics more effectively, and achieve high accuracy and high recognition rate prediction.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119091499A_ABST
    Figure CN119091499A_ABST
Patent Text Reader

Abstract

The invention discloses a human body posture prediction method based on Transform and a graph convolutional network. The human body posture prediction method comprises the steps of obtaining a three-dimensional human body posture data set, dividing the data set, constructing a human body posture prediction network, training the human body posture prediction network and testing the human body posture prediction network. Due to the fact that the human body posture prediction network of the Transform and the graph convolution network is adopted, the technical problem that an existing human body posture prediction method depends on a hidden variable model and probabilistic modeling to synthesize possible human body motion, complexity and high dimension of a human body posture sequence, and the human body posture sequence cannot be accurately predicted is solved. Through combined use of the Transform network and the graph convolution network, local and global space-time dependency relationships of a human body posture sequence can be considered, motion features are extracted more effectively, and the accuracy of human body posture prediction is improved by combining the long-range dependency feature extraction capability of the Transform and the spatial feature modeling capability of graph convolution.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and deep learning, and specifically relates to a method for predicting human body posture. Background Art

[0002] Human posture prediction refers to predicting the human posture sequence in the future based on the human posture sequence observed in the past. It has broad application prospects in many fields such as autonomous driving, virtual reality and intelligent robots. Traditional human posture prediction methods usually rely on latent variable models and probabilistic modeling to synthesize possible human movements, but due to the complexity and high dimensionality of human posture sequences, these methods cannot accurately predict human posture sequences. In recent years, with the rapid development of deep learning, various neural network models such as recurrent neural networks, generative adversarial networks and graph convolutional networks have been widely used in human posture prediction.

[0003] Graph convolutional networks extract features of graph structured data through convolution operations and effectively capture dependencies between nodes. In human posture prediction, key points of the human skeleton can be modeled as nodes on the graph, and connections between joints are modeled as edges on the graph. This modeling method enables graph convolutional networks to fully utilize the spatial topological relationship of the human skeleton and extract complex dependencies and spatial features between joints. However, graph convolutional networks only extract spatial features of human joints at each time step and cannot dynamically capture posture changes at different time steps. Transformer networks can use self-attention mechanisms to capture long-term dependencies of posture sequences. The joint use of graph convolutional networks and transformer networks can take into account local and global spatiotemporal dependencies of human posture sequences and extract motion features more effectively.

[0004] In the field of human posture prediction technology, a technical problem that needs to be urgently solved is to provide a human posture prediction method with accurate prediction and high recognition rate. Summary of the invention

[0005] The main technical problem to be solved by the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a human posture prediction method based on Transformer and graph convolutional network with accurate prediction and high recognition rate.

[0006] The technical solution adopted to solve the above technical problems consists of the following steps:

[0007] (1) Obtaining a 3D human posture dataset

[0008] Download the Human3.6m dataset from the official website http: / / vision.imar.ro / human3.6m / description.php.

[0009] (2) Dividing the Dataset

[0010] The Human 3.6m dataset contains 7 sub-datasets: Action Set S1, Action Set S5, Action Set S6, Action Set S7, Action Set S8, Action Set S9, Action Set S11. Each sub-dataset contains 15 actions: walking f1, eating f2, smoking f3, discussing f4, pointing f5, greeting f6, taking pictures f7, posing f8, buying f9, sitting f10, etc. 10 , sit down 11 , take photos 12 , Wait for 13 , Walking the dog 14 Let's go together 15 ; Use action S1, action set S6, action set S7, action set S8, and action set S9 as training sets, and use action set S5 and action set S11 as test sets.

[0011] (3) Constructing a human posture prediction network

[0012] The human posture prediction network consists of an encoder and a decoder connected in series.

[0013] The encoder is composed of a Transformer encoder and a spatial encoder in parallel.

[0014] The decoder is composed of a learnable position encoding layer 3 and a Transformer encoding layer 3, a convolutional layer 1, and a convolutional layer 2 connected in series in sequence.

[0015] (4) Training the human posture prediction network

[0016] 1) Construct loss function

[0017] The loss function L is constructed as follows MPJPE :

[0018]

[0019] Among them, J is the number of human joints, J value ranges from 16 to 32, T is the number of frames, T value ranges from 10 to 25, represents the three-dimensional predicted coordinates of joint j in the tth frame in the real number range, x jt Represents the three-dimensional real coordinates of joint j in the t-th frame in the real number range.

[0020] 2) Training human posture prediction network

[0021] The training set is input into the human posture prediction network for training. The training parameters are: training batch is 16, training iteration number is 60, training server is GTX 1080, training to loss function L MPJPE convergence.

[0022] (5) Test the human posture prediction network

[0023] The test set is input into the trained human posture prediction network for testing, and the predicted values ​​of action set S5 and action set S11 are output.

[0024] In step (3) of constructing a human posture prediction network of the present invention, the Transformer encoder is composed of a learnable position encoding layer 1 and a Transformer encoding layer 1 connected in series.

[0025] In step (3) of constructing a human posture prediction network of the present invention, the spatial encoder is composed of a learnable position encoding layer 2, a Transformer encoding layer 2, and a graph convolutional network connected in series in sequence.

[0026] In the step (3) of constructing the human posture prediction network of the present invention, the convolution kernel size of the convolution layer 1 is 3×3, the step size is 1, and the convolution kernel size of the convolution layer 2 is 1×1, the step size is 1.

[0027] The learnable position encoding layer 1 of the present invention comprises an embedding layer P e , the construction method of the learnable position encoding layer 1 is as follows:

[0028] P e =E[P]

[0029] Where E is the weight matrix, E is initialized by a normal distribution with a mean of -0.05 to 0.05 and a standard deviation of 0.01 to 0.05, P is the position index, and [·] represents vector mapping.

[0030] The structure and construction method of the learnable position encoding layer 2 of the present invention are the same as those of the learnable position encoding layer 1.

[0031] The structure and construction method of the learnable position encoding layer 3 of the present invention are the same as those of the learnable position encoding layer 1.

[0032] The beneficial effects of the present invention are as follows:

[0033] Since the present invention adopts the human posture prediction network of Transformer and graph convolutional network, it solves the technical problem that the existing human posture prediction method relies on latent variable model and probabilistic modeling to synthesize possible human body movements, the complexity and high dimensionality of human posture sequence, and cannot accurately predict the human posture sequence. The combined use of Transformer network and graph convolutional network can take into account the local and global spatiotemporal dependency of human posture sequence and extract motion features more effectively. The present invention improves the accuracy of human posture prediction by combining the long-range dependency feature extraction capability of Transformer and the spatial feature modeling capability of graph convolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of embodiment 1 of the present invention.

[0035] Figure 2 It is a structural diagram of the human posture prediction network.

[0036] Figure 3 yes Figure 2 Schematic diagram of the encoder structure.

[0037] Figure 4 yes Figure 2 Schematic diagram of the decoder structure. DETAILED DESCRIPTION

[0038] The present invention will be further described in detail with reference to the accompanying drawings and embodiments, but the present invention is not limited to the following embodiments.

[0039] Example 1

[0040] like Figure 1 As shown, the human posture prediction method based on Transformer and graph convolutional network of this embodiment consists of the following steps:

[0041] (1) Obtaining a 3D human posture dataset

[0042] Download the Human3.6m dataset from the official website http: / / vision.imar.ro / human3.6m / description.php.

[0043] (2) Dividing the Dataset

[0044] The Human 3.6m dataset contains 7 sub-datasets: Action Set S1, Action Set S5, Action Set S6, Action Set S7, Action Set S8, Action Set S9, Action Set S11. Each sub-dataset contains 15 actions: walking f1, eating f2, smoking f3, discussing f4, pointing f5, greeting f6, taking pictures f7, posing f8, buying f9, sitting f10, etc.10 , sit down 11 , take photos 12 , Wait for 13 , Walking the dog 14 Let's go together 15 ; Use action S1, action set S6, action set S7, action set S8, and action set S9 as training sets, and use action set S5 and action set S11 as test sets.

[0045] (3) Constructing a human posture prediction network

[0046] Figure 2 The structural diagram of the human posture prediction network is given. Figure 2 In the embodiment, the human body posture prediction network is composed of an encoder and a decoder connected in series.

[0047] Figure 3 Given Figure 2 The structural diagram of the encoder is shown in Figure 3 In the embodiment, the encoder is composed of a Transformer encoder and a spatial encoder in parallel.

[0048] The Transformer encoder of this embodiment is composed of a learnable position encoding layer 1 and a Transformer encoding layer 1 connected in series; the spatial encoder is composed of a learnable position encoding layer 2, a Transformer encoding layer 2, and a graph convolutional network connected in series in sequence.

[0049] The learnable position encoding layer 1 in this embodiment includes an embedding layer P e , the construction method of the learnable position encoding layer 1 is as follows:

[0050] P e =E[P]

[0051] Wherein, E is a weight matrix, E is initialized by a normal distribution with a mean of -0.05 to 0.05 and a standard deviation of 0.01 to 0.05. In this embodiment, the mean is 0 and the standard deviation is 0.02. P is a position index, and [·] represents vector mapping.

[0052] The structures and construction methods of the learnable position coding layer 2 and the learnable position coding layer 3 of this embodiment are the same as those of the learnable position coding layer 1.

[0053] The spatial encoder of this embodiment is composed of a learnable position encoding layer 2, a Transformer encoding layer 2, and a graph convolutional network connected in series.

[0054] Figure 4 Given Figure 2 The schematic diagram of the decoder structure in Figure 4 In the decoder of this embodiment, a learnable position encoding layer 3 and a Transformer encoding layer 3, a convolutional layer 1, and a convolutional layer 2 are sequentially connected in series. The convolution kernel size of the convolutional layer 1 is 3×3, and the step size is 1. The convolution kernel size of the convolutional layer 2 is 1×1, and the step size is 1.

[0055] Since the present invention adopts the human posture prediction network of Transformer and graph convolutional network, it solves the technical problem that the existing human posture prediction method relies on latent variable model and probabilistic modeling to synthesize possible human body movements, the complexity and high dimensionality of human posture sequence, and cannot accurately predict the human posture sequence. The combined use of Transformer network and graph convolutional network can take into account the local and global spatiotemporal dependency of human posture sequence and extract motion features more effectively. The present invention improves the accuracy of human posture prediction by combining the long-range dependency feature extraction capability of Transformer and the spatial feature modeling capability of graph convolution.

[0056] (4) Training the human posture prediction network

[0057] 1) Construct loss function

[0058] The loss function L is constructed as follows MPJPE :

[0059]

[0060] in, represents the three-dimensional predicted coordinates of joint j in the tth frame in the real number range, x jt It represents the three-dimensional real coordinates of joint j in the t-th frame within the real number range, J is the number of human joints, J takes a value of 16 to 32, and the value of J in this embodiment is 22, T is the number of frames, T takes a value of 10 to 25, and the value of T in this example is 15.

[0061] 2) Training human posture prediction network

[0062] The training set is input into the human posture prediction network for training. The training parameters are: training batch is 16, training iteration number is 60, training server is GTX 1080, training to loss function L MPJPE convergence.

[0063] (5) Test the human posture prediction network

[0064] The test set is input into the trained human posture prediction network for testing, and the predicted values ​​of action set S5 and action set S11 are output.

[0065] Completed the human posture prediction method based on Transformer and graph convolutional network.

[0066] Example 2

[0067] The human body posture prediction method based on Transformer and graph convolutional network in this embodiment consists of the following steps:

[0068] (1) Obtaining a 3D human posture dataset

[0069] This step is the same as in Example 1.

[0070] (2) Dividing the Dataset

[0071] This step is the same as in Example 1.

[0072] (3) Constructing a human posture prediction network

[0073] The structure of the human posture prediction network is the same as that of Example 1.

[0074] The learnable position encoding layer 1 in this embodiment includes an embedding layer P e , the construction method of the learnable position encoding layer 1 is as follows:

[0075] P e =E[P]

[0076] Wherein, E is a weight matrix, E is initialized by a normal distribution with a mean of -0.05 to 0.05 and a standard deviation of 0.01 to 0.05. In this embodiment, the mean is -0.05 and the standard deviation is 0.01. P is a position index, and [·] represents vector mapping.

[0077] (4) Training the human posture prediction network

[0078] 1) Construct loss function

[0079] The expression of the loss function is the same as that in Example 1.

[0080] In the expression of the loss function, represents the three-dimensional predicted coordinates of joint j in the tth frame in the real number range, x jt It represents the three-dimensional real coordinates of joint j in the t-th frame within the real number range, J is the number of human joints, J takes a value of 16 to 32, and the value of J in this embodiment is 16, T is the number of frames, T takes a value of 10 to 25, and the value of T in this example is 10.

[0081] The other steps of this step are the same as those in Example 1.

[0082] The other steps are the same as those in Example 1. A human posture prediction method based on Transformer and graph convolutional network is completed.

[0083] Example 3

[0084] The human body posture prediction method based on Transformer and graph convolutional network in this embodiment consists of the following steps:

[0085] (1) Obtaining a 3D human posture dataset

[0086] This step is the same as in Example 1.

[0087] (2) Dividing the Dataset

[0088] This step is the same as in Example 1.

[0089] (3) Constructing a human posture prediction network

[0090] The structure of the human posture prediction network is the same as that of Example 1.

[0091] The learnable position encoding layer 1 in this embodiment includes an embedding layer P e , the construction method of the learnable position encoding layer 1 is as follows:

[0092] P e =E[P]

[0093] Wherein, E is a weight matrix, E is initialized by a normal distribution with a mean of -0.05 to 0.05 and a standard deviation of 0.01 to 0.05. In this embodiment, the mean is 0.05 and the standard deviation is 0.05. P is a position index, and [·] represents vector mapping.

[0094] (4) Training the human posture prediction network

[0095] 1) Construct loss function

[0096] The expression of the loss function is the same as that in Example 1.

[0097] In the expression of the loss function, represents the three-dimensional predicted coordinates of joint j in the tth frame in the real number range, x jt It represents the three-dimensional real coordinates of joint j in the t-th frame within the real number range, J is the number of human joints, J takes a value of 16 to 32, and the value of J in this embodiment is 32, T is the number of frames, T takes a value of 10 to 25, and the value of T in this example is 25.

[0098] The other steps of this step are the same as those in Example 1.

[0099] The other steps are the same as those in Example 1. A human posture prediction method based on Transformer and graph convolutional network is completed.

[0100] In order to verify the beneficial effects of the present invention, the inventors conducted comparative experiments using the human posture prediction method based on Transformer and graph convolutional network of Example 1 of the present invention, the posture prediction method based on spatiotemporal separable graph convolutional network (hereinafter referred to as STS-GCN), and the human motion prediction method based on 3D skeleton with auxiliary tasks (hereinafter referred to as Auxformer). Various experimental conditions are shown in Table 1 below:

[0101] The evaluation index is determined as the average position error ε:

[0102]

[0103] in, are the three-dimensional prediction coordinates in the real number range, is the three-dimensional true coordinate in the real number range. The smaller the value of ε, the higher the prediction accuracy.

[0104] Table 1 Experimental results of the method of Example 1 and the comparative experimental method

[0105]

[0106] As can be seen from Table 1, the average position error of the method in Example 1 is 8.0 when the prediction time is 80 milliseconds, which is lower than the STS-GCN method and the Auxformer method; the average position error of the method in Example 1 is 30.2 when the prediction time is 320 milliseconds, which is lower than the STS-GCN method and the Auxformer method; the average position error of the method in Example 1 is 49.5 when the prediction time is 560 milliseconds, which is lower than the STS-GCN method and the Auxformer method; the average position error of the method in Example 1 is 74.9 when the prediction time is 1000 milliseconds, which is lower than the STS-GCN method and the Auxformer method. Experiments show that the method of the present invention has higher accuracy in human body posture prediction.

Claims

1. A human posture prediction method based on Transformer and graph convolutional network, Features It consists of the following steps: (1) Obtaining a 3D human posture dataset Download the Human 3.6m dataset from the official website http: / / vision.imar.ro / human3.6m / description.php; (2) Dividing the Dataset The Human 3.6m dataset contains 7 sub-datasets: Action Set S1, Action Set S5, Action Set S6, Action Set S7, Action Set S8, Action Set S9, Action Set S11. Each sub-dataset contains 15 actions: walking, 1 , eat 2 , smoking 3 、Discussion 4 、Guide 5 , say hello 6 , Camera 7 , pose 8 、Purchase 9 、Sit 10 , sit down 11 , take photos 12 , Wait for 13 , Walking the dog 14 Let's go together 15 ; Use action S1, action set S6, action set S7, action set S8, action set S9 as training sets, and action set S5 and action set S11 as test sets; (3) Constructing a human posture prediction network The human posture prediction network is composed of an encoder and a decoder in series; The encoder is composed of a Transformer encoder and a spatial encoder in parallel; The decoder is composed of a learnable position encoding layer 3 and a Transformer encoding layer 3, a convolutional layer 1, and a convolutional layer 2 connected in series in sequence; (4) Training the human posture prediction network 1) Construct loss function The loss function L is constructed as follows MPJPE : Among them, J is the number of human joints, J value ranges from 16 to 32, T is the number of frames, T value ranges from 10 to 25, represents the three-dimensional predicted coordinates of joint j in the tth frame in the real number range, x jt represents the three-dimensional real coordinates of joint j in the t-th frame within the real number range; 2) Training human posture prediction network The training set is input into the human posture prediction network for training. The training parameters are: training batch is 16, training iteration number is 60, training server is GTX 1080, training to loss function L MPJPE convergence; (5) Test the human posture prediction network The test set is input into the trained human posture prediction network for testing, and the predicted values ​​of action set S5 and action set S11 are output.

2. According to claim 1, the human posture prediction method based on Transformer and graph convolutional network, Features: In step (3) of constructing a human posture prediction network, the Transformer encoder is composed of a learnable position encoding layer 1 and a Transformer encoding layer 1 connected in series.

3. The human body posture prediction method based on Transformer and graph convolutional network according to claim 1, Features: In step (3) of constructing a human posture prediction network, the spatial encoder is composed of a learnable position encoding layer 2, a Transformer encoding layer 2, and a graph convolutional network connected in series.

4. The human body posture prediction method based on Transformer and graph convolutional network according to claim 1, Features: In step (3) of constructing a human posture prediction network, the convolution kernel size of the convolution layer 1 is 3×3, the step size is 1, and the convolution kernel size of the convolution layer 2 is 1×1, the step size is 1.

5. The human body posture prediction method based on Transformer and graph convolutional network according to claim 2, Features The learnable position encoding layer 1 includes an embedding layer P e , the construction method of the learnable position encoding layer 1 is as follows: P e =E[P] Where E is the weight matrix, E is initialized by a normal distribution with a mean of -0.05 to 0.05 and a standard deviation of 0.01 to 0.05, P is the position index, and [·] represents vector mapping.

6. The human body posture prediction method based on Transformer and graph convolutional network according to claim 3, Features: The structure and construction method of the learnable position encoding layer 2 are the same as those of the learnable position encoding layer 1.

7. The human body posture prediction method based on Transformer and graph convolutional network according to claim 1, Features: The structure of the learnable position encoding layer 3 and the construction method thereof are the same as those of the learnable position encoding layer 1 .

Citation Information

Patent Citations

  • Three-dimensional human body posture estimation method based on graph convolution network

    CN112712019A

  • Transformer based neural network for 3D human pose estimation

    EP4310783A1

  • System and method for performing unified human motion prediction (HMP) for 2d and 3D poses

    EP4358042A1

  • Sparse code multiple access encoding and decoding system based on generative adversarial network

    WO2024016424A1