3D Human Pose Estimation Method Based on Improved GraFormer Network

Abstract

The present invention discloses a method for 3D human pose estimation based on an improved GraFormer network, comprising the following steps: 1) extracting original 2D human pose information from a single human pose image and dividing the image into a training set, a validation set, and a test set; 2) inputting the training set into the improved GraFormer network for training, and outputting 3D human pose information that has a one-to-one mapping relationship with the original 2D human pose information; 3) calculating a loss value of a Euclidean distance between the 3D human pose information and the true value, and then performing weight updates on the improved GraFormer network after the loss value is calculated. An optimizer adjusts network parameters according to the number of iterations during training, updates the learning rate, iterates until the total loss of the validation set is minimized, and saves the pre-trained network parameters for that iteration. After training, the pre-trained network parameters are loaded and tested using the test set, ultimately obtaining the optimal 3D human pose information output by the improved GraFormer network. The present invention improves the accuracy of 3D human pose estimation and alleviates the problem of human occlusion in images.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and image recognition, and in particular to a three-dimensional human body posture estimation method based on an improved GraFormer network. Background Art

[0002] Three-dimensional human pose estimation is a classic research topic in computer vision. The goal is to estimate the position of human joints in three-dimensional space from RBG images or videos.

[0003] In the early days, researchers used handcrafted features and geometric constraints to predict 3D human pose. With the rapid development of deep learning, deep neural networks have also been used to predict 3D human skeletons. Existing work has encountered challenges, including the highly nonlinear fitting problem of mapping 2D to 3D information, the difficulty in predicting occluded parts of the human body in images, and the limited availability of open-source datasets for wild-field human poses, resulting in poor generalization. Summary of the Invention

[0004] The purpose of the present invention is to overcome the shortcomings and deficiencies of the existing technology and propose a 3D human pose estimation method based on an improved GraFormer network to predict reasonable 3D human skeleton information. Compared with the existing methods, the accuracy of 3D human pose prediction is improved, the occlusion problem of the human body in the image is alleviated, and the generalization ability of the network is improved.

[0005] To achieve the above objectives, the present invention provides a technical solution: a three-dimensional human pose estimation method based on an improved GraFormer network, wherein the improved GraFormer network improves the TransFormer and Chebyshev graph convolution of the original GraFormer network; wherein the improvement to the TransFormer is: replacing the MLP layer in the TransFormer with a semantic graph convolution layer, through which more useful feature information is collected and the human topological structure is strengthened; and the improvement to the Chebyshev graph convolution is: modifying the prior constraints of the Chebyshev graph convolution, replacing the first-order adjacency matrix of human joints with an adjacency matrix with first-order and second-order information, and by adding the information of the second-order adjacency matrix, strengthening the relationship between the distant neighbors of the human joints, thereby effectively estimating three-dimensional human pose information;

[0006] The specific implementation of the 3D human posture estimation method includes the following steps:

[0007] 1) Using the cascaded pyramid network (CPN) as a 2D skeleton detector, the original 2D human pose information is extracted from a given single human pose image. The original 2D human pose information is divided into a training set, a validation set, and a test set.

[0008] 2) The training set is fed into the improved GraFormer network. First, the input is globally extracted using the self-attention layer in the improved Transformer. These features are then fed into the semantic graph convolution layer to add human topological structure information to the features. Finally, the features are fed into the improved Chebyshev graph convolution layer. In the Chebyshev graph convolution, the adjacency matrix with first-order and second-order information is used as a priori constraint. This expands the receptive field of the network layer, extends the distant neighbor relationships between joints, and strengthens human structural information. The output is 3D human pose information that has a one-to-one mapping relationship with the original 2D human pose information.

[0009] 3) Calculate a Euclidean distance loss value between the 3D human posture information output in step 2) and the true value. Finally, update the weights based on the loss value calculated by the improved GraFormer network. Adjust the network parameters using the optimizer according to the number of iterations in the training process, update the learning rate, and iterate until the total loss of the validation set reaches the current minimum state and save the pre-trained network parameters. After the training is completed, load the pre-trained network parameters and use the test set for testing, and finally obtain the optimal 3D human posture information output by the improved Graformer network.

[0010] Furthermore, in step 1), in order to obtain the input of the network, a cascade pyramid network CPN is used to obtain the human body posture image I from a given single human body posture image I s Extract the original two-dimensional human posture P s , the original two-dimensional human body posture P s Refers to the two-dimensional human posture joint points corresponding to the characters in the human posture image, and then the original two-dimensional human posture P s The information is stored in the form of npy files.

[0011] Furthermore, in step 2), the training set is used as the input of the improved GraFormer network. The input first passes through the improved TransFormer. The self-attention in the improved TransFormer globally extracts the contextual information of the joint points to form a new feature map X1, as shown in the following formulas (1), (2), (3) and (4):

[0012]

[0013] Q1=W q X0 (2)

[0014] K1=W k X0 (3)

[0015] V1=W v X0 (4)

[0016] In the formula, matrices Q1, K1 and V1 are all generated by the initial input feature X0 through the parameter matrix W q 、W k With W v Converted, K1 T is the transpose of matrix K1, σ s represents the Softmax function, It is the square root of the feature dimension, which is to prevent the distribution of the numerical dot product from becoming sharp and causing the gradient of the Softmax function to disappear;

[0017] After obtaining the feature map of the global information of the joints, the feature map will pass through the semantic map convolution layer. Its significance is that the semantic map convolution layer adds the most basic topological structure information of the human body to the feature map, and performs feature extraction more targetedly. The operation process of the semantic map convolution layer is shown in the following formulas (5) and (6):

[0018]

[0019]

[0020] Where σ r Represents the RELU function, W s represents the parameters of the network layer, X1 represents the feature map containing global information, ρ s represents the activation function, M represents a learnable matrix, Represents the first-order adjacency matrix of the joint point, a ij Represents the adjacency matrix The element in row i and column j, m ij Represents the i-th row and j-th column element in the learnable matrix M, ⊙ represents a Element operation: When a ij =1, returns the element value m ij , otherwise return -∞, X2 represents the global feature map with the basic topology of human joints;

[0021] After the improved TransFormer, a global feature map with the basic topology of human joints is obtained.

[0022] Furthermore, in step 2), after obtaining the global feature map with the basic topology of the human joints, the improved Chebyshev graph convolution is used to further extract features. In the improved Chebyshev graph convolution, the first-order adjacency matrix used in the original Chebyshev graph convolution is replaced by an adjacency matrix with first-order and second-order information, and the improved Chebyshev graph convolution is called the Chebyshev graph convolution with mixed adjacency matrix information. Its significance is to expand the receptive field of the network layer, extend the distant neighbor relationship between joint points and strengthen the human body structure information. The operation process of the Chebyshev graph convolution with mixed adjacency matrix information is shown in the following formulas (7), (8), (9) and (10):

[0023]

[0024]

[0025]

[0026]

[0027] Where, T k Indicated as T k (x) = 2xT k-1 (x)-T k-2 (x) is the kth order Chebyshev function, where k ≥ 2, x represents the eigenvalue of the joint point, T k (x) indicates that the k-th order Chebyshev value of the joint point eigenvalue x is obtained, T k-1 (x) represents the k-1th order Chebyshev value obtained from the joint point eigenvalue x, T k-2 (x) represents the k-2th order Chebyshev value A and A obtained from the joint point eigenvalue x. 2 and Represent the first-order adjacency matrix, the second-order adjacency matrix and the adjacency matrix with first-order and second-order information, respectively. It is represented as the diagonal matrix of node degree, I is represented as the identity matrix of the same specification as the adjacency matrix, L is the Laplacian operator of the graph, λ max represents the maximum eigenvalue in L, X2 represents the global feature with the basic topology of human joints, and W c represents the parameters of the network layer, X3 represents the global features of strengthening the human joint topology;

[0028] After the improved Chebyshev graph convolution, the output is the three-dimensional human posture information P that has a one-to-one mapping relationship with the original two-dimensional human posture information. t .

[0029] Furthermore, in step 3), the improved GraFormer network for the 3D human pose estimation task is used to minimize the output 3D human pose information Pt and real 3D human body posture information The L2 distance between them is used to optimize the generation ability of the network, and the loss L defined is expressed as:

[0030]

[0031] Where N represents the total number of joint points of the generated person, P ti Indicates the information of the i-th joint point in the generated person, P ti Represents the i-th joint information of a real 3D human body;

[0032] After calculating the loss value, the improved GraFormer network is used to update the weights. The optimizer adjusts the network parameters according to the number of iterations in the training process, updates the learning rate, and iterates until the total loss of the validation set reaches the current minimum state and the pre-trained network parameters are saved. After the training is completed, the pre-trained network parameters are loaded and tested using the test set, and finally the optimal three-dimensional human posture information output by the improved Graformer network is obtained.

[0033] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0034] 1. This paper proposes a 3D human pose estimation method based on an improved GraFormer network. Through the improved TransFormer and improved Chebyshev graph convolution, it can predict better 3D human pose information.

[0035] 2. Compared with other 3D human posture methods based on graph convolution, the present invention alleviates the problem of human body occlusion in images and makes full use of 2D human posture information.

[0036] 3. This paper designs an improved TransFormer, replacing the MLP network layer in the original TransFormer with a semantic graph convolution layer to add human body structure topology to the extracted features.

[0037] 4. The present invention designs an improved Chebyshev graph convolution, which replaces the first-order adjacency matrix used in the Chebyshev graph convolution with an adjacency matrix with first-order and second-order information, strengthens the connection between distant neighboring joints between human joints, and further enhances the topological information of the human body structure. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1Schematic diagram of the network structure of the present invention; in the figure, ChebGConv Layer represents the original Chebyshev graph convolution, SGraAttention represents the improved TransFormer, ACChebGConv represents the improved graph convolution, dim represents the human body posture information encoded into high-dimensional features, and N represents the number of improved network layers used.

[0039] Figure 2 This is a schematic diagram of the network structure of the improved TransFormer of the present invention; in the figure, Layer Norm represents the feature normalization layer, Multi-head attention represents the multi-head attention layer, and SemGconv represents the semantic graph convolution layer.

[0040] Figure 3 This is a schematic diagram of the network structure of the improved Chebyshev graph convolution of the present invention; in the figure, ACChebGConv represents the Chebyshev graph convolution with mixed adjacency information. DETAILED DESCRIPTION

[0041] The present invention will be described in further detail below with reference to the embodiments and drawings, but the embodiments of the present invention are not limited thereto.

[0042] This embodiment discloses a method for estimating three-dimensional human pose based on an improved GraFormer network. The improved GraFormer network improves the TransFormer and Chebyshev graph convolution of the original GraFormer network. The improvement to the TransFormer is as follows: the MLP layer in the TransFormer is replaced with a semantic graph convolution layer, which collects more useful feature information and strengthens the human topological structure. The improvement to the Chebyshev graph convolution is as follows: the prior constraints of the Chebyshev graph convolution are modified, the first-order adjacency matrix of human joints is replaced with an adjacency matrix with first-order and second-order information, and the information of the second-order adjacency matrix is ​​added to strengthen the relationship between the distant neighbors of the human joints, thereby effectively estimating three-dimensional human pose information. The specific details are as follows:

[0043] 1) Use the cascade pyramid network CPN as a two-dimensional skeleton detector and input a single human posture image I s , the specific pixel is 1000×1002, output the original two-dimensional human posture P s The original 2D human pose refers to the human pose joints corresponding to the person in a single human pose image. Specifically, it contains a total of 17 joints, each with 2 coordinate information. The predicted original 2D human pose information is stored in the format of npy files and divided into training, validation, and test sets.

[0044] 2) Input the training set into the improved GraFormer network, and then input the improved TransFormer and Chebyshev graph convolution to finally obtain the 3D human posture information that has a one-to-one mapping relationship with the original 2D human posture information. Figure 1 As shown, the following steps are included:

[0045] 2.1) Take a 2D human body pose P s As input, its size is 17×2. It first passes through the improved TransFormer, in which the feature is normalized by the LayerNorm layer. Then, the context information of the joint points is globally extracted through the multi-head attention layer (self-attention) to form a new feature map X1, which is shown in the following formulas (1), (2), (3) and (4):

[0046]

[0047] Q1=W q X0 (2)

[0048] K1=W k X0 (3)

[0049] V1=W v X0 (4)

[0050] In the formula, matrices Q1, K1 and V1 are all generated by the initial input feature X0 through the parameter matrix W q 、W k With W v Converted, K1 T is the transpose of matrix K1, σ s represents the Softmax function, is the square root of the feature dimension, which is set to This is to prevent the distribution from becoming sharp after the numerical dot product, which causes the gradient of the Softmax function to disappear.

[0051] Then the Dropout layer is used to improve the fitting ability of the network, and there is a skip connection between layers.

[0052] After obtaining the feature map of the global information of the joints, the LayerNorm layer is used to normalize it again. Then, we replace the MLP layer in the original Transformer with two layers of semantic graph convolution. The significance of this is that the semantic graph convolution layer adds the most basic topological structure information of the human body to the feature map, and performs feature extraction more targetedly. The operation process of the semantic graph convolution layer is shown in the following formulas (5) and (6):

[0053]

[0054]

[0055] Where σ r Represents the RELU function, W s represents the parameters of the network layer, X1 represents the feature map containing global information, ρ s represents the activation function, M represents a learnable matrix, Represents the first-order adjacency matrix of the joint point, a ij Represents the adjacency matrix The element in row i and column j, m ij Represents the i-th row and j-th column element in the learnable matrix M, ⊙ represents a Element operation: When a ij =1, returns the element value m ij , otherwise returns -∞, X2 represents the global feature map with the basic topology of human joints.

[0056] After the semantic graph convolution layer, a global feature map with the basic topology of human joints is obtained, and then the feature is normalized by the Dropout layer. There is a skip connection between layers. Figure 2 shown.

[0057] 2.2) The obtained global feature map with the basic topology of human joints is input into the improved Chebyshev graph convolution to further extract features. The improved Chebyshev graph convolution is called the Chebyshev graph convolution with mixed adjacency matrix information. The Chebyshev graph convolution with mixed adjacency matrix information refers to the use of adjacency matrices with first-order and second-order information in the Chebyshev graph convolution. Its significance is to expand the receptive field of the network layer, extend the distant neighbor relationship between joint points and strengthen the structural information of the human skeleton. The operation process of the Chebyshev graph convolution layer with mixed adjacency matrix information is shown in the following formulas (7), (8), (9) and (10):

[0058]

[0059]

[0060]

[0061]

[0062] Where, T k Indicated as T k (x) = 2xT k-1 (x)-T k-2 (x) is the kth order Chebyshev function, where k ≥ 2, x represents the eigenvalue of the joint point, T k (x) indicates that the k-th order Chebyshev value of the joint point eigenvalue x is obtained, Tk-1 (x) represents the k-1th order Chebyshev value obtained from the joint point eigenvalue x, T k-2 (x) represents the k-2th order Chebyshev value obtained from the joint point eigenvalue x. In this embodiment, K is set to 3, A, A 2 and A represent the first-order adjacency matrix, the second-order adjacency matrix, and the adjacency matrix with first-order and second-order information, respectively. It is represented as the diagonal matrix of node degree, I is represented as the identity matrix of the same specification as the adjacency matrix, L is the Laplacian operator of the graph, λ max represents the maximum eigenvalue in L, X2 represents the global feature with the basic topology of human joints, and W c represents the parameters of the network layer, and X3 represents the global features of strengthening the human joint topology.

[0063] In this embodiment, the improved GraFormer network includes two stacked Chebyshev graph convolution and RELU layers with mixed adjacency matrix information, with a skip connection in the middle. Figure 3 shown.

[0064] Finally, the global features of the enhanced human joint topology are input to the decoding layer. The decoding layer is the original Chebyshev graph convolution, which outputs the high-dimensional global features into three-dimensional human posture information P that has a one-to-one mapping relationship with the original two-dimensional human posture information. t ,The specification of three-dimensional human posture information is 17×3.

[0065] 3) For the improved GraFormer network of 3D human body task, by minimizing the output 3D human body posture information P t and real 3D human body posture information The L2 distance between them is used to optimize the generation ability of the network, and the loss L defined is expressed as:

[0066]

[0067] Where N represents the total number of joints of the generated person, which is 17 in this embodiment, and P ti Indicates the information of the i-th joint point in the generated person, Represents the i-th joint information of a real 3D human body.

[0068] After calculating the loss value, the improved GraFormer network is used to update the weights. The optimizer adjusts the network parameters according to the number of iterations in the training process, updates the learning rate, and iterates until the total loss of the validation set reaches the current minimum state and the pre-trained network parameters are saved. After the training is completed, the pre-trained network parameters are loaded and tested using the test set, and finally the optimal three-dimensional human posture information output by the improved Graformer network is obtained.

[0069] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A 3D human pose estimation method based on an improved GraFormer network, characterized in that: The improved GraFormer network improves the TransFormer and Chebyshev graph convolution of the original GraFormer network. The improvement to the TransFormer is to replace the MLP layer in the TransFormer with a semantic graph convolution layer, which collects more useful feature information and strengthens the human body topology. The improvement to the Chebyshev graph convolution is to modify the prior constraints of the Chebyshev graph convolution and replace the first-order adjacency matrix of human joints with an adjacency matrix with first-order and second-order information. By adding the information of the second-order adjacency matrix, the relationship between the distant neighbors of the human joints is strengthened, thereby effectively estimating the 3D human skeleton. The specific implementation of the 3D human posture estimation method includes the following steps: 1) Using the cascaded pyramid network (CPN) as a 2D skeleton detector, the original 2D human pose information is extracted from a given single human pose image and the original 2D human pose information is divided into a training set, a validation set, and a test set; 2) The training set is fed into the improved GraFormer network for training. First, the input is globally extracted using the self-attention layer in the improved Transformer. The features are then fed into the semantic graph convolution layer to add human topological structure information to the features. Finally, the features are fed into the improved Chebyshev graph convolution layer. In the Chebyshev graph convolution, the adjacency matrix with first-order and second-order information is used as a priori constraint. This expands the receptive field of the network layer, extends the distant neighbor relationships between joints, and strengthens human structural information. The output is 3D human pose information that has a one-to-one mapping relationship with the original 2D human pose information. 3) Calculate a Euclidean distance loss value between the 3D human posture information output in step 2) and the true value. Finally, update the weights based on the loss value calculated by the improved GraFormer network. Adjust the network parameters using the optimizer according to the number of iterations in the training process, update the learning rate, and iterate until the total loss of the validation set reaches the current minimum state and save the pre-trained network parameters. After the training is completed, load the pre-trained network parameters and use the test set for testing, and finally obtain the optimal 3D human posture information output by the improved Graformer network.

2. The 3D human pose estimation method based on the improved GraFormer network according to claim 1, characterized in that: In step 1), in order to obtain the input of the network, a cascade pyramid network CPN is used to obtain the human body posture image I from a given single human body posture image I s Extract the original two-dimensional human posture information P s , the original two-dimensional human posture information P s Refers to the two-dimensional human posture joint points corresponding to the characters in the human posture image, and then the original two-dimensional human posture information P s Stored in the form of npy files.

3. The 3D human pose estimation method based on the improved GraFormer network according to claim 2, characterized in that: In step 2), the training set is used as the input of the improved GraFormer network. The input first passes through the improved TransFormer. The self-attention in the improved TransFormer globally extracts the contextual information of the joint points to form a new feature map X1, as shown in the following formulas (1), (2), (3) and (4): Q1=W q X0 (2) K1=W k X0 (3) V1=W v X0 (4) In the formula, matrices Q1, K1 and V1 are all generated by the initial input feature X0 through the parameter matrix W q 、W k With W v Converted, K1 T is the transpose of matrix K1, σ s represents the Softmax function, It is the square root of the feature dimension, which is to prevent the distribution of the numerical dot product from becoming sharp and causing the gradient of the Softmax function to disappear; After obtaining the feature map of the global information of the joints, the feature map will pass through the semantic map convolution layer. Its significance is that the semantic map convolution layer adds the most basic topological structure information of the human body to the feature map, and performs feature extraction more targetedly. The operation process of the semantic map convolution layer is shown in the following formulas (5) and (6): Where σ r Represents the RELU function, W s represents the parameters of the network layer, X1 represents the feature map containing global information, ρ s represents the activation function, M represents a learnable matrix, Represents the first-order adjacency matrix of the joint point, a ij Represents the adjacency matrix The element in row i and column j, m ij Represents the i-th row and j-th column element in the learnable matrix M, ⊙ represents a Element operation: When a ij =1, returns the element value m ij , otherwise return -∞, X2 represents the global feature map with the basic topology of human joints; After the improved TransFormer, a global feature map with the basic topology of human joints is obtained.

4. The 3D human pose estimation method based on the improved GraFormer network according to claim 3, characterized in that: In step 2), after obtaining the global feature map with the basic topology of human joints, the improved Chebyshev graph convolution is used to further extract features. In the improved Chebyshev graph convolution, the first-order adjacency matrix used in the original Chebyshev graph convolution is replaced by an adjacency matrix with first-order and second-order information, and the improved Chebyshev graph convolution is called the Chebyshev graph convolution with mixed adjacency matrix information. Its significance is to expand the receptive field of the network layer, extend the distant neighbor relationship between joint points and strengthen the human body structure information. The operation process of the Chebyshev graph convolution with mixed adjacency matrix information is shown in the following formulas (7), (8), (9) and (10): Where, T k Indicated as T k (x) = 2xT k-1 (x)-T k-2 (x) is the kth order Chebyshev function, where k ≥ 2, x represents the eigenvalue of the joint point, T k (x) indicates that the k-th order Chebyshev value of the joint point eigenvalue x is obtained, T k-1 (x) represents the k-1th order Chebyshev value obtained from the joint point eigenvalue x, T k-2 (x) represents the k-2th order Chebyshev value obtained from the joint point eigenvalue x, A, A 2 and Represent the first-order adjacency matrix, the second-order adjacency matrix and the adjacency matrix with first-order and second-order information, respectively. It is represented as the diagonal matrix of node degree, I is represented as the identity matrix of the same specification as the adjacency matrix, L is the Laplacian operator of the graph, λ max represents the maximum eigenvalue in L, X2 represents the global feature with the basic topology of human joints, and W c represents the parameters of the network layer, X3 represents the global features of strengthening the human joint topology; After the improved Chebyshev graph convolution, the output is the three-dimensional human posture information P that has a one-to-one mapping relationship with the original two-dimensional human posture information. t .

5. The 3D human pose estimation method based on the improved GraFormer network according to claim 4, characterized in that: In step 3), the improved GraFormer network for the 3D human pose estimation task is used to minimize the output 3D human pose information P t and real 3D human body posture information The L2 distance between them is used to optimize the generation ability of the network, and the loss L defined is expressed as: Where N represents the total number of joint points of the generated person, P ti Indicates the information of the i-th joint point in the generated person, Represents the i-th joint information of a real 3D human body; After calculating the loss value, the improved GraFormer network is used to update the weights. The optimizer adjusts the network parameters according to the number of iterations in the training process, updates the learning rate, and iterates until the total loss of the validation set reaches the current minimum state and the pre-trained network parameters are saved. After the training is completed, the pre-trained network parameters are loaded and tested using the test set, and finally the optimal three-dimensional human posture information output by the improved Graformer network is obtained.

Images