A method and system for three-dimensional human pose estimation based on graph and attention interweaving

Abstract

The present application relates to a kind of three-dimensional human posture estimation method and system based on graph and attention interweave.The system extracts the two-dimensional skeleton information of human body from image features by pre-trained two-dimensional pose detector;Two-dimensional skeleton is embedded into high-dimensional space;Local and global information of skeleton is mined using graph and attention interweave network module;Multi-layer perception module of U-shaped structure is used to capture multi-level information of skeleton;High-dimensional data is regressed to three-dimensional skeleton using regression head module;The average error of joint is used as the loss function of model training.The present application combines the advantages of graph convolution and attention mechanism in capturing local and global information of skeleton, allows bidirectional communication between graph convolution module and attention module to complement each other, can effectively strengthen the modeling ability of model to human skeleton, can estimate the result closer to real three-dimensional posture.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of target recognition and intelligent human-computer interaction in machine vision, and particularly relates to a three-dimensional human pose estimation method and system based on graph and attention interweaving. BACKGROUND

[0002] The purpose of human pose estimation is to depict the human form in pictures, videos and other objects, including target recognition, image segmentation, regression detection and other tasks. Compared with two-dimensional pose estimation, three-dimensional human pose estimation is more accurate in representing human poses and has higher research value. Nowadays, three-dimensional human pose estimation has become a research hotspot in the field of computer vision and is also the basis of many research works. The three-dimensional human pose extracted from images or videos can be further used for action recognition, three-dimensional mesh reconstruction and other tasks.

[0003] Existing three-dimensional human pose estimation methods can be broadly divided into two categories: (1) three-dimensional human pose estimation based on direct regression. This method does not need to use two-dimensional pose representation, but directly predicts three-dimensional pose coordinates from two-dimensional graphics. The advantage of this method is that it can realize end-to-end network training, but the requirements for network structure and data preprocessing are higher. (2) three-dimensional human pose estimation based on two-dimensional skeleton. This method usually consists of two stages. First, a pre-trained two-dimensional pose estimation network is used to extract a skeleton sequence. Then, the obtained skeleton is input into a three-dimensional pose estimation network for dimension lifting. Thanks to the maturity of existing two-dimensional pose estimation algorithms, the three-dimensional human pose estimation method based on two-dimensional skeleton greatly reduces the complexity of the entire task and outperforms the direct regression-based method, becoming the mainstream. This scheme can greatly reduce the complexity of the network structure and is easier to deploy in real environments. A typical case is to use a network constructed by a fully connected layer (Martinez J, Hossain R, Romero J, et al. A simple yet effective baseline for 3D human pose estimation. in Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2017: 2640-2649.) to map two-dimensional poses to three-dimensional poses. A series of experiments prove the feasibility of this scheme, which shows that a simple and lightweight network can be used to map two-dimensional human poses to three-dimensional human poses.

[0004] Although three-dimensional human pose estimation has made great progress in recent years, it still faces some research difficulties, mainly including occlusion, inherent depth ambiguity in two-dimensional to three-dimensional mapping, and lack of data sets. SUMMARY

[0005] To solve the problems in the prior art, the present application provides a three-dimensional human pose estimation method and system based on graph and attention interweaving. The present application uses graph convolution and attention mechanism to focus on local and global information of human skeleton at the same time, and further allows communication between the two, complementary advantages, to achieve more robust human skeleton modeling. In addition, the U-shaped structure multilayer perceptron designed by the present application is simple and efficient, and can be used to capture multi-level information of the skeleton structure.

[0006] The technical scheme adopted by the present application is as follows:

[0007] A three-dimensional human pose estimation method based on graph and attention interweaving, comprising the following steps:

[0008] The images in the three-dimensional human pose estimation dataset are used as training images;

[0009] The two-dimensional skeleton information of the human body is extracted from the input training image by using a two-dimensional pose detector;

[0010] The extracted two-dimensional skeleton information is mapped to a high-dimensional space by using a skeleton embedding module to obtain a high-dimensional vector;

[0011] The local and global information of the human skeleton contained in the high-dimensional vector obtained by the skeleton embedding module is mined by using a graph and attention interweaving network module;

[0012] The multi-level information of the human skeleton is extracted from the output of the graph and attention interweaving network module by using a U-shaped structure multilayer perceptron module;

[0013] The multi-level information of the human skeleton is regressed by using a regression head module to output a three-dimensional skeleton;

[0014] The mean square error of the joint points is used as a loss function to supervise the learning of the three-dimensional skeleton estimated by the regression head module, so as to train a three-dimensional human pose estimation model;

[0015] The two-dimensional skeleton information extracted from the image to be estimated by the two-dimensional pose detector is used as the input of the trained three-dimensional human pose estimation model, and sequentially passes through the skeleton embedding module, the graph and attention interweaving network module, the U-shaped structure multilayer perceptron module and the regression head module, and finally outputs a three-dimensional human pose estimation result.

[0016] Further, the three-dimensional human pose estimation is performed on the human body that can be detected in the image.

[0017] Further, the first two steps in the above method belong to the preprocessing stage, including the acquisition of training images in the dataset and the extraction of two-dimensional skeletons.

[0018] Further, the two-dimensional skeleton information is a result estimated directly from the image by using an existing two-dimensional human pose estimation algorithm.

[0019] Further, the skeleton embedding module comprises a multi-layer fully connected network, which gradually maps the two-dimensional skeleton to a high-dimensional space.

[0020] Further, the graph and attention interleaved network module combines graph convolutional network and attention mechanism to capture global and local information of the human skeleton. The graph and attention interleaved network module comprises two strategies: 1) Graph2Attention (G2A): the human body topology information extracted by the graph convolutional block is injected into the attention block, so that the attention block can better learn the structure information of the human skeleton under the guidance of the graph convolutional block; 2) Attention2Graph (A2G): the global association between the joints of the skeleton captured by the attention block is fed into the graph convolutional block, so that the graph convolutional block can better perceive the global information while focusing on the neighbor nodes.

[0021] Further, the structure of the graph convolutional block combines the topology prior of the human skeleton to capture the local information of the human skeleton. The local information refers to the fact that each key point focuses on the nodes adjacent to it, and tends to ignore the connection with the nodes far away.

[0022] Further, the topology prior of the human skeleton refers to the fact that in the adjacency matrix used to represent the human skeleton, each joint is not only connected to itself, but also connected to its adjacent joints, and the symmetric joints in the skeleton are connected. The inherent characteristics of the skeleton structure are expressed by means of the adjacency matrix.

[0023] Further, the attention block is used to capture the global information of the human skeleton. The global information refers to the fact that each joint establishes a connection with all joints, and each joint has a global perception of the entire skeleton.

[0024] Further, the U-shaped multi-layer perceptron module is composed of three layers of fully connected networks. The output of the first layer of fully connected network is halved in channel dimension compared to the input, the input and output of the second layer of fully connected network remain unchanged in channel dimension, and the third layer of fully connected network increases the channel dimension of the output to be consistent with the input. The dimension consistent part maintains the short cut connection, such as the shortcut connection between the input of the first layer of fully connected network and the output of the third layer of fully connected network, and the short cut connection between the input and output of the second layer of fully connected network.

[0025] Further, the regression head module comprises a two-layer fully connected network for regressing high-dimensional features to specific joint coordinates.

[0026] A three-dimensional human pose estimation system based on graph and attention interweaving comprises:

[0027] A preprocessing unit is configured to acquire a training image in a three-dimensional human pose estimation dataset and extract two-dimensional skeleton information from the input training image by using a two-dimensional human pose detector;

[0028] A model training unit is configured to map the extracted two-dimensional skeleton information to a high-dimensional space by using a skeleton embedding module, capture local and global information of the skeleton by using a graph and attention interweaving network module, capture multi-level information of the skeleton by using a U-shaped multi-layer perceptron module, finally regress high-dimensional features to obtain a three-dimensional skeleton by using a regression head module, use a mean square error of a joint as a loss function of three-dimensional human pose estimation supervised learning, and train a three-dimensional human pose estimation model.

[0029] A three-dimensional human pose estimation unit is configured to extract two-dimensional skeleton information of a human body in an image to be estimated by using a pre-trained two-dimensional pose estimation detector, input the extracted two-dimensional skeleton information into the trained skeleton embedding module, the graph and attention interweaving network module, the U-shaped multi-layer perceptron module, and the regression head module in sequence, and output a three-dimensional human pose estimation result.

[0030] The present application has the following advantages:

[0031] The present application can solve the problem of insufficient mining of local and global information of a human skeleton by the existing network structure by innovative combination of a graph convolution network and an attention mechanism. In the graph and attention interweaving network module, a graph convolution block structure with human skeleton topology structure prior is combined to capture local information of the human skeleton, and an attention block is used to capture global information of the human skeleton, so that the two blocks can communicate with each other, complement each other's advantages, and enhance the perception ability of the model to the local and global skeleton. Further, the proposed U-shaped multi-layer perceptron module is used to capture multi-level information contained in the skeleton.

[0032] The effect diagram of the present application is as follows Figure 2As shown, it can be seen that the present application can accurately estimate the three-dimensional human pose corresponding to various complex human motions. Compared with the MGCN method (Zhiming Zou and Wei Tang, Modulated graph convolutional network for 3D human pose estimation, in Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021, pp. 11477-11487.), our method can estimate results closer to the real three-dimensional pose. The present application can be introduced into a target recognition system, a human-computer interaction system, and realize more complete intelligent monitoring technology. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 The flow chart of the three-dimensional human pose estimation method based on the graph and attention interweaving of the present application.

[0034] Figure 2 The three-dimensional human pose estimation effect diagram of the present application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It can be understood that the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0036] Figure 1 The flow chart of the three-dimensional human pose estimation method based on the graph and attention interweaving of the present application includes the following steps:

[0037] Step 1: input the training set image and its corresponding data label. In the actual training process, the input image data is usually a batch of data to ensure that the model parameters can be stably optimized in the training optimization process.

[0038] Step 2: Extract human pose in the input training image using a two-dimensional pose estimation detector. This embodiment uses the existing method CPN (Chen Y, Wang Z, Peng Y, et al. Cascaded pyramid network for multi-person pose estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018:7103-7112.) to perform two-dimensional human pose estimation on the image of step 1, obtaining two-dimensional key point coordinates where N is the number of key points, set to 17.

[0039] Step 3: Use the skeleton embedding module (composed of multiple layers of fully connected networks) to encode the two-dimensional key point coordinates obtained in step 2 to obtain a high-dimensional vector where the number of channels C is set to 512.

[0040] Step 4: Graph and attention interwoven network module for obtaining local and global information of skeleton. The present invention combines graph convolutional network and attention mechanism, with two guiding strategies (Graph2Attention and Attention2Graph), to let graph convolution and attention mechanism better learn the representation of human skeleton.

[0041] where the Graph2Attention strategy is to guide the attention block to learn the topological prior of the human skeleton. The skeleton information f graph captured by the graph convolution block is injected into the attention block, and the specific calculation formula is as follows:

[0042]

[0043] where s G2A is the scaling factor of f graph , Softmax is an activation function that normalizes a numerical vector into a probability distribution vector, Q, K, V are the query matrix, key matrix and value matrix in the attention mechanism, d represents the dimension of the matrix Q, K, V, and X G2A represents the result of introducing local information from the graph convolution into the matrix product in the attention block. With the guidance of skeleton information from the graph convolution block, the attention mechanism's ability to capture human skeleton correlation information is enhanced.

[0044] where the Attention2Graph strategy is to make up for the deficiency of the graph convolution block in capturing global dependencies. The global information f global of the human skeleton captured by the attention block is fed back to the graph convolution block, so that the graph convolution block has a better understanding of the global association of the skeleton. The specific formula is as follows:

[0045] X A2G =G1+s A2G ·f global ,

[0046] where G1 represents the first graph convolution layer in the graph convolution block, s A2G is a scaling factor for the global information f global of the human skeleton, and X A2G represents the result after introducing global information from the attention block to G1. In this way, the global information of the skeleton can also be better perceived by the graph convolution block.

[0047] Under the interlaced guidance of complementary information from both sides, the perception ability of the graph convolution block and the attention block is enhanced. Finally, the outputs of the graph convolution block and the attention block are added. The calculation formula can be represented as:

[0048] X IGA =G2(X A2G )+Proj(X G2A ),

[0049] where G2(.) represents the second graph convolution layer in the graph convolution block, Proj(·) is a projection head containing 2 linear layers, and X IGA represents the sum of the outputs of the graph convolution block and the attention block under the guidance of complementary information.

[0050] Step 5: The features obtained in step 4 are further input into a multi-layer perceptron module of U-shaped structure to further extract multi-level information of the skeleton. This module performs up-sampling and down-sampling along the channel dimension. First, the input X IGA is sent to a down-sampling projection layer X down with half the channel dimension, followed by an intermediate layer X mid with the same channel dimension, and finally an up-sampling projection layer X up with doubled output channel dimension. The specific formula is as follows:

[0051] X down =MLP down (LN(X IGA )),

[0052] X mid =MLP mid (X down )+Xdown ,

[0053] X up = MLP up (X mid )+X IGA ,

[0054] Where MLP(·) is an MLP block containing one linear layer, and LN represents a layer normalization operation.

[0055] Step 6: regression is performed on the features obtained in step 5 using a regression head module, i.e., a two-layer fully connected network, to obtain predicted three-dimensional pose joints

[0056] Step 7: the error between the three-dimensional skeleton predicted in step 6 and the three-dimensional pose ground truth is calculated using a mean square error loss function of the joints, thereby training the skeleton embedding module, the graph and attention interleaved network module, the U-shaped multi-layer perceptron module, and the regression head module. The definition of the mean square error is:

[0057]

[0058] Where N = 17 represents the number of joints, J i is the coordinate of the i-th three-dimensional joint ground truth, and X i is the coordinate of the i-th predicted three-dimensional joint.

[0059] Step 8: three-dimensional human pose estimation is performed on the image to be estimated: first, the two-dimensional skeleton information of the human body is extracted from the image to be estimated using a two-dimensional pose detector, and then the extracted two-dimensional information is input into the trained skeleton embedding module, graph and attention interleaved network module, U-shaped multi-layer perceptron module, and regression head module to output the three-dimensional human pose estimation result.

[0060] The effect diagram of the present application is shown in Figure 2 , which shows that the present application can achieve accurate three-dimensional human pose estimation for various human actions.

[0061] Based on the same inventive concept, another embodiment of the present application is a three-dimensional human pose estimation system based on graph and attention interleaving, which comprises:

[0062] A preprocessing unit is configured to obtain a training image in a three-dimensional human pose estimation dataset and extract two-dimensional skeleton information from the input training image using a two-dimensional human pose detector.

[0063] The model training unit maps the extracted two-dimensional skeleton information to a high-dimensional space by using a skeleton embedding module, captures local and global information of the skeleton by using a graph and attention interwoven network module, captures multi-level information of the skeleton by using a U-shaped multi-layer perception module, and finally regresses high-dimensional features by using a regression head module to obtain a three-dimensional skeleton. A mean square error of a joint node is used as a loss function of supervised learning of three-dimensional human pose estimation, and a three-dimensional human pose estimation model is trained.

[0064] The three-dimensional human pose estimation unit extracts two-dimensional skeleton information of a human body in an image by using a pre-trained two-dimensional pose estimation detector, sequentially inputs the extracted two-dimensional skeleton information into a trained skeleton embedding module, a graph and attention interwoven network module, a U-shaped multi-layer perception module, and a regression head module, and outputs a three-dimensional human pose estimation result.

[0065] The specific implementation process of each unit and each module can be referred to the description of the method of the present application.

[0066] Based on the same inventive concept, another embodiment of the present application provides a computer device (computer, server, smart phone, etc.), which includes a memory and a processor, the memory stores a computer program, the computer program is configured to be executed by the processor, and the computer program includes instructions for executing each step in the method of the present application.

[0067] Based on the same inventive concept, another embodiment of the present application provides a computer readable storage medium (such as ROM / RAM, magnetic disk, optical disk), which stores a computer program, and the computer program is executed by a computer to realize each step of the method of the present application.

[0068] The above examples are only examples of the present application, although the best embodiments and drawings of the present application are disclosed for illustrative purposes, but those skilled in the art can understand that various alternatives, changes and modifications are possible without departing from the spirit and scope of the present application and the appended claims. Therefore, the present application should not be limited to the disclosed content of the best embodiments and drawings.

Claims

1. A 3D human pose estimation method based on graph and attention interleaving, comprising the following steps: Images from the 3D human pose estimation dataset are used as training images; Two-dimensional pose detectors are used to extract two-dimensional skeleton information of the human body from the input training images. The extracted two-dimensional skeleton information is mapped to a high-dimensional space using the skeleton embedding module to obtain a high-dimensional vector; We utilize a network module that interweaves graphs and attention to mine local and global information about the skeleton contained in the high-dimensional vectors obtained by the skeleton embedding module; A U-shaped multilayer perceptron module is used to extract multi-level information about the human skeleton from the output of a network module where graphs and attention are intertwined. The regression head module is used to regress the multi-level information of the extracted human skeleton and output a three-dimensional skeleton. The mean square error of the joints is used as the loss function for supervised learning to supervise the learning of the 3D skeleton estimated by the regression head module in order to train the 3D human pose estimation model. The two-dimensional skeleton information extracted from the image to be estimated by the two-dimensional pose detector is used as the input of the trained three-dimensional pose estimation model. It is then passed through the skeleton embedding module, the graph and attention intertwined network module, the U-shaped multilayer perceptron module and the regression head module in sequence, and finally the three-dimensional human pose estimation result is output. The graph and attention intertwined network module includes graph convolutional blocks and attention blocks, which combine the advantages of graph convolution and attention mechanisms in capturing local and global features of the human skeleton, and allow communication between the two to enhance the model's ability to model the skeleton. The U-shaped multilayer perceptron module consists of three fully connected networks. The output of the first fully connected network is halved in the channel dimension compared to the input. The input and output of the second fully connected network remain unchanged in the channel dimension. The third fully connected network increases the channel dimension of the output to match that of the input. Short-cut connections are maintained where the dimensions match, including a short-cut connection between the input of the first fully connected network and the output of the third fully connected network, and a short-cut connection between the input and output of the second fully connected network.

2. The method of claim 1, wherein, The graph convolutional block incorporates the topological prior of the human skeleton to capture local information about the human skeleton; the local information refers to focusing on the nodes that are close to each key point, while ignoring the connection with more distant nodes.

3. The method of claim 2, wherein, The topological prior of the human skeleton refers to the adjacency matrix used to represent the human skeleton, in which each joint is not only connected to itself, but also adjacent to its neighboring joints. Symmetrical joints in the skeleton are connected, and the inherent features of the skeleton structure are characterized by the adjacency matrix.

4. The method of claim 1, wherein, The attention block is used to capture global information of the human skeleton; the global information refers to the fact that each joint point establishes a connection with all relevant nodes, and each joint point has a global perception of the entire skeleton.

5. The method of claim 1, wherein, The skeleton embedding module contains a multi-layer fully connected network that gradually maps the two-dimensional input to a high-dimensional space; the regression head module contains a two-layer fully connected network that regresses the high-dimensional features to specific key coordinates.

6. A graph and attention interweaved based 3D human pose estimation system using the method of any one of claims 1-5, comprising: a pre-processing unit configured to obtain training images in a 3D human pose estimation dataset and extract 2D skeleton information from the input training images using a 2D human pose detector; a model training unit configured to map the extracted 2D skeleton information to a high-dimensional space using a skeleton embedding module, capture local and global information of the skeleton using a graph and attention interweaved network module, capture multi-level information of the skeleton using a U-shaped multi-layer perceptron module, and finally regress a 3D skeleton using a regression head module, use mean square error of the joints as a loss function for 3D human pose estimation supervised learning, and train a 3D human pose estimation model; a 3D human pose estimation unit configured to extract 2D skeleton information of a human body in an image using a pre-trained 2D pose estimation detector, sequentially input the extracted 2D skeleton information into the trained skeleton embedding module, the graph and attention interweaved network module, the U-shaped multi-layer perceptron module, and the regression head module, and output a 3D human pose estimation result.

7. A computer device, comprising: a computer readable storage medium storing a computer program configured to be executed by a processor, the computer program comprising instructions for performing the method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, a computer readable storage medium storing a computer program configured to be executed by a processor, the computer program comprising instructions for performing the method of any one of claims 1-5.

Images