A 3D Human Pose Estimation Method and System Based on Human Topology Sensing Network

Abstract

This invention relates to a 3D human pose estimation method and system based on a human topology perception network. This method utilizes the hierarchical structure and motion constraints of the human topology to fully capture prior information from sparse human joint data, effectively alleviating the problem of insufficient estimation accuracy of end-effector joints. This method constructs a multi-level feature extraction and aggregation framework, which can extract feature information at the joint level, limb level, and whole-body level sequentially from fine-grained to coarse-grained. It also constructs motion constraint relationships between joints within the same limb, using parent nodes to assist in the prediction of high-error end-effector joints. This invention fully utilizes the highly structured prior information in the human topology, achieving richer human pose representation learning capabilities while controlling the number of model parameters without introducing additional data.

Description

Technical Field

[0001] This invention belongs to the field of target recognition and intelligent human-computer interaction in computer vision, specifically relating to a three-dimensional human pose estimation method and system based on human topology perception network. Background Technology

[0002] 3D human pose estimation is a fundamental and active research area in computer vision. It aims to predict the 3D coordinates of key points on a human body from an image and thereby construct a skeletal representation of the human body. The human skeleton estimated by 3D human pose estimation algorithms can be further applied to tasks such as virtual reality, action recognition, and 3D human reconstruction, thus possessing significant research importance and application value.

[0003] Currently, 3D human pose estimation algorithms are mainly divided into two implementation methods: one-stage methods and two-stage methods. One-stage methods process RGB images, aiming to regress the 3D coordinates of human joints end-to-end. Although RGB images possess rich semantic information, various forms of clothing, arbitrary occlusion, viewpoint-induced occlusion, and background context all increase the difficulty of estimation. Classic one-stage pose estimation algorithms (Pavlakos G, Zhou X, Derpanis KG, et al. Coarse-to-fine volumetric prediction for single-image 3D humanpose. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2017:7025-7034.) are computationally intensive and struggle to accurately identify and track fine-grained joint coordinates in these complex scenes. Thanks to existing robust 2D human pose estimation algorithms, the two-stage method has been widely used since its emergence (Martinez J, Hossain R, Romero J, et al. A simple yet effective baseline for 3D human pose estimation. Proceedings of the IEEE International Conference on Computer Vision (ICCV). 2017:2640-2649.). It first uses an existing efficient 2D pose estimator to estimate 2D coordinates from an image, and then uses a 2D-to-3D boosting network to achieve the final 3D coordinate output. Extensive experiments have demonstrated that, whether in mainstream datasets or real-world scenarios, the two-stage method can achieve more accurate estimates using a more lightweight network.

[0004] However, unlike image-based input methods, the data processed by the two-stage method to enhance the network consists of sparse human joints. Therefore, fully utilizing the prior information about the human topology between these joints is crucial. Current work mainly references mainstream computer vision tasks, aiming to analyze and process human joints at both global and local levels, while neglecting the fact that human joints themselves are highly structured topological components. The hierarchical structure and motion constraints of the human body, as prior knowledge, can greatly help improve the accuracy of 3D human pose estimation. Summary of the Invention

[0005] To address the problems existing in current technologies, this invention analyzes the structural characteristics of the human body and provides a 3D human pose estimation method and system based on a human topology perception network. According to the hierarchical structure of the human body, this invention extracts and aggregates feature information sequentially from three levels: joints, limbs, and the whole body. Simultaneously, at the limb level, this invention designs a limb-internal constraint module, using parent joints within the same limb to constrain the movement of the distal joints, significantly reducing the motion error of the distal joints. Without introducing additional information, this invention effectively acquires richer representation learning capabilities by utilizing the topological relationships between human joints, and effectively improves the accuracy of prediction.

[0006] The technical solution adopted in this invention is as follows:

[0007] A three-dimensional human pose estimation method based on a human topology sensing network includes the following steps:

[0008] The coordinates of two-dimensional joints are extracted from the input image containing human information using a pre-trained two-dimensional human pose estimator.

[0009] The spatial feature embedding module is used to fill the two-dimensional joint coordinates into a high-dimensional vector;

[0010] Divide the high-dimensional vector into three sub-vectors;

[0011] In the first part of the sub-vectors, the physical connections between joints are modeled using the local joint construction module;

[0012] In the second part of the sub-vectors, the motion constraint relationship at the limb level is modeled using the limb internal constraint module;

[0013] In the third part of the sub-vectors, the global information interaction module is used to model long-distance interactions at the body level;

[0014] The hierarchical information extracted from each sub-vector is sequentially connected and aggregated;

[0015] High-level semantic features are extracted from aggregated high-dimensional vectors using a spatial perceptron model.

[0016] The prediction head module is used to regress high-level semantic features, and finally outputs the three-dimensional coordinates of the key points.

[0017] Furthermore, the root mean square loss function is used to perform supervised learning between the model's output and the true value.

[0018] Furthermore, the local joint construction module primarily utilizes graph convolutional networks, employing adjacency matrices to mimic the physical connections between human joints. The adjacency matrix in the graph convolutional network only considers the physical connections between human joints and the connection between each keypoint and itself. Preferably, the local joint construction module uses a unified framework of two graph convolutions, two layer normalizations, and an activation function to simulate the hierarchical physical connections between human joints.

[0019] Furthermore, the limb constraint module consists of two sets of one-dimensional standard convolutions and a spatial perceptron. This module uses simple one-dimensional standard convolutions and replacement operations to introduce parent nodes within the same limb to constrain the motion of end joints with high motion complexity. This topological constraint can effectively mitigate errors in end joints and assist in generating more reasonable three-dimensional pose estimations.

[0020] Furthermore, the two sets of one-dimensional standard convolutions have kernels of 2 and 3, respectively, and their stride values ​​are also 2 and 3, respectively.

[0021] Furthermore, the global information interaction module utilizes a self-attention mechanism to capture long-distance features at the level of whole-body joints.

[0022] Furthermore, the self-attention mechanism includes multiple self-attention heads and only considers spatial information interaction between key points.

[0023] Furthermore, the process of sequentially connecting and aggregating the hierarchical information combines parallel and serial frameworks. On the one hand, the high-dimensional feature channels of the input are split and parallel processing is used to control the number of model parameters. On the other hand, the three sub-modules are sequentially connected to obtain feature extraction from fine-grained to coarse-grained.

[0024] Furthermore, the prediction head module includes a fully connected network layer.

[0025] A 3D human pose estimation system based on a human topology sensing network, comprising:

[0026] The 2D joint coordinate extraction module is used to extract 2D joint coordinates from images containing human information using a pre-trained 2D human pose estimator.

[0027] The spatial feature embedding module is used to fill the coordinates of two-dimensional key points into a high-dimensional vector;

[0028] The channel splitting module is used to divide a high-dimensional vector into three sub-vectors;

[0029] The local joint construction module is used to model the physical connections between joints using the first part of the sub-vectors;

[0030] The limb internal constraint module is used to model the motion constraint relationships at the limb level using the second part of the sub-vectors;

[0031] The global information interaction module is used to model long-distance interactions at the body level using the third part of the sub-vectors;

[0032] The channel merging module is used to sequentially connect and aggregate the hierarchical information extracted from each sub-vector;

[0033] The spatial perceptron module is used to extract high-level semantic features from the aggregated high-dimensional vectors.

[0034] The prediction head module is used to regress high-level semantic features and ultimately output the three-dimensional coordinates of the key points.

[0035] The beneficial effects of this invention are as follows:

[0036] This invention, through further exploration of human body topology, utilizes the hierarchical structure and motion constraints of human body topology to propose a 3D human pose estimation method and system based on a human topology perception network. This method can fully leverage prior information about the human body structure and solve the problem of sparse input joint point data density. The effect of this invention in application is shown in the figure below. Figure 3 , Figure 4 As shown, even in unfamiliar outdoor scenes, this invention can accurately estimate complex and occluded human movements. As an effective baseline model, this invention can be incorporated into downstream human tasks such as 3D human reconstruction and motion recognition, thereby achieving more comprehensive human modeling and analysis. Attached Figure Description

[0037] Figure 1 The flowchart of the human body topology sensing network of the present invention.

[0038] Figure 2 The flowcharts for the local joint construction module, the limb internal constraint module, and the global information interaction module of the present invention are shown below.

[0039] Figure 3 The comparison between the predicted values ​​and the actual values ​​of this invention.

[0040] Figure 4 Visualization of 3D human pose estimation in outdoor scenes based on the present invention. Detailed Implementation

[0041] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] Reference Figure 1 The present invention provides a three-dimensional human pose estimation method based on a human topology sensing network, comprising the following steps:

[0043] 1) Extract the two-dimensional coordinates of the joints from the input image using an existing two-dimensional human pose estimator;

[0044] 2) The spatial coordinate information in 1) is filled into a high-dimensional vector using the feature embedding module; the filling includes spatial filling and position filling. Spatial filling is to encode the skeleton coordinates into a high-dimensional vector using a fully connected network layer, and position filling is to use a set of learnable variables to define the sequential relationship between the joints.

[0045] 3) Divide the high-dimensional vector in 2) into three parts;

[0046] 4) For the first part in 3), the physical connection between joints is modeled using the local joint construction module;

[0047] 5) For the second part in 3), the motion constraint relationship at the limb level is modeled using the limb internal constraint module;

[0048] 6) For the third part in 3), the global information interaction module is used to model long-distance interactions at the body level;

[0049] 7) Connect and aggregate the information extracted from each level in 4), 5), and 6) sequentially;

[0050] 8) Utilize modern spatial multilayer perceptron networks to perform advanced semantic feature processing on the high-dimensional vectors in 7).

[0051] 9) The high-level semantic features extracted in 8) are regressed using the prediction head module, and the three-dimensional coordinates of the joints are finally output.

[0052] 10) Supervised learning is performed between the output and the true value in 9) using the root mean square loss function.

[0053] In step 1), the 2D human pose estimator references CPN (Chen Y, Wang Z, Peng Y, et al. Cascaded pyramid network for multi-person pose estimation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018:7103-7112.), which is a mainstream and robust 2D human pose estimator in current two-stage algorithms. The input is an RGB image containing human information, and the pre-trained 2D human pose estimator extracts the 2D joint coordinates from the image.

[0054] In step 4), the local joint construction module mainly uses graph convolutional networks and uses adjacency matrices to simulate the physical connection relationship between human joints. It mainly extracts features between directly connected joints.

[0055] In step 5), the limb constraint module focuses on the features inside the limbs. It uses a convolutional neural network to extract the features of each limb and fuses these limb features with the high-error limb end joints. It effectively uses the movement trend of the entire limb to constrain the end joints that are highly flexible and therefore difficult to predict.

[0056] Among them, the global information interaction module in step 6) mainly focuses on long-distance inter-joint information interaction, and uses a self-attention mechanism to capture inter-joint interactions at the whole body level.

[0057] In steps 4), 5), and 6), the structural information in the human body topology is extracted hierarchically by channel, connected sequentially, and aggregated in step 8), thereby obtaining richer multi-level information while maintaining the total number of parameters.

[0058] The regression process described in step 9) is implemented through a fully connected layer.

[0059] The following further explains that the human topology perception network of the present invention mainly includes three sub-modules targeting different levels of human body structure: local joint point construction module, limb internal constraint module, and global information interaction module.

[0060] (1) Local joint construction module

[0061] This module utilizes graph convolutional networks to construct physical connections between human body joints, defined as:

[0062]

[0063] in, This represents the high-dimensional features on C channels of the 17 joints of the human body in layer l, where σ is the activation function. W is a diagonal node degree matrix, and W is a learnable weight matrix. Let A denote the symmetric normalized adjacency matrix, and I be the identity matrix. In this invention, the adjacency matrix focuses only on physically directly connected joints. Therefore, this module primarily focuses on the feature information at the level of adjacent joints.

[0064] In one embodiment of the present invention, the local joint construction module utilizes a unified framework of two graph convolutions, two layer regularizations, and a "Gaussian error linear unit" activation function to simulate the physical connection relationship of human joint hierarchy.

[0065] (2) Limb restraint module

[0066] Because joints within the same limb often exhibit similar movement trends, and can be categorized as upstream and downstream joints based on their distance from the trunk, upstream joints (e.g., the shoulder joint) have smaller ranges of motion and simpler postures, often assisting in and constraining the prediction of downstream joints (e.g., the elbow and wrist joints). (Refer to...) Figure 2 The present invention first selects two sets of joint points in the limbs: and The former consists of two joints at the distal ends of the four limbs, while the latter consists of all three joints of the four limbs. Next, this invention utilizes convolution operations to extract limb features characterizing the movement trends of each limb.

[0067]

[0068]

[0069] In this context, GELU(·) is the activation function, Conv1 and Conv2 are one-dimensional standard convolutions with kernels of 2 and 3, respectively, LN(·) is the standard layer regularization, and MLP is a spatial multilayer perceptron.

[0070] Subsequently, this module utilizes this motion constraint relationship to construct two topological constraints through a simple replacement operation. For X1, limb features are used... To replace the joint at the very end of the limb, we obtain Y1, because... It includes features of both upstream parent joints and end joints, thus allowing for effective use of the motion features of upstream parent joints to constrain the motion of end joints with high errors; similarly, for X2, the constraint range is expanded, represented by limb features. This replaces the two distal joints of the limb, resulting in Y2. Finally, the output of this module is a combination of the original input X and the replaced features:

[0071] Y = Y1 + Y2 + X

[0072] In summary, introducing a parent node within the same limb to constrain the motion of end joints with high motion complexity can effectively mitigate end joint errors and assist in generating more reasonable 3D pose estimations.

[0073] (3) Global Information Interaction Module

[0074] This module utilizes a self-attention mechanism to effectively capture long-distance features between joints throughout the body, a technique widely used in human pose estimation. Its specific implementation is shown below:

[0075] X l+1 =Concat(H1,H2,...,H) h W out

[0076]

[0077] Where h is the number of self-attention heads, Q l ,K l V l They are respectively made by X l The input query, index, and content matrix are calculated through linear transformations. `Concat` represents the join operation, where H1, H2, ..., H... h W represents the attentional head in the self-attention mechanism, h represents the number of attentional heads, and W represents the number of attentional heads. out This represents the learnable weight matrix.

[0078] To address the connection method of these three sub-modules, this invention designs a hierarchical connection structure. On one hand, the high-dimensional feature channels of the input are split, and parallel processing is used to control the number of model parameters; on the other hand, this invention serially connects the three sub-modules that process the features at the joint, limb, and whole-body levels, respectively, thereby obtaining feature extraction from fine-grained to coarse-grained levels. Figure 3 and Figure 4 The visualization results fully demonstrate the accuracy and robustness of the invention, and can inspire more inventive research on the prior knowledge of human topology.

[0079] Another embodiment of the present invention provides a three-dimensional human pose estimation system based on a human topology sensing network, comprising:

[0080] The 2D joint coordinate extraction module is used to extract 2D joint coordinates from images containing human information using a pre-trained 2D human pose estimator.

[0081] The spatial feature embedding module is used to fill the coordinates of two-dimensional key points into a high-dimensional vector;

[0082] The channel splitting module is used to divide a high-dimensional vector into three sub-vectors;

[0083] The local joint construction module is used to model the physical connections between joints using the first part of the sub-vectors;

[0084] The limb internal constraint module is used to model the motion constraint relationships at the limb level using the second part of the sub-vectors;

[0085] The global information interaction module is used to model long-distance interactions at the body level using the third part of the sub-vectors;

[0086] The channel merging module is used to sequentially connect and aggregate the hierarchical information extracted from each sub-vector;

[0087] The spatial perceptron module is used to extract high-level semantic features from the aggregated high-dimensional vectors.

[0088] The prediction head module is used to regress high-level semantic features and ultimately output the three-dimensional coordinates of the key points.

[0089] For the specific implementation process of each module, please refer to the description of the method of the present invention above.

[0090] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0091] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0092] The above examples are merely illustrative of the present invention. Although preferred embodiments and accompanying drawings have been disclosed for illustrative purposes, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the preferred embodiments and accompanying drawings.

Claims

1. A three-dimensional human pose estimation method based on a human topology sensing network, characterized in that, Includes the following steps: Two-dimensional joint coordinates are extracted from images containing human information using a pre-trained two-dimensional human pose estimator. The spatial feature embedding module is used to fill the two-dimensional joint coordinates into a high-dimensional vector; Divide the high-dimensional vector into three sub-vectors; In the first part of the sub-vectors, the physical connection between joints is modeled using a local joint construction module; the local joint construction module adopts a graph convolutional network and uses an adjacency matrix to simulate the physical connection relationship between human joints. The adjacency matrix only considers the physical connections between human joints and the connection between each key point and itself. In the second part of the sub-vectors, the limb-level motion constraint relationship is modeled using the limb-internal constraint module. The limb-internal constraint module consists of two sets of one-dimensional standard convolutions and a spatial perceptron. This module uses one-dimensional standard convolutions and a replacement operation to introduce parent nodes in the same limb to constrain the motion of end joints with high motion complexity. The two sets of one-dimensional standard convolutions have convolution kernels of 2 and 3, and their stride values ​​are also 2 and 3, respectively. In the third sub-vector, the global information interaction module is used to model long-distance interactions at the body level; the global information interaction module uses a self-attention mechanism to capture long-distance features at the whole body joint level; the self-attention mechanism includes multiple self-attention heads and only considers spatial information interactions between joints. The hierarchical information extracted from each sub-vector is sequentially connected and aggregated. The process of sequentially connecting and aggregating the hierarchical information extracted from each sub-vector combines parallel and serial frameworks. On the one hand, the high-dimensional feature channels of the input are split and parallel processing is used to control the number of model parameters. On the other hand, the local joint construction module, the limb constraint module and the global information interaction module are sequentially connected to obtain feature extraction from fine-grained to coarse-grained. High-level semantic features are extracted from aggregated high-dimensional vectors using a spatial perceptron model. The prediction head module is used to regress high-level semantic features, and finally outputs the three-dimensional coordinates of the key points.

2. The method as described in claim 1, characterized in that, The root mean square loss function is used to supervise the learning process between the output 3D coordinates of the joints and the true values.

3. The method as described in claim 1, characterized in that, The prediction head module contains a fully connected layer.

4. A three-dimensional human pose estimation system based on a human topology sensing network, employing the method described in any one of claims 1 to 3, characterized in that, include: The 2D joint coordinate extraction module is used to extract 2D joint coordinates from images containing human information using a pre-trained 2D human pose estimator. The spatial feature embedding module is used to fill the coordinates of two-dimensional key points into a high-dimensional vector; The channel splitting module is used to divide a high-dimensional vector into three sub-vectors; The local joint construction module is used to model the physical connections between joints using the first part of the sub-vectors; The limb internal constraint module is used to model the motion constraint relationships at the limb level using the second part of the sub-vectors; The global information interaction module is used to model long-distance interactions at the body level using the third part of the sub-vectors; The channel merging module is used to sequentially connect and aggregate the hierarchical information extracted from each sub-vector; The spatial perceptron module is used to extract high-level semantic features from the aggregated high-dimensional vectors. The prediction head module is used to regress high-level semantic features and ultimately output the three-dimensional coordinates of the key points.

5. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 3.

Images