A three-dimensional human pose estimation method based on two-dimensional and three-dimensional real label double manifold alignment
By separating the structural and noise information of 2D detected pose using a dual manifold alignment method, an autoencoder network is constructed, achieving high-precision and robust 3D human pose estimation from 2D detection to 3D reconstruction. This solves the problems of low accuracy and reliance on high-cost labeled data in existing technologies, and improves the adaptability and accuracy of the model.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING POLYTECHNIC
- Filing Date
- 2026-03-20
- Publication Date
- 2026-06-05
AI Technical Summary
Existing 3D human pose estimation techniques suffer from low accuracy and insufficient robustness in complex scenarios. They also rely on high-cost 3D labeled data, making it difficult to generalize to diverse scenarios. Furthermore, the lack of effective correlation and collaborative learning between 2D and 3D features leads to unreasonable reconstructed pose structures.
By using a dual-manifold alignment method based on 2D and 3D real annotations, the geometric structure information of the pose and the distribution information of the detection noise are separated, and 2D and 3D autoencoder networks are constructed. By minimizing mutual information and using the alignment loss function, end-to-end reconstruction from 2D detected pose to 3D real pose is achieved.
It achieves high-precision and robust 3D human pose estimation in complex scenarios, reduces the dependence on 3D labeled data, ensures that the reconstructed pose conforms to the laws of human kinematics, and improves the generalization ability and robustness of the model.
Smart Images
Figure CN122157310A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision and artificial intelligence technology, specifically relating to a 3D human pose estimation method based on the alignment of two-dimensional and 3D real-world labeled manifolds. This method can be applied to scenarios such as motion capture, human-computer interaction, virtual reality / augmented reality, intelligent monitoring, sports motion analysis, and medical rehabilitation assessment. It primarily addresses the technical problem of accurately estimating 3D human pose from 2D pose data. By using a two-manifold alignment mechanism, it improves the robustness and physiological plausibility of pose estimation, reduces dependence on 3D labeled data, and provides an efficient solution for low-cost 3D human pose perception. Background Technology
[0002] 3D human pose estimation technology is one of the core research directions in computer vision and artificial intelligence. Its core objective is to recover the 3D spatial coordinates of human joints from a single 2D image, video frame, or multi-view visual data, thereby achieving accurate quantification and semantic understanding of human pose. As a key bridge connecting visual perception and interaction with the physical world, this technology has been widely applied in various sectors of the national economy and people's livelihoods, including virtual reality and augmented reality, motion capture for film and animation, human-computer intelligent interaction, sports motion analysis, medical rehabilitation assessment, intelligent security monitoring, and robot collaborative control, demonstrating extremely high engineering application value and industrial development potential.
[0003] With the rapid iteration of deep learning technology, two main technical routes have gradually emerged for 3D human pose estimation: single-stage end-to-end estimation and two-stage 2D pose enhancement. Single-stage methods directly map from image pixel space to 3D pose coordinates. While offering the convenience of end-to-end training, they are limited by the inherent depth ambiguity of the 2D-to-3D mapping. In scenarios with human limb occlusion, complex pose changes, low lighting, or cluttered backgrounds, they are prone to sharp drops in pose estimation accuracy and unreasonable skeletal topology. Furthermore, single-stage models need to learn feature extraction and spatial inference simultaneously, resulting in high network complexity, unstable training, and difficulty adapting to large-scale real-world deployment requirements. Two-stage methods, relying on mature 2D human pose detection technology, first extract 2D coordinates of joints from the image using a 2D pose estimation network, and then map the 2D features to 3D space to complete pose reconstruction. This approach fully utilizes the accumulated technology in 2D detection, offering advantages such as stable convergence and high inference efficiency, making it the current mainstream choice. However, existing two-stage technologies still suffer from three major technical bottlenecks: First, the ambiguity and robustness of 2D-to-3D mapping are insufficient, making it prone to pose deviations and structural inconsistencies in complex scenarios. Second, the high cost and insufficient scene diversity of 3D labeled data acquisition lead to weak model generalization ability and susceptibility to domain shift. Third, the lack of effective correlation and collaborative learning between 2D and 3D features results in structural defects such as skeletal disproportion during pose reconstruction. Although existing technologies attempt to introduce human prior models, graph convolutions, or temporal convolutional networks for optimization, they still suffer from limitations such as poor generalization, insufficient utilization of the distribution characteristics of 2D real-world labels, and inability to bridge the semantic gap between 2D and 3D features, making it difficult to balance accuracy and robustness.
[0004] Therefore, how to overcome the limitations of existing technical approaches and construct a three-dimensional human pose estimation method that can effectively integrate the distribution characteristics of two-dimensional real annotations with the characteristics of three-dimensional structural manifolds, does not rely on large-scale three-dimensional annotation data, and ensures that the reconstructed pose conforms to the laws of human anatomy has become a key technical problem that urgently needs to be solved in this field, and is also the core breakthrough point for promoting the deep application of three-dimensional human pose estimation technology. Summary of the Invention
[0005] The purpose of this invention is to provide a three-dimensional human pose estimation method based on the alignment of two-dimensional and three-dimensional real-labeled manifolds, so as to solve the problems mentioned in the background art.
[0006] To achieve the above objectives, the present invention provides the following technical solution: A three-dimensional human pose estimation method based on alignment of two-dimensional and three-dimensional ground truth labeled dual manifolds includes the following steps: 1. Two-dimensional detection pose feature encoding and decoupling. Input two-dimensional detection attitude data Where N=17 represents the number of joints in the human body. First, a two-dimensional pose encoding network is used to map this number into high-dimensional features. To separate the geometric structure information of the pose from the distribution information of the detection noise, the feature F is input into the decoupled network and decomposed into two independent low-dimensional features: structure-related features. and distribution-related characteristics The decoupling process is achieved by minimizing mutual information, ensuring that the two are semantically independent. Through this decoupling mechanism, structurally relevant features focus on encoding the topological relationships and geometric structures between joints, while distribution-related features capture noise patterns introduced by factors such as detection errors and occlusion, laying the foundation for subsequent separate alignment.
[0007] 2. Construction of 3D structural manifolds During the training phase, real 3D pose data is used. Construct a 3D pose autoencoder network, which consists of an encoder. and decoder Composition. The encoder maps the three-dimensional true pose to a low-dimensional latent space. The decoder reconstructs the 3D pose from the latent variables. The following constraints are used during training to ensure that the latent space can effectively represent the distribution of reasonable 3D poses: 3D Reconstruction Loss : To ensure reconstruction accuracy.
[0008] Bone length consistency loss: for each bone connection The ratio of the predicted bone length to the actual bone length is close to 1, i.e. This loss can prevent abnormal postures that are not in proportion to the human body.
[0009] After training, fix the encoder. and decoder To obtain the three-dimensional structure manifold space Any point in this space can be decoded into a three-dimensional posture that conforms to the laws of human kinematics.
[0010] 3. Construction of Two-Dimensional Realistic Attitude Distribution Space Simultaneously, using two-dimensional real-world pose annotations Construct a two-dimensional attitude autoencoder network, which is a routing coding network. and decoder Composition. The encoder maps the two-dimensional real pose to the latent space. The decoder reconstructs the two-dimensional pose from the latent variables. During training, a two-dimensional reconstruction loss is used to ensure that the latent space can effectively represent the distribution of reasonable two-dimensional poses. Two-dimensional reconstruction loss : To ensure reconstruction accuracy.
[0011] After training, fix the encoder. and decoder The two-dimensional true attitude distribution space is obtained. This space describes an ideal two-dimensional pose distribution without detection noise, providing a reference benchmark for subsequent distribution alignment.
[0012] 4. Dual spatial alignment To achieve accurate mapping from 2D detected pose to 3D real pose, this invention designs an alignment mechanism: 3D manifold alignment: aligning structurally relevant features via a mapping network M. Mapping to the latent space of a three-dimensional manifold yields the predicted latent variables. Alignment loss constraint Potential representation of near-real three-dimensional pose : This loss prompts the network to learn embeddings from two-dimensional structural features into three-dimensional manifolds, thereby recovering accurate geometry.
[0013] Two-dimensional distribution alignment: for distribution-related features Apply alignment constraints to approximate the true two-dimensional distribution space. Feature representation in: This constraint guides the network to extract statistical features consistent with the true distribution from the detection noise, enabling the distribution-related features to encode the patterns of detection errors, thereby assisting the subsequent decoding process.
[0014] 5. 3D Attitude Regression and Output Aligned 3D latent variables and distribution-related characteristics After stitching, input into the fixed 3D decoder The final 3D pose prediction result is obtained: The loss function is as follows: To ensure the accuracy of 3D human pose estimation.
[0015] During the inference phase, only the two-dimensional detection pose needs to be input. Through encoding, decoupling, mapping, and decoding, high-precision 3D pose can be output without the need for 2D real-world annotations or 3D real-world data. The entire process achieves end-to-end reconstruction from noisy 2D detection to clean 3D pose.
[0016] 6. Loss Function The total training loss function is a weighted sum of the losses from the above components: in To balance the weights of the various losses, these weights are chosen to ensure that the magnitudes of the various losses are similar, thus preventing any one loss from dominating the training process.
[0017] Compared with existing technologies, the beneficial effects of this invention are as follows: Through joint optimization, the network fully learns the mapping relationship between 2D detected pose, 2D real distribution, and 3D structural manifold during the training phase, achieving effective alignment between the two spaces. This results in robust and accurate 3D reconstruction results during the inference phase. The entire technical solution theoretically guarantees that even with large detection errors in the input, a high-quality 3D pose conforming to human kinematics can be recovered through the decoupling of structural features and distribution features, as well as a multi-level alignment mechanism. Attached Figure Description
[0018] Figure 1 This is a schematic diagram illustrating the principle of the method of the present invention.
[0019] Figure 2 This is a diagram showing the 3D human pose estimation results on the Human3.6M dataset in an embodiment of the present invention.
[0020] Figure 3 This is a diagram showing the encoding and decoupling of two-dimensional detection posture features in an embodiment of the present invention.
[0021] Figure 4 This is a diagram showing the construction of a three-dimensional structural manifold in an embodiment of the present invention.
[0022] Figure 5 This is a diagram showing the construction of a three-dimensional structural manifold in an embodiment of the present invention.
[0023] Figure 6 This is a dual-space alignment diagram in an embodiment of the present invention.
[0024] Figure 7 This is a three-dimensional attitude output diagram in an embodiment of the present invention. Detailed Implementation
[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0026] Please see Figure 1In this embodiment of the invention, a three-dimensional human pose estimation method based on the alignment of two-dimensional and three-dimensional real-labeled dual manifolds includes the following steps: 1. Two-dimensional detection pose feature encoding and decoupling This step aims to encode the 2D pose to extract feature information and effectively separate structural information from noise information. First, the input is 2D detected pose data output by a general 2D human pose detector. Where N=17 is the number of human joints, and each joint contains planar coordinates (x, y). A two-dimensional pose coding network is used. Map it to a high-dimensional feature space and output the high-dimensional features. d represents the high-dimensional feature dimension. To separate the geometric structure information of the pose from the distribution information of the detection noise, the high-dimensional feature F is input into a decoupling network. This network contains two independent feature branches, decomposing F into two independent low-dimensional features: structure-related features. and distribution-related characteristics The decoupling process is achieved by minimizing mutual information, ensuring that the two are semantically independent. Through this decoupling mechanism, structurally relevant features focus on encoding stable and invariant structural information such as joint topological relationships and limb geometric proportions; distribution-related features focus on capturing changing information such as detection errors, occlusion patterns, and noise distribution, providing a foundation for subsequent alignment of the two manifolds.
[0027] 2. Construction of 3D structural manifolds This step utilizes real 3D pose data to construct a reasonable 3D pose manifold space, ensuring that subsequent pose predictions conform to human kinematics. During the training phase, real 3D pose data is used... Each joint contains three-dimensional coordinates (x, y, z), and a three-dimensional pose autoencoder network is constructed. This network consists of an encoder. and decoder Composition. Encoder Map the 3D true pose to a low-dimensional latent space and output the latent vector. Let k be the latent dimension of the three-dimensional manifold space. Decoder By inverse mapping the low-dimensional latent vectors, the three-dimensional pose can be reconstructed. The following constraints are used during training to ensure that the latent space can effectively represent the distribution of reasonable 3D poses: 3D Reconstruction Loss : To ensure reconstruction accuracy.
[0028] Bone length consistency loss: for each bone connection The ratio of the predicted bone length to the actual bone length is close to 1, i.e. This loss can prevent abnormal postures that are not in proportion to the human body.
[0029] After the 3D pose autoencoder is trained, the encoder is fixed. and decoder To obtain the three-dimensional structure manifold space Any point in this space can be decoded into a three-dimensional posture that conforms to the laws of human kinematics.
[0030] 3. Construction of Two-Dimensional Realistic Attitude Distribution Space This step utilizes two-dimensional ground truth annotations to construct a noise-free standard distribution space, providing a benchmark for subsequent noise feature alignment. The pose is then utilized based on the two-dimensional ground truth annotations. Construct a two-dimensional attitude autoencoder network, which consists of an encoding network. and decoder Composition. The encoder maps the two-dimensional real pose to the latent space. The decoder reconstructs the two-dimensional pose from the latent variables. Two-dimensional reconstruction loss is used during training. : This ensures accurate representation and reconstruction of the true 2D pose. After training, the encoder is fixed. and decoder The two-dimensional true attitude distribution space is obtained. This space represents an ideal two-dimensional pose distribution with no detection noise and no bias, providing a standard reference for subsequent distribution alignment.
[0031] 4. Dual spatial alignment This step aligns the decoupled features with the 3D manifold and the 2D real distribution, respectively, to achieve synergistic optimization of structural and distribution information. Therefore, to achieve accurate mapping from 2D detected pose to 3D real pose, this invention designs an alignment mechanism: 3D manifold alignment: aligning structurally relevant features via a mapping network M. Mapping to the latent space of a three-dimensional manifold yields the predicted latent variables. Alignment loss constraint Potential representation of near-real three-dimensional pose : This loss prompts the network to learn embeddings from two-dimensional structural features into three-dimensional manifolds, thereby recovering accurate geometry.
[0032] Two-dimensional distribution alignment: aligning distribution-related features With two-dimensional true latent vectors Apply alignment constraints to approximate the true two-dimensional distribution space. Feature representation in: This constraint guides the network to extract statistical features consistent with the true distribution from the detection noise, enabling the distribution-related features to encode the patterns of detection errors, thereby assisting the subsequent decoding process.
[0033] 5. 3D Attitude Regression and Output This step fuses the aligned features and inputs them into the 3D decoder, outputting the final high-precision 3D pose. Specifically, it fuses the aligned 3D latent variables... and distribution-related characteristics After stitching, input into the fixed 3D decoder The decoder, based on fused features, utilizes both structural geometry and noise correction information to output the final 3D predicted pose: To ensure output accuracy, a 3D attitude prediction loss is set. The direct constraint ensures that the final prediction result is consistent with the actual 3D pose.
[0034] During the inference phase, only the two-dimensional detection pose needs to be input. Through encoding, decoupling, mapping, and decoding, high-precision 3D pose can be output without the need for 2D real-world annotations or 3D real-world data. The entire process achieves end-to-end reconstruction from noisy 2D detection to clean 3D pose.
[0035] 6. Loss Function The total training loss function is a weighted sum of the losses from the above components: in To balance the weights of the various losses, these weights are chosen to ensure that the magnitudes of the various losses are similar, thus preventing any one loss from dominating the training process. For 3D pose prediction loss, For 3D manifold alignment loss, The alignment loss is a two-dimensional distribution.
[0036] like Figure 2 As shown, this invention has been validated on the public dataset Human3.6M and has achieved good experimental results. Figure 2 The qualitative experimental results of 3D pose estimation on the Human3.6M dataset are presented. The figures show the 2D input pose, the true 3D pose, and the 3D pose predicted by this invention. As can be seen from the figures, the prediction results of the proposed method are in high agreement with the true 3D pose for different human movements, verifying that the method can accurately and robustly predict 3D human pose from 2D input.
[0037] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.
[0038] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.
Claims
1. A three-dimensional human pose estimation method based on alignment of two-dimensional and three-dimensional real-annotated dual manifolds, characterized in that, Includes the following steps: 1) Decoupling of 2D detection pose features: Input two-dimensional detection attitude data Where N is the number of joints in the human body, and each joint has 2 coordinates, the first step is to use a two-dimensional pose encoding network. Map it to high-dimensional features d represents the dimension of the high-dimensional feature, which is the geometric structure information of the separation pose and the distribution information of the detection noise. The obtained high-dimensional feature... The input decoupled network is decomposed into two independent low-dimensional features: structurally relevant features. and distribution-related characteristics , and These are the structural feature dimension and the distribution feature dimension, respectively. 2) Construction of three-dimensional structural manifolds: During the training phase, real 3D pose data is used. Construct a 3D pose autoencoder network, with 3 coordinates for each joint. The 3D pose autoencoder network consists of an encoder. and decoder Composition, encoder Mapping the true 3D pose to a low-dimensional potential space decoder Reconstructing 3D pose from latent variables After training, fix the encoder. and decoder To obtain the three-dimensional structure manifold space Any point in this space can be decoded into a three-dimensional posture that conforms to the laws of human kinematics; 3) Construction of a two-dimensional real distribution space: Using 2D real-world pose annotation Construct a two-dimensional attitude autoencoder network, which is a routing coding network. and decoder Composition, encoder Mapping the two-dimensional real pose to the latent space l represents the potential spatial dimension of the two-dimensional true distribution; decoder Reconstructing 2D Pose from Latent Variables After training, fix the encoder. and decoder The two-dimensional true attitude distribution space is obtained. This space describes an ideal two-dimensional pose distribution without detection noise, providing a reference benchmark for subsequent distribution alignment. 4) Dual spatial alignment: 3D manifold alignment: aligning structurally relevant features via a mapping network M. Mapping to the latent space of a three-dimensional manifold yields the predicted latent variables. , k represents the latent space dimension of the three-dimensional manifold, and the alignment loss constraint Potential representation of near-real three-dimensional pose Then the alignment loss of the three-dimensional manifold as follows: in The loss is L2 norm, which enables the network to learn embeddings from two-dimensional structural features to three-dimensional manifolds, thereby recovering accurate geometric structures. Two-dimensional distribution alignment: for distribution-related features Apply alignment constraints to approximate the true two-dimensional distribution space. The feature representation in the two-dimensional distribution alignment loss as follows: This constraint guides the network to extract statistical features consistent with the true distribution from the detection noise, enabling the distribution-related features to encode the pattern of the detection error, thereby assisting the subsequent decoding process; 5) 3D pose regression and output: Aligned 3D latent variables and distribution-related characteristics After stitching, input into the fixed 3D decoder The final 3D pose prediction result is obtained: Where [,] represents feature concatenation, and The pieces are then stitched together and used as input for the 3D decoder. The loss function for the 3D pose prediction in this part is as follows: ; During the inference phase, only the two-dimensional detection pose needs to be input. High-precision three-dimensional pose can be output through encoding, decoupling, mapping and decoding; 6) Loss function: Total training loss function The weighted sum of the losses from the above components: in Weighting coefficients to balance the various losses.
2. The three-dimensional human pose estimation method based on the alignment of two-dimensional and three-dimensional real-annotated manifolds according to claim 1, characterized in that, The decoupling process in step 1) also introduces mutual information minimization to ensure that the two are semantically independent.
3. The three-dimensional human pose estimation method based on the alignment of two-dimensional and three-dimensional real-annotated dual manifolds according to claim 2, characterized in that, During the training process in step 2), the following constraints are used to ensure that the latent space can effectively represent the distribution of reasonable three-dimensional poses: 3D Reconstruction Loss : ,in The pose is reconstructed in 3D, output by a 3D pose decoder, and contains the 3D coordinates of N human joints. For the true 3D pose annotation, the dataset provides the true 3D pose, and... Consistent dimensions; The L2 norm is used to measure the overall distance between the reconstructed pose and the true pose, and to constrain the spatial position of the reconstructed pose and the true pose. Bone length consistency loss: for each bone connection The ratio of the predicted bone length to the actual bone length is close to 1, i.e. ,in In this context, E represents the set of edges connecting the human skeleton. Indicates the first The joint and the first A skeleton is made up of joints. The first one in the reconstructed posture The joint and the first The three-dimensional coordinates of each joint The first is the true posture. The joint and the first The three-dimensional coordinates of each joint To predict bone length, i.e., the first The joint and the first The distance of the reconstructed coordinates of each joint. The actual bone length, i.e., the first... The joint and the first The distance between the actual coordinates of each joint.
4. The three-dimensional human pose estimation method based on the alignment of two-dimensional and three-dimensional real-annotated dual manifolds according to claim 3, characterized in that, In the training process of step 3), a two-dimensional reconstruction loss is used to ensure that the latent space can effectively represent the distribution of reasonable two-dimensional poses. Two-dimensional reconstruction loss : ,in Represents the 2D reconstructed pose, output by the 2D pose decoder, containing the 2D coordinates of N human joints; Represents the true 2D labeled pose, which is the true 2D pose provided by the dataset, and... Consistent dimensions The L2 norm measures the distance between the reconstructed 2D pose and the true 2D pose, and constraining the 2D latent space can effectively characterize a reasonable 2D pose distribution.