Human body posture estimation method based on novel asymmetric gating fusion

Abstract

The invention discloses a three-dimensional human body posture estimation method and system based on an asymmetric double-current gating converter, and belongs to the technical field of computer vision and artificial intelligence. Aiming at the problems in the prior art, the invention provides a novel solution; the method comprises the following steps of: firstly, dynamically attenuating input characteristics by utilizing a confidence score output by an upstream 2D attitude detector through a 2D confidence gate so as to suppress the influence of noise and uncertainty; then, adopting an asymmetric double-flow architecture to process the spatial-temporal characteristics in parallel; and finally, combining the double-flow output features, predicting a final three-dimensional attitude through a regression head, and carrying out joint optimization by adopting a composite loss function. According to the method, the robustness of the model to noise input, the expression ability of spatial-temporal characteristics and the understanding ability of the model to complex actions are effectively improved, and higher estimation precision is achieved under the condition that post-processing is not needed.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, and in particular relates to a human pose estimation method. By adopting a novel asymmetric gating fusion method, more effective human pose estimation and detection results are obtained. Background Technology

[0002] In the field of computer vision, human pose estimation is a crucial task, widely applied in scenarios such as autonomous driving, robot navigation, augmented reality, and 3D reconstruction. The goal of human pose estimation is to identify and predict the three-dimensional coordinates of key anatomical joints in the human body from various sensor data such as images, videos, or depth maps. These key points together constitute the human kinematic skeleton, providing a structured three-dimensional representation foundation for subsequent human behavior analysis, action recognition, and intent understanding.

[0003] Mainstream methods for 3D human pose estimation can be broadly categorized into three types. The first type is based on monocular RGB image / video methods. These are inexpensive and widely used, primarily achieved through two approaches: one is direct end-to-end regression of 3D coordinates from the image, but this method faces challenges due to the inherent ambiguity of 2D to 3D conversion; the other is the more prevalent "two-stage" strategy, which first uses a mature 2D pose estimator to detect key points, and then maps them to 3D space through a "lifting network." However, this method is susceptible to 2D detection errors. The second type is based on multi-view image methods. These methods use multiple synchronized and calibrated cameras to accurately reconstruct 3D pose using geometric principles. They offer high accuracy but are complex to deploy, costly, and only usable in specific scenarios. The third type is based on RGB-D depth images. These methods directly acquire 3D geometric information using depth sensors, significantly reducing the estimation difficulty. However, they are limited by the sensor's effective range, performance in strong outdoor light, and sensitivity to specific object materials. Therefore, monocular pose estimation using images is the lowest-cost and most widely applicable method.

[0004] In recent years, Transformer models have achieved revolutionary success, initially in Natural Language Processing (NLP) and subsequently expanding to computer vision and other fields. It has become a popular model for achieving state-of-the-art results in numerous tasks such as image classification, object detection, and video recognition. The inherent long-range dependencies in skeletal joint sequences are particularly well-suited to the Transformer architecture. Previous research has detailed the feasibility of using Transformers as backbone networks with significant success. However, these models often lack prior physical knowledge of human anatomy. Therefore, they require large amounts of data to learn the connections between human skeletons and are prone to anatomically inaccurate poses. Furthermore, the computational redundancy and noise interference introduced by the global self-attention mechanism lead to computational inefficiency and may introduce noise that adversely affects overall prediction performance.

[0005] Introducing skeletal prior knowledge into 3D human pose estimation effectively addresses the inherent problem of inferring 3D structure from 2D images. This is because a single 2D projection can correspond to an infinite number of 3D spatial poses, leading to severe depth ambiguity and vagueness. Introducing skeletal prior knowledge imposes strong physical and geometric constraints on this vast solution space, ensuring the accuracy, reasonableness, and robustness of the prediction results. Graph structures have become the most natural and powerful tool for describing this type of structured knowledge. Specifically, the human skeleton is modeled as a graph structure, with joints as nodes and bones as edges. Existing researchers have applied Transformers to graph-based methods; while these methods improve accuracy, they also increase structural complexity and computational redundancy. Some researchers have proposed novel prior attention modules, deriving a new structure called "joint motion trajectory topology." Despite its innovation, the proposed temporal attention mechanism is insufficient for modeling non-smooth high-frequency motions, and the use of fixed time windows neither captures long-term dependencies nor limits adaptability, thus hindering the model's deep understanding of complex action sequences.

[0006] To address the aforementioned challenges, a novel Transformer architecture is proposed, innovatively employing a dual-stream temporal attention module. This module enables the model to more effectively capture global and local dependencies, thereby achieving efficient utilization of prior knowledge. The model architecture is based on a nested dual-stream fusion strategy. Externally, it fuses information from independent temporal and spatial streams; internally, it employs an innovative dual-stream temporal attention module. One stream in this module consists of a conventional temporal attention mechanism, while the other employs a temporal topological attention mechanism. The outputs of these two attention streams are integrated through a gating fusion mechanism, considering not only the spatial constraints of the human skeleton but also addressing the necessity of mutual guidance between spatiotemporal dimensions in parallel. More importantly, this architecture retains the Transformer's inherent ability to capture long-range dependencies and achieves accurate predictions by dissolving ambiguity through complementary information, thus enhancing the model's robustness and generalization ability. This process constructs richer and more discriminative spatiotemporal features. Summary of the Invention

[0007] To achieve the above objectives, the technical approach of this invention is as follows: First, confidence weighting is performed using 2D gating confidence. This module utilizes the confidence score *c* provided by the 2D detector as a "gating signal" to control the flow of information from each 2D joint to the subsequent 3D enhancement model. This component adds only a negligible number of parameters to the model, yet effectively filters unreliable or harmful inputs. This not only protects the model from noise interference but also encourages it to utilize internal knowledge and contextual cues for higher-level reasoning. Next, processing is performed through an external dual-stream structure, namely spatial and temporal streams. The spatial stream constructs a local spatial topology based on human anatomy. The temporal stream itself is also a dual-stream module: one branch is a temporal attention mechanism responsible for capturing long-range dependencies and processing the global skeleton sequence topology. The other branch is a trajectory prior attention mechanism, which constructs a local temporal topology. The collaborative modeling of these two temporal streams mutually promotes each other, thereby guiding the spatial stream to process data more efficiently. Finally, after a regression head, 2D to 3D human pose estimation and prediction are performed.

[0008] Step 1: Utilize a 2D confidence gate to address 2D input uncertainty based on the dynamic decay of the input characteristic according to reliability.

[0009] This invention provides a method for optimizing the input features of a 3D pose enhancement network using 2D keypoint detection confidence scores. This step aims to dynamically attenuate the initial features input to the 3D enhancement network based on the confidence scores output by the upstream 2D ​​pose detector through a 2D confidence gate mechanism. This explicitly models the uncertainty of the input data, effectively suppressing error propagation caused by 2D detection errors (e.g., those caused by occlusion, motion blur, etc.), significantly improving the robustness and accuracy of the final 3D human pose estimation results.

[0010] Specifically, this step is described in detail below:

[0011] al. Obtain a two-dimensional keypoint sequence containing confidence scores:

[0012] First, the human 2D pose detection results for one or more image frames are obtained from the upstream 2D ​​pose detector. For a sequence containing T frames, each with N keypoints, the obtained input data P... 2D This can be formally represented as:

[0013]

[0014] in, Let represent the two-dimensional coordinates of the j-th human body keypoint in the t-th frame, and It is the confidence score corresponding to the key point detection result, and its value ranges between [0, 1], which is used to quantify the reliability of the detection result.

[0015] a2. Initial feature mapping:

[0016] For each key point in the above sequence Using a predefined embedding function φ(·), its two-dimensional coordinates are mapped to a high-dimensional feature space R. D In this way, the initial feature vector is generated. This process can be represented as:

[0017]

[0018] This embedding function is typically implemented by one or more fully connected layers and is designed to transform sparse coordinate information into richer feature representations that are more conducive to network learning.

[0019] a3. Perform confidence gating operation:

[0020] To incorporate the confidence information of key points into the features, this step introduces a core 2D confidence gating operation. This operation transforms the initial feature vector generated in the previous step... This is then combined with its corresponding confidence score. Specifically, the scalar confidence score is first combined using a gating function g(·). This is transformed into a modulated signal. Subsequently, this modulated signal is compared with the initial feature vector. Perform element-wise Hadamard product (⊙) to generate a gated feature vector modulated by confidence level. The calculation process is as follows:

[0021]

[0022] In this operation, the gating function g(c) can be designed in various forms, such as directly using the confidence value (g(c) = c) or through a small neural network, such as one or more fully connected layers with activation functions, performing a nonlinear transformation. The function maps the confidence score to an effective attenuation coefficient. Through a broadcast mechanism, the modulated signal is extended to match the feature vector. The same dimension D is used to achieve weighted decay for each element in the feature vector.

[0023] a4. Output gating characteristics:

[0024] The gated feature vector generated after confidence gating operation This is the final output of this step. This vector carries key point information adjusted for uncertainty: for high confidence levels... Key points approaching 1 retain more of their original feature information; while for points with low confidence... Key points that approach 0 have significantly reduced feature information.

[0025] Through the above processing, the obtained gated feature sequence will be used as a more reliable input and fed into the subsequent dual-stream spatiotemporal information processing network for further attitude estimation.

[0026] Step 2: Employ an asymmetric two-stream architecture to extract motion and structural information in parallel, resolving the decoupling of spatiotemporal features.

[0027] This step receives the confidence-modulated gated feature sequence generated in step one and performs parallel spatiotemporal feature extraction and modeling on it using an asymmetric dual-stream architecture. This architecture consists of a temporal stream processing module and a spatial stream processing module, designed to capture the temporal dynamics of human motion and the spatial topology of body joints, respectively.

[0028] Specifically, this step is described in detail below:

[0029] b1. Temporal Stream Path Processing:

[0030] In the time-stream path, a nested dual-stream structure and a gated fusion mechanism are employed to process features, aiming to deeply mine and model the temporal dynamics of the input sequence, ultimately generating time-stream output features. This path includes the following operations:

[0031] (a) Global long-range dependency modeling: This part uses the standard temporal multi-head self-attention mechanism to capture the global dependencies between all frames in the sequence.

[0032] Input and Projection: For the input time series features X T First, through a learnable weight matrix Linearly project it onto three different spaces: Query, Key, and Value.

[0033] Attention calculation: The output of each head is calculated using a scaled dot product attention mechanism, and the calculation formula is as follows:

[0034]

[0035] Output integration: The outputs of all attention heads are concatenated and passed through a final linear projection layer to obtain the output of this branch, namely the global temporal context feature X. B .

[0036] (b) Multi-scale and trajectory prior dependency modeling: To simultaneously capture short-, medium-, and long-term motion dependencies, a multi-scale time graph edge set is first constructed based on a predefined set of time dilation rates, such as s∈{1, 2, 4}. A sparse original adjacency matrix A is then generated based on this edge set. raw After undergoing operations such as symmetrization, degree matrix normalization, and the addition of self-loops, a multi-scale time adjacency matrix A is finally generated. ms This sparse connection method effectively reduces computational complexity while injecting prior knowledge.

[0037] (c) Gated fusion: The output features X of the two parallel branches mentioned above are combined. A and X B Adaptive fusion is performed. A gated signal is generated through a gated network, and the two features are weighted and summed to obtain the fused temporal feature Y. fused The calculation formula is as follows:

[0038] Y fused =G⊙X A +(1-G)⊙X B

[0039] b2. Spatial Stream Path Processing:

[0040] Simultaneously with the temporal flow path processing, the spatial flow path employs a second-order adjacency matrix-enhanced graph convolutional attention (SAMGCA) mechanism to process the same gated feature sequence, aiming to explicitly model the physical topology of the human skeleton and ultimately generate the spatial flow output feature PS. This path includes the following operations:

[0041] (a) Enhanced kinematic topology matrix construction: First, an enhanced kinematic topology matrix is ​​constructed by fusing three different topology matrices. These three types of topological matrices include: first-order local topology based on the physical connections of the human skeleton. Second-order extended topology for capturing indirect joint dependencies And a learnable global simulation topology for capturing the implicit associations of arbitrary joint pairs.

[0042] (b) Enhanced Graph Convolution and Spatial Feature Extraction: The constructed enhanced kinematic topology matrix is ​​used to perform graph convolution on the input pose features, injecting rich prior information about human structure into the feature representation to generate intermediate features. Subsequently, these intermediate features are input into a spatial multi-head self-attention (MHSA) module, and combined with operations such as residual connections, further global spatial relation extraction is performed to finally generate output features of spatial flow.

[0043] b3. Output:

[0044] This step ultimately outputs two parallel feature streams: a temporal stream feature that deeply integrates temporal dynamic information and a spatial stream feature that contains rich spatial structure knowledge. These two feature streams will then be merged and fed into the subsequent Regression Head for final 3D pose coordinate prediction.

[0045] Step 3: Employ a composite loss to simultaneously constrain the accuracy, smoothness, and rationality of the pose, thus solving the multi-objective joint optimization problem.

[0046] This step aims to fuse the spatiotemporal features obtained from the parallel processing in step two and map them into a final 3D human pose sequence through a regression head. Simultaneously, this step defines a composite loss function to optimize the entire estimation process end-to-end during network training, ensuring the geometric accuracy, temporal continuity, and structural rationality of the final output.

[0047] cl. Feature flow merging and regression prediction:

[0048] (a) Feature merging: Merging the time stream features Y output by the time stream processing module in step two. fused Spatial flow characteristics P output by the spatial flow processing module S The merging process is then performed. This merging operation can be achieved through feature concatenation or other forms of fusion strategies, aiming to integrate complementary spatiotemporal information to generate a comprehensive and discriminative final feature representation.

[0049] (b) Regression Prediction: The merged final feature representation is input into a regression head. This regression head typically consists of one or more linear layers, and its function is to map the high-dimensional feature vectors to the target 3D coordinate space, thereby predicting the final 3D pose sequence. This sequence is the final output of the method of the present invention.

[0050] c2. Definition of composite loss function and joint optimization:

[0051] To effectively guide the learning of network parameters during training, this step employs a composite loss function L, consisting of multiple sub-objectives, to jointly optimize the model. This loss function ensures that the predictions remain consistent with reality across multiple dimensions.

[0052] (a) Loss of positional accuracy This part is the core geometric accuracy target, which uses the Mean Per Joint Position Error (MPJPE) to calculate the average Euclidean distance between the predicted 3D joint coordinates and the true ground-truth coordinates.

[0053] (b) Time consistency loss This section aims to ensure that the output motion sequence is smooth and continuous, due to joint velocity errors. and joint acceleration error Weighted composition is used to penalize jitter or abrupt changes that are inconsistent in time.

[0054] (c) Biomechanical structural loss This section applies prior knowledge of human anatomy to the predicted results to ensure they conform to physical laws. It integrates information on bone length... Joint angle and body symmetry Constraints.

[0055] Overall Loss Function: The final overall loss function is obtained by weighted summation of the above items, and its calculation formula is as follows:

[0056]

[0057] Where, λtemp and λ struct This is a hyperparameter used to balance the contributions of time consistency and structural rationality constraints to the overall optimization objective.

[0058] Through this step, the present invention not only completes the mapping from two-dimensional key points to three-dimensional pose, but also ensures that the generated 3D human skeleton sequence achieves a high level in multiple aspects such as geometric position, motion smoothness, and biological structural rationality through a carefully designed joint optimization objective.

[0059] Compared with existing structures, the present invention has the following advantages:

[0060] 1. Innovative Novel Asymmetric Two-Stream Architecture Enhances the Complementarity and Fusion of Spatiotemporal Features: This invention proposes an asymmetric two-stream network architecture. This architecture focuses on processing the spatial structural priors of the human skeleton and the temporal dynamic information of motion through parallel spatial and temporal streams, respectively. An external fusion mechanism allows the complementary information of the two streams to eliminate ambiguity, constructing richer and more discriminative spatiotemporal features, thereby improving the robustness and generalization ability of the model.

[0061] 2. A unique nested gating fusion strategy enables more refined temporal information modeling: The temporal flow module of this invention is itself a nested dual-flow structure. It includes a regular temporal attention flow for capturing global long-range dependencies, and an attention flow that incorporates trajectory priors and focuses on local temporal topology. A gating fusion mechanism, Gated Fusion, adaptively combines these two flows, enabling the model to more effectively coordinate global and local temporal dependencies, thus achieving a deeper understanding of complex action sequences.

[0062] 3. Introducing a 2D Confidence Gate Mechanism Significantly Improves Robustness to Noise Input: This invention applies a 2D Confidence Gate mechanism at the network input. This mechanism uses the confidence score provided by the upstream 2D ​​detector as a gating signal to dynamically attenuate input features of low-confidence 2D keypoints, which are often caused by occlusion or motion blur. This explicitly models the uncertainty of the input data, effectively suppresses the propagation of 2D detection errors, protects the model from noise interference with almost no increase in parameters, and improves the final accuracy and robustness.

[0063] 4. Enhanced Spatial Topology Modeling for More Accurate Representation of Complex Human Structures: In spatial flow, this invention extends traditional graph convolution-based methods by introducing a second-order adjacency matrix. This method integrates first-order direct connections, second-order indirect connections, and a learnable global adjacency matrix to construct an enhanced kinematic topology. This is equivalent to expanding the receptive field of graph convolution, enabling the model to capture long-range spatial dependencies across joints, thereby more accurately modeling complex human spatial structures and generating poses that are more anatomically consistent.

[0064] 5. Achieves state-of-the-art performance without complex post-processing: Experimental results show that this invention outperforms existing state-of-the-art methods on two major benchmark datasets, Human3.6M and MPI-INF-3DHP. In particular, this invention achieves state-of-the-art performance without using complex post-processing techniques such as diffusion models, demonstrating that the proposed network architecture itself possesses a deeper understanding of complex human dynamics and superior modeling capabilities. Attached Figure Description

[0065] Figure 1 This is a flowchart of the method of the present invention.

[0066] Figure 2 This is the network architecture diagram of the present invention.

[0067] Figure 3 This is the spatial attention module of the present invention.

[0068] Figure 4 This is the temporal attention module of the present invention.

[0069] Figure 5 This is a performance comparison of the Human 3.6m characteristic benchmark of the present invention.

[0070] Figure 6 This is a performance comparison of the Human 3.6m characteristic benchmark of the present invention.

[0071] Figure 7 This is a performance comparison of the Human 3.6m characteristic benchmark of the present invention.

[0072] Figure 8 This is a performance comparison of the Human 3.6m characteristic benchmark of the present invention.

[0073] Figure 9 This is a performance comparison of the MPI-INF-3DHP feature benchmark of the present invention.

[0074] Figure 10 This is the ablation experiment of the present invention. Detailed Implementation

[0075] To make the above and other objects, features and advantages of the present invention more apparent, specific embodiments of the present invention are described below in detail with reference to the accompanying drawings:

[0076] Figure 1 The specific flowchart of this method can be divided into three steps:

[0077] Step 1: Utilize a 2D confidence gate and dynamically decay the input characteristics based on reliability to address 2D input uncertainty.

[0078] This invention provides a method for optimizing 3D pose enhancement network input features using 2D keypoint detection confidence. The network architecture diagram is shown below. Figure 2 As shown, this step aims to dynamically attenuate the initial features input to the 3D boosting network using a 2D confidence gate mechanism, based on the confidence scores output by the upstream 2D ​​pose detector. This explicitly models the uncertainty of the input data, effectively suppressing error propagation caused by 2D detection errors (such as those caused by occlusion, motion blur, etc.), and significantly improving the robustness and accuracy of the final 3D human pose estimation results.

[0079] Specifically, this step is described in detail below:

[0080] 1. Obtain a two-dimensional keypoint sequence containing confidence scores:

[0081] First, the human 2D pose detection results for one or more image frames are obtained from the upstream 2D ​​pose detector. For a sequence containing T frames, each with N keypoints, the obtained input data P... 2D This can be formally represented as:

[0082]

[0083] in, Let represent the two-dimensional coordinates of the j-th human body keypoint in the t-th frame, and It is the confidence score corresponding to the key point detection result, and its value ranges between [0, 1], which is used to quantify the reliability of the detection result.

[0084] 2. Initial feature mapping:

[0085] For each key point in the above sequence Using a predefined embedding function φ(·), its two-dimensional coordinates are mapped to a high-dimensional feature space R. D In this way, the initial feature vector is generated. This process can be represented as:

[0086]

[0087] This embedding function is typically implemented by one or more fully connected layers and is designed to transform sparse coordinate information into richer feature representations that are more conducive to network learning.

[0088] 3. Perform confidence gating operation:

[0089] To incorporate the confidence information of key points into the features, this step introduces a core 2D confidence gating operation. This operation transforms the initial feature vector generated in the previous step... This is then combined with its corresponding confidence score. Specifically, the scalar confidence score is first combined using a gating function g(·). This is transformed into a modulated signal. Subsequently, this modulated signal is compared with the initial feature vector. Perform element-wise Hadamard product (⊙) to generate a gated feature vector modulated by confidence level. The calculation process is as follows:

[0090]

[0091] In this operation, the gating function g(c) can be designed in various forms, such as directly using the confidence value (g(c) = c) or through a small neural network, such as one or more fully connected layers with activation functions, performing a nonlinear transformation. The function maps the confidence score to an effective attenuation coefficient. Through a broadcast mechanism, the modulated signal is extended to match the feature vector. The same dimension D is used to achieve weighted decay for each element in the feature vector.

[0092] 4. Output gating characteristics:

[0093] The gated feature vector generated after confidence gating operation This is the final output of this step. This vector carries key point information adjusted for uncertainty: for high confidence levels... Key points approaching 1 retain more of their original feature information; while for points with low confidence... Key points that approach 0 have significantly reduced feature information.

[0094] Through the above processing, the obtained gated feature sequence will be used as a more reliable input and fed into the subsequent dual-stream spatiotemporal information processing network for further attitude estimation.

[0095] Step 2: Employ an asymmetric two-stream architecture to extract motion and structural information in parallel, resolving the decoupling of spatiotemporal features.

[0096] This step receives the confidence-modulated gated feature sequence generated in step one and performs parallel spatiotemporal feature extraction and modeling on it using an asymmetric dual-stream architecture. This architecture consists of a temporal stream processing module and a spatial stream processing module, designed to capture the temporal dynamics of human motion and the spatial topology of body joints, respectively.

[0097] Specifically, this step is described in detail below:

[0098] 1. Temporal Stream Path Processing:

[0099] Structure as Figure 3 As shown, in the time-stream path, a nested dual-stream structure and a gated fusion mechanism are used to process features, aiming to deeply mine and model the temporal dynamics of the input sequence, and finally generate time-stream output features. This path includes the following operations:

[0100] (a) Global long-range dependency modeling: This part uses the standard temporal multi-head self-attention mechanism to capture the global dependencies between all frames in the sequence.

[0101] Input and Projection: For the input time series features X T First, through a learnable weight matrix Linearly project it onto three different spaces: Query, Key, and Value.

[0102] Attention calculation: The output of each head is calculated using a scaled dot product attention mechanism, and the calculation formula is as follows:

[0103]

[0104] Output integration: The outputs of all attention heads are concatenated and passed through a final linear projection layer to obtain the output of this branch, namely the global temporal context feature X. B .

[0105] (b) Multi-scale and trajectory prior dependency modeling: To simultaneously capture short-, medium-, and long-term motion dependencies, a multi-scale time graph edge set is first constructed based on a predefined set of time dilation rates, such as s∈{1, 2, 4}. A sparse original adjacency matrix A is then generated based on this edge set. raw After undergoing operations such as symmetrization, degree matrix normalization, and the addition of self-loops, a multi-scale time adjacency matrix A is finally generated. ms This sparse connection method effectively reduces computational complexity while injecting prior knowledge.

[0106] (c) Gated fusion: The output features X of the two parallel branches mentioned above are combined. A and X B Adaptive fusion is performed. A gated signal is generated through a gated network, and the two features are weighted and summed to obtain the fused temporal feature Y. fused The calculation formula is as follows:

[0107] Y fused =G⊙X A +(1-G)⊙X B

[0108] 2. Spatial Stream Path Processing:

[0109] Structure as Figure 4 As shown, while processing the temporal flow path, the spatial flow path employs a graph convolutional attention mechanism enhanced by a second-order adjacency matrix to process the same gated feature sequence. This aims to explicitly model the physical topology of the human skeleton, ultimately generating the spatial flow output feature PS. This path includes the following operations:

[0110] (a) Enhanced kinematic topology matrix construction: First, an enhanced kinematic topology matrix is ​​constructed by fusing three different topology matrices. These three types of topological matrices include: first-order local topology based on the physical connections of the human skeleton. Second-order extended topology for capturing indirect joint dependencies And a learnable global simulation topology for capturing the implicit associations of arbitrary joint pairs.

[0111] (b) Enhanced Graph Convolution and Spatial Feature Extraction: The constructed enhanced kinematic topology matrix is ​​used to perform graph convolution on the input pose features, injecting rich prior information about human structure into the feature representation to generate intermediate features. Subsequently, these intermediate features are input into a spatial multi-head self-attention (MHSA) module, and combined with operations such as residual connections, further global spatial relation extraction is performed to finally generate output features of spatial flow.

[0112] 3. Output:

[0113] This step ultimately outputs two parallel feature streams: a temporal stream feature that deeply integrates temporal dynamic information and a spatial stream feature that contains rich spatial structure knowledge. These two feature streams will then be merged and fed into the subsequent Regression Head for final 3D pose coordinate prediction.

[0114] Step 3: Employ a composite loss to simultaneously constrain the accuracy, smoothness, and rationality of the pose, thus solving the multi-objective joint optimization problem.

[0115] This step aims to fuse the spatiotemporal features obtained from the parallel processing in step two and map them into a final 3D human pose sequence through a regression head. Simultaneously, this step defines a composite loss function to optimize the entire estimation process end-to-end during network training, ensuring the geometric accuracy, temporal continuity, and structural rationality of the final output.

[0116] 1. Feature Flow Merging and Regression Prediction:

[0117] (a) Feature merging: Merging the time stream features Y output by the time stream processing module in step two. fused Spatial flow characteristics P output by the spatial flow processing module S The merging process is then performed. This merging operation can be achieved through feature concatenation or other forms of fusion strategies, aiming to integrate complementary spatiotemporal information to generate a comprehensive and discriminative final feature representation.

[0118] (b) Regression Prediction: The merged final feature representation is input into a regression head. This regression head typically consists of one or more linear layers, and its function is to map the high-dimensional feature vectors to the target 3D coordinate space, thereby predicting the final 3D pose sequence. This sequence is the final output of the method of the present invention.

[0119] 2. Definition of composite loss function and joint optimization:

[0120] To effectively guide the learning of network parameters during training, this step employs a composite loss function L, consisting of multiple sub-objectives, to jointly optimize the model. This loss function ensures that the predictions remain consistent with reality across multiple dimensions.

[0121] (a) Position accuracy loss L pos This part is the core geometric accuracy target, which uses the Mean Per Joint Position Error (MPJPE) to calculate the average Euclidean distance between the predicted 3D joint coordinates and the true ground-truth coordinates.

[0122] (b) Time consistency loss L temp This section aims to ensure that the output motion sequence is smooth and continuous, determined by the joint velocity error L. vel and joint acceleration error L acc Weighted composition is used to penalize jitter or abrupt changes that are inconsistent in time.

[0123] (c) Biomechanical structural loss Lstruct: This part applies prior knowledge of human anatomy to the predicted results to ensure they conform to physical laws. It integrates the data on bone length L... bone Joint angle L angele and body symmetry L sym Constraints.

[0124] Overall Loss Function: The final overall loss function is obtained by weighted summation of the above items, and its calculation formula is as follows:

[0125] L = L pos +λ temp L temp +λ struct L struct

[0126] Where, λ temp and λ struct This is a hyperparameter used to balance the contributions of time consistency and structural rationality constraints to the overall optimization objective.

[0127] Through this step, the present invention not only completes the mapping from two-dimensional key points to three-dimensional pose, but also ensures that the generated 3D human skeleton sequence achieves a high level in multiple aspects such as geometric position, motion smoothness, and biological structural rationality through a carefully designed joint optimization objective.

[0128] The invention will be further described below with reference to simulation examples.

[0129] Simulation example:

[0130] Our model is implemented using the PyTorch framework and trained on an NVIDIA GeForce RTX 4090 GPU. During training, we employ random flipping and test-time augmentation strategies. The learning rate is managed via cosine annealing scheduling. The input 2D keypoints are derived from a Stacked Hourglass 2D pose detector. The encoder has a depth of 5, with 8 attention heads and a feature dimension of 512. We train the model using the Adam optimizer with an initial learning rate of 1e. - 4. The decay factor after each epoch is 0.99. For the Human3.6M dataset, we use 243 frames as training input, while for the MPI-INF-3DHP dataset, we use 81 frames as input.

[0131] To verify the effectiveness of this invention, we selected two datasets, Human3.6M and MPI-INF-3DHP, for testing. For Human3.6M, we used two error metrics: Mean Per Joint Position Error (MPJPE), which is the average Euclidean distance between the predicted pose and the true pose after root joint alignment; and Procrustes Analysis MPJPE, P-MPJPE, which is the MPJPE calculated after aligning the predicted pose with the true pose through rigid transformation using Procrustes analysis. The results are as follows: Figure 5 , Figure 6 , Figure 7 and Figure 8 As shown.

[0132] For MPI-INF-3DHP, we follow the standard protocol and use noiseless 2D poses as input. We evaluate performance using three metrics: MPJPE, Percentage of Correct Keypoints (with a tolerance threshold of 150mm), PCK, and Area Under the Curve (AUC) to comprehensively measure model performance. The results are as follows: Figure 9 As shown.

[0133] To verify the advancement of the proposed module, we conducted several independent ablation experiments to verify the importance of experimental parameters and components.

[0134] All ablation experiments were performed on the Human3.6M dataset. Each independent ablation experiment changed only one parameter or isolated a specific component, while keeping other parameters unchanged, to test the impact of a single parameter or component on the final result. Below, we will describe each ablation experiment in detail, and the results are as follows: Figure 10 As shown.

[0135] Starting with the baseline model, we progressively added our proposed modules and evaluated their performance using MPJPE as the evaluation metric, with lower values ​​being better. The overall results clearly demonstrate that all our proposed components make positive and quantifiable contributions to model performance. With the gradual integration of each module, the model error continuously decreases, and the final complete model achieves the lowest error across both metrics. This fully demonstrates the effectiveness and advanced design of our overall architecture.

[0136] In summary, compared with existing methods, the proposed method innovatively integrates a dual-stream structure into a meticulously designed temporal framework. Its temporal structure, by fusing traditional temporal attention mechanisms with prior-injected attention mechanisms, achieves progressive optimization of the 3D human skeleton sequence through multi-stage processing. Comprehensive evaluations on Human3.6M and MPI-INF-3DHP benchmarks demonstrate a deeper understanding and superior modeling capabilities for complex human dynamics compared to existing methods. Particularly noteworthy is its ability to effectively address common challenges in real-world scenarios by recognizing complex human movements. These breakthroughs establish it as a benchmark framework in the field of 3D human pose estimation.

Claims

1. Depth estimation based on a novel iterative refinement module includes the following steps: Step 1: Utilize a 2D confidence gate to address 2D input uncertainty based on the dynamic decay of the input characteristic according to reliability. This invention provides a method for optimizing 3D pose enhancement network input features using 2D keypoint detection confidence. This step aims to dynamically drive the decay of initial features input to the 3D boosting network using a 2D confidence gate mechanism, based on the confidence scores output by the upstream 2D ​​pose detector. This explicitly models the uncertainty of the input data, thereby effectively suppressing error propagation caused by 2D detection errors such as occlusion and motion blur, and significantly improving the robustness and accuracy of the final 3D human pose estimation results. Specifically, this step is described in detail below: a1. Obtain a two-dimensional keypoint sequence containing confidence scores: First, the human 2D pose detection results for one or more image frames are obtained from the upstream 2D ​​pose detector; for a sequence containing T frames, each with N keypoints, the obtained input data F 2D This can be formally represented as: in, Let represent the two-dimensional coordinates of the j-th human body keypoint in the t-th frame, and It is the confidence score corresponding to the key point detection result, and its value ranges between [0, 1], which is used to quantify the reliability of the detection result; a2. Initial feature mapping: For each key point in the above sequence Using a predefined embedding function φ(·), its two-dimensional coordinates are mapped to a high-dimensional feature space R. D In this way, the initial feature vector is generated. This process can be represented as: This embedding function is typically implemented by one or more fully connected layers, and is designed to transform sparse coordinate information into richer feature representations that are more conducive to network learning. a3. Perform confidence gating operation: To incorporate the confidence information of key points into the features, this step introduces a core 2D confidence gating operation; this operation will transform the initial feature vector generated in the previous step... This is combined with its corresponding confidence score; specifically, the scalar confidence score is first combined using a gating function g(·). The signal is transformed into a modulated signal; subsequently, this modulated signal is compared with the initial feature vector. Perform element-wise Hadamard product (⊙) to generate a gated feature vector modulated by confidence level. The calculation process is as follows: In this operation, the gating function g(c) can be designed in various forms, such as directly using the confidence value (g(c) = c) or through a small neural network, such as one or more fully connected layers with activation functions, to perform a nonlinear transformation. The function maps the confidence score to an effective attenuation coefficient. Through a broadcast mechanism, the modulated signal is extended to the feature vector. The same dimension D is used to achieve weighted decay for each element in the feature vector; a4. Output gating characteristics: The gated feature vector generated after confidence gating operation This is the final output of this step; the vector carries key point information adjusted for uncertainty: for high confidence levels... Key points approaching 1 retain more of their original feature information; while for points with low confidence... The key point is that it approaches 0. Its feature information is significantly attenuated; Through the above processing, the obtained gated feature sequence will be used as a more reliable input and fed into the subsequent dual-stream spatiotemporal information processing network for further attitude estimation. Step 2: Adopt an asymmetric dual-stream architecture to extract motion and structural information in parallel, thus solving the problem of spatiotemporal feature decoupling; This step receives the gated feature sequence generated in step one, which has been modulated with confidence, and performs parallel spatiotemporal feature extraction and modeling on it through an asymmetric dual-stream architecture. This architecture consists of a temporal stream processing module and a spatial stream processing module, which are designed to capture the temporal dynamic information of human motion and the spatial topology of body joints, respectively. Specifically, this step is described in detail below: b1. Temporal Stream Path Processing: In the time-stream path, a nested dual-stream structure and a gated fusion mechanism are used to process features, aiming to deeply mine and model the temporal dynamics of the input sequence, and finally generate time-stream output features; this path includes the following operations: (a) Global long-range dependency modeling: This part adopts the standard temporal multi-head self-attention mechanism to capture the global dependencies between all frames in the sequence; Input and Projection: For the input time series features X T First, through a learnable weight matrix Linearly project it onto three different spaces: Query, Key, and Value; Attention calculation: The output of each head is calculated using a scaled dot product attention mechanism, and the calculation formula is as follows: Output integration: The outputs of all attention heads are concatenated and passed through a final linear projection layer to obtain the output of this branch, namely the global temporal context feature X. B ; (b) Multi-scale and trajectory prior dependency modeling: To simultaneously capture short-, medium-, and long-term motion dependencies, a multi-scale time graph edge set is first constructed based on a preset time dilation rate set, such as s∈{1, 2, 4}; a sparse original adjacency matrix A is then generated based on this edge set. raw After undergoing operations such as symmetrization, degree matrix normalization, and the addition of self-loops, a multi-scale time adjacency matrix A is finally generated. ms ; This sparse connection method effectively reduces computational complexity while injecting prior knowledge; (c) Gated fusion: The output features X of the two parallel branches mentioned above are combined. A and X B Adaptive fusion is performed; a gated signal is generated through a gated network, and the two features are weighted and summed to obtain the fused temporal flow feature Y. fused The calculation formula is as follows: Y fused =G⊙X A +(1-G)⊙X B b2. Spatial Stream Path Processing: While processing the temporal flow path, the spatial flow path employs a graph convolutional attention mechanism enhanced by a second-order adjacency matrix to process the same gated feature sequence, aiming to explicitly model the physical topology of the human skeleton and ultimately generate the spatial flow output feature PS. This path contains the following operations: (a) Enhanced kinematic topology matrix construction: First, an enhanced kinematic topology matrix is ​​constructed by fusing three different topology matrices. These three types of topological matrices include: first-order local topology based on the physical connections of the human skeleton. Second-order extended topology for capturing indirect joint dependencies And a learnable global simulation topology for capturing the implicit associations of arbitrary joint pairs. (b) Enhanced graph convolution and spatial feature extraction: The constructed enhanced kinematic topology matrix is ​​used to perform graph convolution on the input pose features, injecting rich prior information of human structure into the feature representation to generate intermediate features; then, the intermediate features are input into a spatial multi-head self-attention module, and combined with residual connection and other operations to further extract global spatial relationships, and finally generate output features of spatial flow. b3. Output: This step ultimately outputs two parallel feature streams: a temporal stream feature that deeply integrates temporal dynamic information and a spatial stream feature that contains rich spatial structure knowledge; these two feature streams will then be merged and fed into the subsequent Regression Head for final 3D pose coordinate prediction. Step 3: Employ composite loss to simultaneously constrain the accuracy, smoothness, and rationality of the pose, thereby solving the multi-objective joint optimization problem; This step aims to fuse the spatiotemporal features obtained from the parallel processing in step two and map them into the final 3D human pose sequence through a regression head. At the same time, this step defines a composite loss function to optimize the entire estimation process end-to-end during the network training phase to ensure the geometric accuracy, temporal continuity and structural rationality of the final output. c1. Feature Flow Merging and Regression Prediction: (a) Feature merging: Merging the time stream features Y output by the time stream processing module in step two. fused Spatial flow characteristics P output by the spatial flow processing module S The merging process can be achieved through feature concatenation or other fusion strategies, aiming to integrate complementary spatiotemporal information to generate a comprehensive and discriminative final feature representation. (b) Regression Prediction: The merged final feature representation is input into a regression head; this regression head typically consists of one or more linear layers, and its function is to map the high-dimensional feature vectors to the target 3D coordinate space, thereby predicting the final 3D pose sequence. This sequence is the final output of the method of the present invention; c2. Definition of composite loss function and joint optimization: In order to effectively guide the learning of network parameters during training, this step uses a composite loss function L composed of multiple sub-objectives to jointly optimize the model; this loss function ensures that the prediction results are consistent with the real situation in multiple dimensions. (a) Position accuracy loss L pos This part is the core geometric accuracy target, which uses Mean Per Joint Position Error (MPJPE) to calculate the average Euclidean distance between the predicted 3D joint coordinates and the true ground-truth coordinates. (b) Time consistency loss L temp This section aims to ensure that the output motion sequence is smooth and continuous, determined by the joint velocity error L. vel and joint acceleration error L acc Weighted composition is used to penalize jitter or abrupt changes that are inconsistent in time; (c) Biomechanical structural loss Lstruct: This part applies prior knowledge of human anatomy to the prediction results to ensure they conform to physical laws; it integrates the information on bone length L bone Joint angle L angele and body symmetry L sym Constraints; Overall Loss Function: The final overall loss function is obtained by weighted summation of the above values, and its calculation formula is as follows: L=L pos +λ temp L temp +λ struct L struct Where, λ temp and λ struct This is a hyperparameter used to balance the contributions of time consistency and structural rationality constraints to the overall optimization objective; Through this step, the present invention not only completes the mapping from two-dimensional key points to three-dimensional pose, but also ensures that the generated 3D human skeleton sequence achieves a high level in multiple aspects such as geometric position, motion smoothness, and biological structural rationality through a carefully designed joint optimization objective.

Images