A three-dimensional human pose estimation method based on dynamic spatio-temporal topology dependency

Abstract

The application discloses a three-dimensional human body posture estimation method based on dynamic space-time topology dependence, comprising the following steps: obtaining a two-dimensional joint node sequence in a continuous video frame, generating a two-dimensional depth map and a joint depth value through a diffusion model, and constructing a relative depth indication matrix; mapping the two-dimensional coordinates, depth indications and position encodings to a high-dimensional space through feature embedding to obtain three-dimensional posture features; constructing a parallel global and local branch model, using a serial Transformer and Mamba module to extract global structure features in the global branch, using a serial space graph convolution layer and Mamba module to extract local structure features in the local branch, and introducing space-time topology constraints to guide the local branch according to physical laws; adaptively fusing the global and local features based on kinematic confidence; and finally, regressing three-dimensional posture coordinates through the fused features to obtain the final estimation result. The application improves the accuracy and space-time consistency of three-dimensional human body posture estimation.

Description

Technical Field

[0001] This invention relates to the field of human pose estimation technology, and in particular to a three-dimensional human pose estimation method based on dynamic spatiotemporal topological dependence. Background Technology

[0002] 3D human pose estimation (3DHPE) aims to accurately recover the 3D joint positions of the human body from 2D images or videos, and has wide applications in motion capture, virtual reality, and human-computer interaction. The human body's topology reflects the spatial and kinematic relationships between joints and is core prior knowledge for accurate prediction. For 3D human pose estimation of video sequences, the most similar existing methods can be mainly divided into the following three categories:

[0003] 1. A global spatiotemporal modeling method based on Transformer lacks topological priors: the self-attention mechanism mainly captures global dependencies, ignoring the prior knowledge of the natural topological structure of the human skeleton, lacking local structural constraints and corrections for the generated pose results, and is prone to producing abnormal poses that do not conform to the laws of human anatomy.

[0004] Extremely high computational complexity: When processing long video sequences, the computational complexity of the self-attention mechanism increases quadratically, leading to a sharp increase in network complexity, huge computational overhead and memory consumption, making it difficult to meet the needs of real-time applications.

[0005] 2. The local topology modeling method based on graph convolutional networks (GCN) suffers from the problem of over-smoothing: In order to obtain a larger receptive field, GCN usually needs to increase the number of network layers. However, as the number of layers increases, the features of each joint become more and more similar through continuous aggregation, resulting in the phenomenon of "over-smoothing". This erases the diversity of the human body topology itself and the details of local motion, resulting in larger prediction errors for flexible joints such as the extremities.

[0006] 3. Sequence modeling methods based on the traditional state-space model (Mamba) have insufficient spatiotemporal and topological coordination capabilities: Existing Mamba applications have not fully explored its potential in 3D human pose estimation, especially when dealing with complex topological dependencies of the human body, particularly under occlusion or complex movements, the rationality of local structures cannot be effectively guaranteed. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a three-dimensional human pose estimation method based on dynamic spatiotemporal topological dependence, which solves the technical problems in existing technologies such as the lack of local structure modeling ability of self-attention mechanisms, the tendency of GCN to lead to excessive feature smoothing, and the lack of adaptive perception of high and low frequency kinematic changes by the time step parameters of standard Mamba.

[0008] This invention provides a three-dimensional human pose estimation method based on dynamic spatiotemporal topological dependence, characterized by the following steps:

[0009] Step 1: Obtain the two-dimensional joint video frame sequence in the continuous video frames, and obtain the two-dimensional depth map of the two-dimensional joint video frame sequence and the depth value of the joint position through the diffusion model to form a relative depth indication matrix.

[0010] Step 2: Map the 2D joint coordinates, relative depth indication matrix, and position encoding to a high-dimensional feature space through a feature embedding layer to obtain 3D pose features;

[0011] Step 3: Construct a model including global branches, local branches, and spatiotemporal topological constraints to obtain global and local structural features. 3D pose features are used as input features for the parallel global and local branches. The global branch includes two parallel branches: one consisting of a serial Transformer module and a Mamba module, and the other consisting of a serial Mamba module and a Transformer module. The global branch is used to extract global structural features. The local branch includes a serial Mamba module and a spatial graph convolutional layer to extract local structural features. Spatiotemporal topological constraints provide strong physical guidance for the feature extraction of the local branches.

[0012] Step 4: Adaptive feature fusion of global and local structural features based on kinematic confidence;

[0013] Step 5: Combine the fused features to perform 3D pose coordinate regression and obtain the 3D human pose estimation results.

[0014] Furthermore, in step 1, the specific method for obtaining the two-dimensional depth map of the two-dimensional joint video frame sequence and the depth value of the joint position using the diffusion model is as follows:

[0015] Two-dimensional depth maps are extracted from the RGB original frames of a two-dimensional joint video frame sequence using a diffusion model, and the depth values ​​at the corresponding joint positions are sampled from these maps based on the two-dimensional coordinates. The specific formula is as follows:

[0016] ;

[0017] ;

[0018] In the formula, It is a two-dimensional depth map; For the diffusion model; I is the RGB original frame of the 2D joint video frame; d is the depth value of the joint position; u and v are the coordinate values ​​of the 2D coordinates; Sample(⋅) is the spatial sampling operation;

[0019] The specific formula for the resulting relative depth indication matrix is:

[0020] ;

[0021] In the formula, y is the relative depth indicator matrix; sgn( ) is a symbolic function; d i d j These are the depth values ​​for the i-th and j-th joint positions, respectively.

[0022] Furthermore, in step 1, the diffusion model is trained using a deep ranking loss, specifically formulated as follows:

[0023] ;

[0024] The total loss function is:

[0025] ;

[0026] In the formula, y is the relative depth indicator matrix; , Let be the predicted depth coordinates of the t-th frame; To generate the set of joint pairs for depth occlusion; The set physical depth threshold; , These are the loss weighting coefficients; Let be the predicted 3D spatial coordinates of the j-th joint point output by the model at frame t. The predicted true 3D coordinates of the j-th joint point at frame t; To find the vector Norm, is the Euclidean norm, which is the square root of the sum of the squares of the absolute values ​​of the vector's elements; T is the total number of frames.

[0027] Furthermore, in step 2, the three-dimensional pose features are:

[0028] ;

[0029] In the formula, X represents the three-dimensional pose feature; y represents the relative depth indication matrix; and PE represents the position code. The input is the coordinates of the key points of the two-dimensional human pose. Embed the weight matrix for linear features; This is a feature splicing operation.

[0030] Furthermore, in step 3, the global branch processing procedure includes:

[0031] The three-dimensional pose features are used as input features for the two branches respectively;

[0032] In the branch where the Transformer module and the Mamba module run in sequence: 3D pose features are used as input to the Transformer module, which then uses a self-attention mechanism to output purified spatial features. The specific formula is as follows:

[0033] ;

[0034] In the formula: For standard spatial Transformer encoder operation; The purified spatial characteristics of the output;

[0035] The purified spatial features are sliced ​​over time to obtain instantaneous velocities, and an adaptive step size is generated. The adaptive step size and the purified spatial features are used as inputs to the Mamba module to form spatiotemporal features, as shown in the following formula:

[0036] ;

[0037] in, ;

[0038] ;

[0039] In the formula: The purified spatial features are time sliced ​​at frame t. The purified spatial features are time sliced ​​at frame t-1. Instantaneous velocity; To find the vector Norm, It is the Euclidean norm, which is the square root of the sum of the squares of the absolute values ​​of the vector's elements; For adaptive step size; The learnable scaling factor; For smoothing nonlinear activation functions; A Mamba operator with an adaptive step size as a dynamic parameter; The output is the spatiotemporal feature; This is a linear mapping operation;

[0040] In the branch where the Mamba and Transformer modules run sequentially: instantaneous velocity is obtained by time-slicing the 3D pose features, and an adaptive step size is generated. The adaptive step size and 3D pose features are used as input to the Mamba module to model a long-time trajectory and output purified trajectory features. The specific formula is as follows:

[0041] ;

[0042] in, ;

[0043] ;

[0044] In the formula: The time slice of the 3D pose feature in frame t; The time slice of the 3D pose feature in the (t-1)th frame; Instantaneous velocity; For adaptive step size; The learnable scaling factor; The purified trajectory features are output;

[0045] The purified trajectory features are used as input to the Transformer module, which outputs spatiotemporal features using a self-attention mechanism. The specific formula is as follows:

[0046] ;

[0047] In the formula, Spatiotemporal characteristics;

[0048] The spatiotemporal features are fused with the spatiotemporal features to form a global structural feature. The specific formula is as follows:

[0049] ;

[0050] In the formula, For global structural features; For layer normalization operation;

[0051] The process of handling local branches includes:

[0052] Using 3D pose features as input to the Mamba module, the Mamba module outputs intermediate features with local temporal smoothness, as shown in the following formula:

[0053] ;

[0054] in, ;

[0055] ;

[0056] ;

[0057] In the formula, This refers to the adaptive time step parameter in the Mamba module; and This is the continuous-time state transition parameter matrix in the Mamba model; and It is the discretized local state transition matrix; The hidden historical state preserved for local branches at the previous time step; This is the projection matrix within a local Mamba module; The intermediate feature is E; E is the identity matrix.

[0058] The intermediate features are physically aggregated using spatial graph convolutional layers to form local structural features. The specific formula is as follows:

[0059] ;

[0060] in, ;

[0061] In the formula, This is the normalized topological adjacency matrix; This is the learnable projection weight matrix for the spatial graph convolutional layer; ( ) is a non-linear activation function; This is a pre-defined standard human anatomical skeleton-based adjacency matrix; It is an identity matrix with dimension equal to the number of joints J; For corresponding The degree matrix; The output consists of local structural features.

[0062] Furthermore, in step 3, the specific method for applying strong physical guidance constraints to the feature extraction of local branches using spatiotemporal topological constraints is as follows:

[0063] A dynamic topological constraint matrix is ​​constructed and applied directly to the local structural features of the local branch outputs using a multiplication mask.

[0064] Furthermore, the method for constructing the spatiotemporal topological constraints is as follows:

[0065] First, construct a spatial rigid constraint matrix: use the standard human anatomical skeleton adjacency matrix as the spatial rigid constraint matrix;

[0066] Then, a time-dynamic constraint matrix is ​​constructed based on the constraint triggering mechanism of composite similarity: the composite trajectory similarity metric of the current joint is obtained, and when the composite trajectory similarity metric changes abruptly, and a certain joint is in front of the current joint... When the set of the most similar topological neighbors is within the set of the most similar topological neighbors, a time-dynamic constraint is triggered.

[0067] Then, the spatial rigid constraint matrix and the time dynamic constraint matrix are fused together with adaptive parameters to form a dynamic topological constraint matrix;

[0068] Finally, the dynamic topological constraint matrix is ​​applied directly to the local structural features of the local branch output in the form of matrix dot product.

[0069] Furthermore, in step 3, before constructing the spatiotemporal topological constraints, the method further includes: dividing all key points into two groups using two grouping methods, wherein...

[0070] The first grouping method is to divide all joints into five groups based on anatomical semantics: left arm joint group, right arm joint group, left leg joint group, right leg joint group, and static joint group.

[0071] The second grouping method is to divide all joints into two groups based on kinematic stability: a high-stability joint group and a low-stability joint group.

[0072] When constructing the spatial rigid constraint matrix, the standard human anatomical skeleton adjacency matrix is ​​combined with the five parts group to form the spatial rigid constraint matrix;

[0073] When constructing the time-dynamic constraint matrix, the current joint point belongs only to the low-stability joint group.

[0074] Furthermore, the specific formula for the spatial rigid constraint matrix is ​​as follows:

[0075] ;

[0076] in, ;

[0077] In the formula, The matrix represents the spatial rigidity constraint matrix. This is a standard human anatomical skeleton adjacency matrix; This is a spatial decoupling mask matrix constructed based on five parts. J represents the total number of joints in the human skeleton. It is a set of five parts. ; This is a dot product operation;

[0078] The specific formula for the time dynamic constraint matrix is ​​as follows:

[0079] ;

[0080] in, ;

[0081] In the formula, The connection weights between node i and node j in the time-dynamic constraint matrix; This is an operation to select the set of the K nearest neighbor nodes; For the joint node of frame t Joint nodes of frame t' The spatiotemporal distance function between them; and These are the three-dimensional spatial coordinates of the corresponding node; and These are the first-order motion features of the corresponding nodes; This is a hyperparameter used to balance the weights of the Euclidean distance between position and velocity difference; This indicates finding the vector. Norm, It is the Euclidean norm, which is the square root of the sum of the squares of the absolute values ​​of the elements of the vector.

[0082] Furthermore, the specific formula for the dynamic topological constraint matrix is ​​as follows:

[0083] ;

[0084] In the formula, This is a dynamic topological constraint matrix; For adaptive parameters; Use the Sigmoid activation function; The matrix represents the spatial rigidity constraint matrix. This is the time-dynamic constraint matrix.

[0085] The beneficial effects of this invention are:

[0086] This invention comprehensively improves the accuracy, computational efficiency, and modeling ability of complex human topology in 3D human pose estimation. Specifically, in terms of expected quantitative indicators, the core evaluation metric, MPJPE (Mean Joint Position Error), is reduced to below 37.6 mm, representing a significant performance improvement compared to existing conventional methods. Furthermore, this invention effectively alleviates the computational bottlenecks and excessive memory overhead faced by existing methods when processing long-term sequences, achieving excellent synergy between global temporal modeling and local physical structure constraints.

[0087] This invention achieves the corresponding effects primarily through the combined application of the following core technologies:

[0088] 1. Global-Local Dual-Branch Network Architecture: A parallel Mamba-Transformer dual-branch structure is constructed, with each branch responsible for feature extraction from the global sequence and local topology, respectively, and then fused through an interactive mechanism. This overcomes the limitations of a single mechanism, satisfying both the local motion continuity required for high-quality pose representation and ensuring the consistency of global pose configuration, thus comprehensively improving the accuracy and structural rationality of prediction.

[0089] 2. The hybrid spatiotemporal modeling mechanism employs the following technique: In the global branch, the Transformer, which excels at global self-attention mechanisms in both space and time, is chained with a state-space model that features selective scanning and linear time complexity. The Transformer ensures the extraction of long-range dependency features across time and joints; while the introduction of Mamba effectively solves the problem of quadratic explosion in computational complexity when dealing with long video sequences using traditional self-attention mechanisms, significantly reducing computational and memory overhead and improving computational efficiency.

[0090] 3. Local modeling technique incorporating human skeletal topological priors: In local branches, instead of simply stacking to expand the receptive field, the technique explicitly introduces the real physical connections of the human skeleton as priors, focusing on constraining adjacent joints and their relative movements. This enhances the model's perception of local motion, avoids the "over-smoothing" phenomenon of node features caused by increasing the number of layers in conventional graph convolutional networks, and thus significantly improves the model's prediction accuracy in high-degree-of-freedom and error-prone areas such as the extremities.

[0091] 4. The dynamic spatiotemporal topology constraint module employs the following techniques: In the spatial dimension, it relies on a rigid skeletal topology; in the temporal dimension, it uses the K-nearest neighbor algorithm to dynamically generate a topology constraint matrix based on the similarity of joint motion features throughout the time series. This adaptively guides the feature update process, eliminating redundant connections and strengthening the weight allocation of strongly correlated nodes. It forces the network output to conform to the laws of human physical movement, significantly reducing the generation of unreasonable postures and enhancing the model's robustness in complex scenarios.

[0092] 5. Adaptive Feature Fusion Based on Gated Mechanism: The features output from the two branches are concatenated and fed into a gated projection network with Softmax. The fusion weights of local temporal features and global structural features are dynamically calculated and allocated. This enables the network to adaptively adjust for different action modes and joint states, ensuring that the model can find the optimal balance between local physical continuity and global spatiotemporal consistency in any state, further solidifying the robustness of the final feature representation. Attached Figure Description

[0093] The features and advantages of the invention will be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the invention in any way. In the drawings:

[0094] Figure 1 This is a flowchart of a specific embodiment of the present invention;

[0095] Figure 2 This is a schematic diagram of the local spatial features of the present invention;

[0096] Figure 3 This is a schematic diagram of the global timing characteristics of the present invention;

[0097] Figure 4 This is a schematic diagram of the grouping results of the first grouping method in a specific embodiment of the present invention;

[0098] Figure 5 This is a schematic diagram of the grouping results of the second grouping method in a specific embodiment of the present invention;

[0099] Figure 6 This is a visualization of the evolution of topological dependencies of network layer nodes at different depths in a specific embodiment of the present invention;

[0100] Figure 7 This is a visualization result of three-dimensional human pose estimation in a specific embodiment of the present invention. Detailed Implementation

[0101] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0102] The present invention will be further illustrated below with reference to specific embodiments. Those skilled in the art should understand that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. Modifications to the present invention in various equivalent forms all fall within the scope defined by the appended claims.

[0103] like Figures 1-3 As shown, this invention provides a three-dimensional human pose estimation method based on dynamic spatiotemporal topological dependence, comprising the following steps:

[0104] Step 1: Obtain the sequence of two-dimensional keypoint video frames in a continuous video frame. The specific formula is as follows:

[0105] ;

[0106] In the formula, This represents a set of two-dimensional joint video frame sequences; T is the frame sequence length; J is the number of joints; t is the frame number; and j is the joint number. These are the two-dimensional coordinates of the j-th joint in the t-th frame; This indicates that the coordinates of the key points are points in a two-dimensional real space;

[0107] The two-dimensional depth map and depth values ​​of the joint positions in the two-dimensional joint video frame sequence are obtained through a diffusion model. The specific process is as follows:

[0108] Using a pre-trained diffusion model From RGB original frame Extracting dense depth maps Based on the two-dimensional coordinates u and v, the depth value corresponding to the joint position is sampled. The specific formula is as follows:

[0109] ;

[0110] ;

[0111] In the formula, It is a two-dimensional depth map; For the diffusion model; I is the RGB original frame of the 2D joint video frame; d is the depth value of the joint position; u and v are the coordinate values ​​of the 2D coordinates; This is a spatial sampling operation, which uses the coordinate values ​​(u,v) of two-dimensional coordinates and the two-dimensional depth map d to extract the depth value at the corresponding location;

[0112] A relative depth indicator matrix is ​​formed as a priori for spatial relative ordering, and the specific formula is as follows:

[0113] ;

[0114] In the formula, y is the relative depth indicator matrix; sgn( ) is a symbolic function; d i d j These are the depth values ​​at the i-th and j-th joint positions, respectively;

[0115] To further solidify the introduced spatial relative ranking prior during the model optimization phase, a deep ranking loss function (DepthRankingLoss) is employed during the diffusion model training / optimization phase. The specific formula is as follows:

[0116] ;

[0117] The model is trained end-to-end using the joint standard mean squared position error (MPJPE) and depth ranking loss, with the total loss function being:

[0118] ;

[0119] In the formula, y is the relative depth indicator matrix; , To predict depth coordinates; To generate a set of joint pairs with depth occlusion, specifically, during the data preprocessing stage, for any keypoint pair (i,j), when their projected Euclidean distance on the two-dimensional image plane is less than a decision threshold... And the depth difference in real three-dimensional space is greater than When this occurs, it is determined to be a severe occlusion and included in the set. It participates in the calculation of depth-sorting loss; The set physical depth threshold; , These are the loss weighting coefficients; The predicted 3D spatial coordinates of the j-th joint point output by the model; The predicted true 3D coordinates of the j-th joint point; To find the vector Norm, The vector is defined as the Euclidean norm, which is the square root of the sum of the squares of the absolute values ​​of the vector's elements; T is the total number of frames.

[0120] By using a combined standard coordinate regression loss to optimize the network, the problem of depth inversion under complex perspectives is completely solved, and the final pose estimation accuracy is greatly improved.

[0121] Step 2: Map the 2D joint coordinates, relative depth indication matrix, and position encoding to a high-dimensional feature space using a feature embedding layer to obtain 3D pose features. The specific formula is as follows:

[0122] ;

[0123] In the formula, X represents the three-dimensional pose feature; y represents the relative depth indication matrix; and PE represents the position code. The input is the coordinates of the key points of the two-dimensional human pose. Embed the weight matrix for linear features; This is a feature splicing operation.

[0124] Step 3: Construct a model including global branches, local branches, and spatiotemporal topological constraints, and obtain global and local structural features. Specifically, use 3D pose features as input features for the parallel global and local branches.

[0125] Global branches and local branches are set up in parallel.

[0126] The global branch includes two parallel branches: one branch is a serial Transformer module and a Mamba module, and the other branch is a serial Mamba module and a Transformer module, to perform decoupled feature extraction.

[0127] The Transformer module is a space-first Transformer module; the Mamba module is a time-first Mamba module.

[0128] For a branch containing a serial Transformer module and a Mamba module, the whole-body global physical structure within a single frame is extracted first, and then the temporal evolution of the structural features is traced. The specific process is as follows:

[0129] First, the 3D pose features X are fed into the spatial Transformer module, and the spatial dependencies across joints are directly modeled using a self-attention mechanism:

[0130] ;

[0131] In the formula: For standard spatial Transformer encoder operation; The purified spatial characteristics of the output;

[0132] Spatial characteristics of purification Obtain instantaneous velocity by performing time slicing And generate adaptive step size Adaptive step size Spatial characteristics of purification The state transition is performed as input to the Mamba module, forming a space-time feature, as shown in the following formula:

[0133] ;

[0134] in, ;

[0135] ;

[0136] In the formula: The purified spatial features are time sliced ​​at frame t. The purified spatial features are time sliced ​​at frame t-1. Instantaneous velocity; To find the vector Norm, It is the Euclidean norm, which is the square root of the sum of the squares of the absolute values ​​of the vector's elements; For adaptive step size; The learnable scaling factor; For smoothing nonlinear activation functions; A Mamba operator with an adaptive step size as a dynamic parameter; The output is the spatiotemporal feature; This is a linear mapping operation;

[0137] For the other branch, which includes serial Mamba and Transformer modules, the independent long-term motion trajectory of each joint is first extracted, and then the motion trajectories across joints are aligned and interacted in spatial dimensions. The specific process is as follows:

[0138] Instantaneous velocity is obtained by performing time slices on the 3D pose features. And generate adaptive step size By using adaptive step size and 3D pose features as input to the Mamba module, long-term trajectories are modeled, and purified trajectory features are output. The specific formula is as follows:

[0139] ;

[0140] in, ;

[0141] ;

[0142] In the formula: The time slice of the 3D pose feature in frame t; The time slice of the 3D pose feature in the (t-1)th frame; Instantaneous velocity; For adaptive step size; The learnable scaling factor; The purified trajectory features are output;

[0143] The purified trajectory features are used as input to the Transformer module, which outputs spatiotemporal features using a self-attention mechanism. The specific formula is as follows:

[0144] ;

[0145] In the formula, Spatiotemporal characteristics;

[0146] The spatiotemporal features are fused with the spatiotemporal features to form a global structural feature. The specific formula is as follows:

[0147] ;

[0148] In the formula, For global structural features; This is a layer normalization operation.

[0149] The local branches consist of sequential Mamba modules and spatial graph convolutional layers, which extract local structural features;

[0150] The Mamba module is a time-first Mamba module; the spatial graph convolutional layer is a GCN module;

[0151] To capture the continuity of local motion for each individual joint over time, the Mamba module generates local state transition matrices by discretizing the continuous-time state-space model:

[0152] ;

[0153] ;

[0154] By combining the local state transition matrix, the input features of each joint node are sliced ​​along the time dimension. Independently execute state updates and output intermediate features with local temporal smoothness:

[0155] ;

[0156] in, ;

[0157] In the formula, This refers to the adaptive time step parameter in the Mamba module; and This is the continuous-time state transition parameter matrix in the Mamba model; and It is the discretized local state transition matrix; The hidden historical state preserved for local branches at the previous time step; This is the projection matrix within a local Mamba module; The intermediate feature is E; E is the identity matrix.

[0158] We introduce prior knowledge of the physical connections of the human skeleton. First, we perform symmetric normalization on the basic adjacency matrix of the standard human skeleton:

[0159] ;

[0160] Based on the normalized topological matrix By using spatial graph convolutional layers, intermediate features are forced to propagate and aggregate strictly along the physically real connection edges to form local structural features. The specific formula is as follows:

[0161] ;

[0162] In the formula, This is the normalized topological adjacency matrix; This is the learnable projection weight matrix for the spatial graph convolutional layer; () is a nonlinear activation function; This is a pre-defined standard human anatomical skeleton-based adjacency matrix; It is an identity matrix with dimension equal to the number of joints J; For corresponding The degree matrix; The output consists of local structural features.

[0163] Spatiotemporal topological constraints provide strong physical guidance for feature extraction in local branches:

[0164] To ensure that the feature update process strictly satisfies the physical laws of non-rigid human motion, a dynamic topological constraint matrix is ​​constructed. It is applied directly to the local structural features of the local branch output through a multiplication mask, thereby strongly correcting local abnormal features.

[0165] Before constructing spatiotemporal topological constraints, the entire system... Two different dimensions of rigid skeleton region partitioning are performed in parallel for each keypoint. All keypoints are divided into two groups using two different grouping methods.

[0166] like Figure 4 As shown in the diagram, numbers 1-17 represent the 17 joints, i.e., joints 1 through 17. Joint 1 is the pelvis (root node), joint 2 is the right hip joint, joint 3 is the right knee, joint 4 is the right ankle, joint 5 is the left hip joint, joint 6 is the left knee, joint 7 is the left ankle, joint 8 is the spine, joint 9 is the thoracic vertebrae, joint 10 is the neck, joint 11 is the head, joint 12 is the left shoulder, joint 13 is the left elbow, joint 14 is the left wrist, joint 15 is the right shoulder, joint 16 is the right elbow, and joint 17 is the right wrist.

[0167] The first grouping method is as follows: based on anatomical semantics, all joints are divided into five groups. According to the natural structure of the human body, the joints are divided into the left arm joint group, the right arm joint group, the left leg joint group, the right leg joint group, and the static joint group. The static joint group includes the joints of the trunk and head. This grouping aims to strengthen the decoupling of local kinematic features, forcing them to prioritize information flow within the same limb to prevent cross-limb feature confusion. It fundamentally sets up a "physical isolation zone" in the spatiotemporal domain to block the loss of high-frequency motion details caused by traditional deep networks.

[0168] like Figure 5As shown in the diagram, numbers 1-17 represent the 17 joints, i.e., joints 1 through 17. Joint 1 is the pelvis (root node), joint 2 is the right hip joint, joint 3 is the right knee, joint 4 is the right ankle, joint 5 is the left hip joint, joint 6 is the left knee, joint 7 is the left ankle, joint 8 is the spine, joint 9 is the thoracic vertebrae, joint 10 is the neck, joint 11 is the head, joint 12 is the left shoulder, joint 13 is the left elbow, joint 14 is the left wrist, joint 15 is the right shoulder, joint 16 is the right elbow, and joint 17 is the right wrist.

[0169] The second grouping method is as follows: based on kinematic stability, all joints are divided into two groups, namely, a high-stability joint group and a low-stability joint group, according to the jitter characteristics of the joints in non-rigid motion. This grouping aims to construct adaptive error correction anchor points, and use the high-stability joint features with strong anti-occlusion ability and high prediction accuracy as the absolute reference benchmark of the spatial coordinate system, while also guiding the asymmetric soft trigger weights of the subsequent dynamic graph edges.

[0170] The method for constructing spatiotemporal topological constraints is as follows:

[0171] First, construct a spatial rigid constraint matrix: combine the standard human anatomical skeleton adjacency matrix with the five part groups obtained by the first grouping method to form a spatial rigid constraint matrix, specifically:

[0172] First, define a spatial decoupling mask matrix in the spatial dimension: This is used to quantitatively characterize whether any two joints belong to the same anatomical group, and its mathematical condition equation is:

[0173] ;

[0174] Based on the standard human skeleton adjacency matrix In this context, the aforementioned spatial decoupling mask matrix is ​​introduced. Perform Hadamard product operations to construct a spatially rigid constraint matrix ultimately constrained by a strong anatomical semantic atlas. :

[0175] ;

[0176] In the formula, The matrix represents the spatial rigidity constraint matrix. This is a standard human anatomical skeleton adjacency matrix; This is a spatial decoupling mask matrix constructed based on five parts. J represents the total number of joints in the human skeleton. It is a set of five parts. ; This is a dot product operation;

[0177] By using a grouping mechanism and a spatial decoupling mask matrix, the spatial decoupling mask matrix forces local features to be strictly "locked" within their respective groups (such as "left arm") during updates, effectively cutting off the overlap of erroneous features between limbs in different groups.

[0178] The construction of the time-dynamic constraint matrix employs asymmetric condition triggering, and also introduces the K-nearest neighbor algorithm in the time dimension:

[0179] The "high-stability joint" is used as an absolutely rigid reference system for spatial composite distance calculation to ensure that the benchmark does not drift; at the same time, the extremities of the limbs classified as "low-stability joints" are given extremely high asymmetric trigger sensitivity.

[0180] When a low-stability joint experiences a sudden change in the composite trajectory similarity metric due to violent movement or occlusion, it is necessary to determine whether other joints fall before the current joint. Within the Top-K most similar topological neighbors, if other key points exist that fall within the Top-K, the non-adjacent high-dimensional connection edges in the time dimension are automatically activated, i.e., the time dynamic constraint is triggered.

[0181] The specific formula for the time dynamic constraint matrix is ​​as follows:

[0182] ;

[0183] in, ;

[0184] The composite trajectory similarity measure integrates "Euclidean distance of spatial location" and "kinematic velocity difference".

[0185] In the formula, The connection weights between node i and node j in the time-dynamic constraint matrix; This is an operation to select the set of the K nearest neighbor nodes; For the joint node of frame t Joint nodes of frame t' The spatiotemporal distance function between them; and These are the three-dimensional spatial coordinates of the corresponding node; and These are the first-order motion features of the corresponding nodes; This is a hyperparameter used to balance the weights of the Euclidean distance between position and velocity difference; This indicates finding the vector. Norm.

[0186] Then, the spatial rigid constraint matrix and the time dynamic constraint matrix are fused together with adaptive parameters to form the dynamic topological constraint matrix, the specific formula of which is:

[0187] ;

[0188] In the formula, This is a dynamic topological constraint matrix; For adaptive parameters; () is the Sigmoid activation function; The matrix represents the spatial rigidity constraint matrix. This is the time-dynamic constraint matrix;

[0189] Finally, the dynamic topological constraint matrix is ​​directly applied to the local structural features of the local branch output in the form of matrix dot product, as shown in the following formula:

[0190] .

[0191] Step 4: To avoid feature drift caused by conventional feature concatenation, an adaptive fusion strategy based on confidence gating is adopted. First, the features output by the global branch are... Features of local branch output Perform the splicing operation and input it into the gating network to implicitly evaluate the motion uncertainty of the current joint and generate fusion weights:

[0192] ;

[0193] In the formula, The projection matrix; This is the normalized activation function.

[0194] Then, using this weight, a weighted sum is performed to obtain the final features of the current layer:

[0195] ;

[0196] By dynamically allocating the weights of local rigid features and global temporal features, a perfect balance between local continuity and global consistency is achieved.

[0197] Step 5: Combine the fused features to perform 3D pose coordinate regression and obtain the 3D human pose estimation results:

[0198] ;

[0199] In the formula, This is the final predicted sequence of three-dimensional human pose coordinates; For use as a linear mapping layer in coordinate regression; This represents the final feature representation output by the network after adaptive fusion via a gating mechanism. The tensor dimension space in which the prediction result is located, where T is the number of time series frames of the input video, J is the number of key joints of the human skeleton, and the number 3 is the (x,y,z) coordinate component in the three-dimensional physical space.

[0200] like Figure 6 As shown, the dynamic evolution of feature interaction weights and topological dependencies among 17 joints of the human body at different network depths is intuitively demonstrated during the feature forward propagation process of the model proposed in this invention. The horizontal and vertical axes of the matrix in the figure correspond to the 17 key physical joints of the human skeleton, and the heatmap color transitions from dark blue to dark red, quantitatively representing the strength of feature attention or topological connections applied between joint nodes.

[0201] Shallow global mapping (Layer0-Layer1): In the shallow layers of the network, the attention weights of each key point exhibit a relatively uniform strip-like or globally generalized distribution. At this stage, the network mainly extracts basic image features and coarse-grained initial relative spatial ordering priors, and strong local boundaries have not yet been formed.

[0202] Intermediate-layer physical clustering (Layer 2-Layer 4): As the network depth increases, thanks to the multi-dimensional joint decoupling grouping prior and kinematic gradient sensing mechanism introduced in this invention, the interaction matrix undergoes significant structural reorganization. The weight distribution begins to break the initial global indifference state and rapidly converges to local physical clustering near the diagonal, with high-response regions gradually focusing within the same anatomical group.

[0203] Deep Powerful Correction and Anti-Smoothing (Layer 5): Observing the feature matrix of the deepest layer, it can be found that it not only does not exhibit the fatal "over-smoothing" phenomenon in traditional deep graph neural networks, but also shows extremely clear block boundaries and sparse asymmetric activation features.

[0204] like Figure 7 As shown, the visualization results of this model are presented, proving that the method of this invention can achieve good results in the task of 3D human pose estimation, and the generated 3D pose is relatively accurate.

[0205] The following section compares the joint estimation errors of the method of this invention with other existing methods using the Human3.6M dataset. The comparison results are shown in Table 1 below.

[0206] Table 1 shows the average joint error on the Human 3.6M dataset.

[0207]

[0208] Experimental results show that, on the Human3.6M dataset, the average joint error of this invention is only 37.5 mm, which is a significant reduction in average joint error compared to previous methods.

[0209] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A three-dimensional human pose estimation method based on dynamic spatiotemporal topological dependence, characterized in that, Includes the following steps: Step 1: Obtain the two-dimensional joint video frame sequence in the continuous video frames, and obtain the two-dimensional depth map of the two-dimensional joint video frame sequence and the depth value of the joint position through the diffusion model to form a relative depth indication matrix. Step 2: Map the 2D joint coordinates, relative depth indication matrix, and position encoding to a high-dimensional feature space through a feature embedding layer to obtain 3D pose features; Step 3: Construct a model including global branches, local branches, and spatiotemporal topological constraints to obtain global and local structural features. 3D pose features are used as input features for the parallel global and local branches. The global branch includes two parallel branches: one consisting of a serial Transformer module and a Mamba module, and the other consisting of a serial Mamba module and a Transformer module. The global branch is used to extract global structural features. The local branch includes a serial Mamba module and a spatial graph convolutional layer to extract local structural features. Spatiotemporal topological constraints provide strong physical guidance for the feature extraction of the local branches. Step 4: Adaptive feature fusion of global and local structural features based on kinematic confidence; Step 5: Combine the fused features to perform 3D pose coordinate regression and obtain the 3D human pose estimation results.

2. The three-dimensional human pose estimation method based on dynamic spatiotemporal topological dependence as described in claim 1, characterized in that, In step 1, the specific method for obtaining the two-dimensional depth map of the two-dimensional joint video frame sequence and the depth value of the joint position using the diffusion model is as follows: Two-dimensional depth maps are extracted from the RGB original frames of a two-dimensional joint video frame sequence using a diffusion model, and the depth values ​​at the corresponding joint positions are sampled from these maps based on the two-dimensional coordinates. The specific formula is as follows: ； ； In the formula, It is a two-dimensional depth map; For the diffusion model; I is the RGB original frame of the 2D joint video frame; d is the depth value of the joint position; u and v are the coordinate values ​​of the 2D coordinates; For spatial sampling operations; The specific formula for the resulting relative depth indication matrix is: ； In the formula, y is the relative depth indicator matrix; sgn( ) is a symbolic function; d i d j These are the depth values ​​for the i-th and j-th joint positions, respectively.

3. The three-dimensional human pose estimation method based on dynamic spatiotemporal topological dependence as described in claim 1 or 2, characterized in that, In step 1, the diffusion model is trained using a depth-ranking loss, the specific formula of which is: ； The total loss function is: ； In the formula, y is the relative depth indicator matrix; , Let be the predicted depth coordinates of the t-th frame; To generate the set of joint pairs for depth occlusion; The set physical depth threshold; , These are the loss weighting coefficients; Let be the predicted 3D spatial coordinates of the j-th joint point output by the model at frame t. The predicted true 3D coordinates of the j-th joint point at frame t; To find the vector Norm, is the Euclidean norm, which is the square root of the sum of the squares of the absolute values ​​of the vector's elements; T is the total number of frames.

4. The three-dimensional human pose estimation method based on dynamic spatiotemporal topological dependence as described in claim 1 or 2, characterized in that, In step 2, the three-dimensional pose features are: ； In the formula, X represents the three-dimensional pose feature; y represents the relative depth indication matrix; and PE represents the position code. The input is the coordinates of the key points of the two-dimensional human pose. Embed the weight matrix for linear features; This is a feature splicing operation.

5. The three-dimensional human pose estimation method based on dynamic spatiotemporal topological dependence as described in claim 4, characterized in that, In step 3, the global branch processing procedure includes: The three-dimensional pose features are used as input features for the two branches respectively; In the branch where the Transformer module and the Mamba module run in sequence: 3D pose features are used as input to the Transformer module, which then uses a self-attention mechanism to output purified spatial features. The specific formula is as follows: ； In the formula: For standard spatial Transformer encoder operation; The purified spatial characteristics of the output; The purified spatial features are sliced ​​over time to obtain instantaneous velocities, and an adaptive step size is generated. The adaptive step size and the purified spatial features are used as inputs to the Mamba module to form spatiotemporal features, as shown in the following formula: ； in, ; ； In the formula: The purified spatial features are time sliced ​​at frame t. The purified spatial features are time sliced ​​at frame t-1. Instantaneous velocity; To find the vector Norm, It is the Euclidean norm, which is the square root of the sum of the squares of the absolute values ​​of the vector's elements; For adaptive step size; The learnable scaling factor; For smoothing nonlinear activation functions; A Mamba operator with an adaptive step size as a dynamic parameter; The output is the spatiotemporal feature; This is a linear mapping operation; In the branch where the Mamba and Transformer modules run sequentially: instantaneous velocity is obtained by time-slicing the 3D pose features, and an adaptive step size is generated. The adaptive step size and 3D pose features are used as input to the Mamba module to model a long-time trajectory and output purified trajectory features. The specific formula is as follows: ； in, ; ； In the formula: The time slice of the 3D pose feature in frame t; The time slice of the 3D pose feature in the (t-1)th frame; Instantaneous velocity; For adaptive step size; The learnable scaling factor; The purified trajectory features are output; The purified trajectory features are used as input to the Transformer module, which outputs spatiotemporal features using a self-attention mechanism. The specific formula is as follows: ； In the formula, Spatiotemporal characteristics; The spatiotemporal features are fused with the spatiotemporal features to form a global structural feature. The specific formula is as follows: ； In the formula, For global structural features; For layer normalization operation; The process of handling local branches includes: Using 3D pose features as input to the Mamba module, the Mamba module outputs intermediate features with local temporal smoothness, as shown in the following formula: ； in, ; ； ； In the formula, This refers to the adaptive time step parameter in the Mamba module; and This is the continuous-time state transition parameter matrix in the Mamba model; and It is the discretized local state transition matrix; The hidden historical state preserved for local branches at the previous time step; This is the projection matrix within a local Mamba module; The intermediate feature is E; E is the identity matrix. The intermediate features are physically aggregated using spatial graph convolutional layers to form local structural features. The specific formula is as follows: ； in, ; In the formula, This is the normalized topological adjacency matrix; This is the learnable projection weight matrix for the spatial graph convolutional layer; ( ) is a non-linear activation function; This is a pre-defined standard human anatomical skeleton-based adjacency matrix; It is an identity matrix with dimension equal to the number of joints J; For corresponding The degree matrix; The output consists of local structural features.

6. The three-dimensional human pose estimation method based on dynamic spatiotemporal topological dependence as described in claim 1, characterized in that, In step 3, the specific method for applying strong physical law-guided constraints to the feature extraction of local branches using spatiotemporal topological constraints is as follows: A dynamic topological constraint matrix is ​​constructed and applied directly to the local structural features of the local branch outputs using a multiplication mask.

7. The three-dimensional human pose estimation method based on dynamic spatiotemporal topological dependence as described in claim 6, characterized in that, The method for constructing the spatiotemporal topological constraints is as follows: First, construct a spatial rigid constraint matrix: use the standard human anatomical skeleton adjacency matrix as the spatial rigid constraint matrix; Then, a time-dynamic constraint matrix is ​​constructed based on the constraint triggering mechanism of composite similarity: the composite trajectory similarity metric of the current joint is obtained, and when the composite trajectory similarity metric changes abruptly, and a certain joint is in front of the current joint... When the set of the most similar topological neighbors is within the set of the most similar topological neighbors, a time-dynamic constraint is triggered. Then, the spatial rigid constraint matrix and the time dynamic constraint matrix are fused together with adaptive parameters to form a dynamic topological constraint matrix; Finally, the dynamic topological constraint matrix is ​​applied directly to the local structural features of the local branch output in the form of matrix dot product.

8. The three-dimensional human pose estimation method based on dynamic spatiotemporal topological dependence as described in claim 7, characterized in that, Step 3, before constructing the spatiotemporal topological constraints, further includes: dividing all key points into two groups using two grouping methods, wherein... The first grouping method is to divide all joints into five groups based on anatomical semantics: left arm joint group, right arm joint group, left leg joint group, right leg joint group, and static joint group. The second grouping method is to divide all joints into two groups based on kinematic stability: a high-stability joint group and a low-stability joint group. When constructing the spatial rigid constraint matrix, the standard human anatomical skeleton adjacency matrix is ​​combined with the five parts group to form the spatial rigid constraint matrix; When constructing the time-dynamic constraint matrix, the current joint point belongs only to the low-stability joint group.

9. The three-dimensional human pose estimation method based on dynamic spatiotemporal topological dependence as described in claim 8, characterized in that, The specific formula for the spatial rigid constraint matrix is ​​as follows: ； in, ; In the formula, The matrix represents the spatial rigidity constraint matrix. This is a standard human anatomical skeleton adjacency matrix; This is a spatial decoupling mask matrix constructed based on five parts. J represents the total number of joints in the human skeleton. It is a set of five parts. ; This is a dot product operation; The specific formula for the time dynamic constraint matrix is ​​as follows: ； in, ; In the formula, The connection weights between node i and node j in the time-dynamic constraint matrix; This is an operation to select the set of the K nearest neighbor nodes; For the joint node of frame t Joint nodes of frame t' The spatiotemporal distance function between them; and These are the three-dimensional spatial coordinates of the corresponding node; and These are the first-order motion features of the corresponding nodes; This is a hyperparameter used to balance the weights of the Euclidean distance between position and velocity difference; This indicates finding the vector. Norm, It is the Euclidean norm, which is the square root of the sum of the squares of the absolute values ​​of the elements of the vector.

10. The three-dimensional human pose estimation method based on dynamic spatiotemporal topological dependence as described in claim 9, characterized in that, The specific formula for the dynamic topological constraint matrix is ​​as follows: ； In the formula, This is a dynamic topological constraint matrix; For adaptive parameters; Use the Sigmoid activation function; The matrix represents the spatial rigidity constraint matrix. This is the time-dynamic constraint matrix.

Images