3D human pose estimation method using adaptive graph convolution and state space model

Abstract

The application discloses a 3D human posture estimation method based on adaptive graph convolution and a state space model, and specifically comprises the following steps: step 1, obtaining a human posture estimation dataset, processing a training set and a test set in the dataset, and performing online enhancement on the training set data; step 2, constructing a 3D human posture estimation network model based on adaptive graph convolution and a state space model; step 3, performing end-to-end training on the model constructed in step 2 by using the training set to obtain an optimal model after training; and step 4, performing prediction on the optimal model after training in step 3 by using the test set to output a 3D human posture sequence consistent in space and time. The application solves the problem of inaccurate estimation results existing in the prior art.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and artificial intelligence technology, and relates to a 3D human pose estimation method based on adaptive graph convolution and state space model. Background Technology

[0002] 3D human pose estimation aims to reconstruct the three-dimensional spatial coordinates of human joints from two-dimensional images or video sequences, and is a core technology for computers to understand human behavior. This technology plays an irreplaceable role in areas such as pedestrian intent prediction in autonomous driving, abnormal behavior recognition in security monitoring, and virtual reality (VR) and motion capture. For example, in autonomous driving scenarios, accurate and real-time pose estimation can help the system predict pedestrian movements in advance, thereby avoiding traffic accidents; in human-computer interaction, high-fidelity pose reconstruction is the foundation for achieving natural interaction.

[0003] However, existing 3D human pose estimation techniques still face two major bottlenecks in practical applications: First, the high latency caused by long sequence processing. To address occlusion and depth blur in videos, systems typically need to analyze long temporal contexts, which leads to a sharp increase in computational model complexity and large inference latency, making it difficult to achieve efficient processing in scenarios such as human-computer interaction where low latency is critical. Second, insufficient accuracy and stability of pose reconstruction in complex scenes. Human motion is highly heterogeneous, with significant differences in motion patterns between different joints (such as flexible limbs and stable torso). Existing methods often fail to adequately model this joint motion heterogeneity, resulting in reconstructed 3D poses prone to severe temporal jitter, loss of local details, and anatomical distortion in complex scenes with rapid motion or severe occlusion. Summary of the Invention

[0004] The purpose of this invention is to provide a 3D human pose estimation method based on adaptive graph convolution and state space model, thereby solving the problem of inaccurate estimation results in existing 3D human pose estimation methods.

[0005] The technical solution adopted in this invention is a 3D human pose estimation method based on adaptive graph convolution and a state-space model, which specifically includes the following steps: Step 1: Obtain the human pose estimation dataset, process the training and test sets in the dataset, and perform online augmentation on the training set data; Step 2: Construct a 3D human pose estimation network model with adaptive graph convolution and state space model; Step 3: Use the training set to perform end-to-end training on the model built in Step 2 to obtain the optimal model after training; Step 4: Use the test set to predict the optimal model trained in Step 3, and output a spatiotemporally consistent 3D human pose sequence.

[0006] The invention is further characterized by: The specific process of step 1 is as follows: Step 1.1: Extract video frame sequences from the human pose dataset. For both the training and test sets, use a 2D pose detector to obtain the coordinates of 2D human key points for each frame. As input, obtain the corresponding 3D human body key point coordinates. Labels are ground truth values; Step 1.2, for the 3D human body key point coordinate labels, denoted as the original coordinates. If the joint with index 0 is set as the root joint, then the first joint... The first frame Relative 3D coordinates of each joint It is calculated using the following formula: (1) in, Indicates the first Frame number The original absolute coordinates of each joint Indicates the first The original absolute coordinates of the frame root joint; For 2D human keypoint coordinate input, let the original coordinates be denoted as . , No. The first frame Relative 2D coordinates of each joint It is calculated using the following formula: (2) in, Indicates the first Frame number The original coordinates of each joint in the image coordinate system Indicates the first The original coordinates of the frame root joint; Step 1.3 involves sequentially horizontally flipping the training data, injecting noise, and performing masking operations.

[0007] In step 2, the 3D human pose estimation network model of adaptive graph convolution and state space model includes a linear projection layer, a spatial adaptive symmetric graph convolution sub-network, a spatiotemporal position coding module, a temporal Mamba sub-network, a token pruning-recovery module, and a regression head.

[0008] The specific process of step 2 is as follows: Step 2.1: After linear projection, the 2D human keypoint coordinate sequence output in Step 1 is used to output the initial feature tensor. ; Step 2.2, convert the initial feature tensor output in Step 2.1 into a single value. Input space adaptive symmetric graph convolutional subnetwork outputs feature tensor ; Step 2.3, extract the spatial augmentation features output from Step 2.2. Spatiotemporal sensing features are output through the spatiotemporal location coding module. ; Step 2.4: Develop the spatiotemporal awareness features output from Step 2.3. Input to by N In a temporal Mamba subnetwork composed of temporal Mamba blocks, the deep temporal feature tensor is output. Step 2.5, in the token pruning-recovery module, the pruning module receives the output characteristics of the Layer 4 time-series Mamba block. Output sparse feature tensors after token pruning The recovery module receives the sparse characteristics of the last layer output of the temporal Mamba subnetwork. After token recovery, output ; Step 2.6: Recover the full-length temporal features from Step 2.5. 3D human pose sequence is output through the regression head. .

[0009] The specific process of step 2.1 is as follows: Step 2.1.1: Take the 2D human keypoint coordinate sequence output from Step 1 as input, denoted as... The dimension of the sequence is defined as , where T represents the number of frames in the time series and J represents the number of human joints; Step 2.1.2: Apply a linear projection layer to the input sequence. Perform a linear affine transformation, mapping to a feature dimension of d. m The feature space yields the initial feature tensor. :

[0010] in, , This is the weight matrix. This is the bias vector.

[0011] The specific process of step 2.2 is as follows: Step 2.2.1: Initialize a learnable joint embedding matrix for the network. The original adjacency matrix is ​​calculated by matrix multiplication. The formula is as follows:

[0012] Step 2.2.2, based on the original adjacency matrix obtained in Step 2.2.1 Introducing the identity matrix The enhanced adjacency matrix is ​​calculated. The calculation formula is: (5) Degree matrix The diagonal elements are calculated using the following formula:

[0013] in, Degree matrix The Middle Line number The diagonal elements of the column; In the enhanced adjacency matrix, the first... Line number The element values ​​of the column; The small constant representing the prevention of the denominator being zero is set as follows in this invention. ; This indicates taking the maximum of the two values ​​within the parentheses.

[0014] Finally, symmetric normalization is performed to obtain the final normalized support matrix. The calculation formula is as follows:

[0015] in, Degree matrix The inverse square root matrix; Step 2.2.3, using Chebyshev polynomials recursively generate graph propagation operators at three different scales. :

[0016] in, ( Aggregate its own node information, ( Aggregate information from directly adjacent joints. Aggregate information from next nearest neighbor joints; Step 2.2.4, convert the initial feature tensor output in step 2.1 into... Expanding along the time dimension, let's assume Indicates the first The feature matrices at each time step, and the spatial adaptive symmetric graph convolutional subnetwork, are fused with the spatial topology operators generated in step 2.2.3. The node features are updated according to the following formula, along with the dynamically synthesized node-specific weights:

[0017] in, It is a time step Updated features; Represents the current frame in the 1st position. Aggregation features at various scales; Indicates by along the embedding dimension The set of node-specific weight matrices generated by tensor contraction; operator This represents node-wise matrix multiplication; after independently applying a spatially adaptive symmetric graph convolutional subnetwork to each frame of the input sequence, the output results are combined along the time dimension to obtain the spatially augmented sequence tensor. .

[0018] The specific process of step 2.3 is as follows: Step 2.3.1: Construct two independent sets of learnable parameter matrices: spatial location embedding. and time location embedding ; Step 2.3.2, Spatial Augmentation Features Based on the Output of Step 2.2 Using the broadcast mechanism to respectively and Copy and expand along the time axis, then embed the expanded position into the input features. By adding elements one by one, we can obtain the position-aware features. :

[0019] Step 2.3.3, location-aware features Layer normalization is performed using the following formula:

[0020] Finally, spatiotemporal perception features are obtained. .

[0021] The specific process of step 2.4 is as follows: Step 2.4.1: Within each time-series Mamba block, parameter generation is first performed for each time step in the input keypoint coordinate sequence. Receive the input features of the current frame Using a linear projection layer according to Dynamically generate time scale parameters and input projection matrix and output projection matrix Based on the zero-order preservation principle, the continuous state transition matrix is... Discretization, the calculation formula is as follows:

[0022]

[0023] (16) (17) in, The input feature vector; Represents a learnable linear projection function; Represents globally learnable scalar baseline parameters; This indicates tensor dimension expansion; Represents a smooth, non-linear activation function; Indicates time step The dynamic time scale parameters; In a continuous-time state-space model, , They represent the input respectively Dynamically generated continuous-time input and output projection matrices; This represents the state transition matrix obtained after zero-order preserved discretization; This represents the input projection matrix obtained after discretization; Represents the identity matrix; Represents the matrix exponential function; Step 2.4.2, each bidirectional time SSM module internally executes a bidirectional scan strategy: Forward scan: utilizing state equations Recursively update the latent state and utilize the output equation. Capture the causal dependencies of motion; Backward scan: Perform the same recursive operations as the forward scan, where, Indicates time step The potential state vector; This represents the potential state vector of the previous time step; after the scan is completed, the forward scan output features and the backward scan output features are added and fused to obtain the global temporal feature response of the current Mamba block; Step 2.4.3, for the first A sequence of Mamba blocks, first input the first one. Timing Mamba block output characteristics After layer normalization, depthwise separable convolution and SiLU activation are applied. Then, global temporal modeling is performed using the bidirectional temporal SSM module described in step 2.4.2 above. Subsequently, intermediate features are obtained through residual connections and MLP layers. ,at last After layer normalization and MLP layers, residual connections are added to obtain the output features of this time-series Mamba block. :

[0024] in, Represents SiLU activation. This represents a bidirectional time-based SSM module.

[0025] The specific process of step 2.5 is as follows: Step 2.5.1, features output from the fourth-layer temporal Mamba block. Before entering the next-level temporal Mamba block, the feature tensor is pruned, and the output features of the fourth-level temporal Mamba block are received. Along the joint dimension Adaptive average pooling is performed to obtain frame-level feature sequences. ; Subsequently, a density-based clustering algorithm was used to filter keyframes, specifically the first keyframe in the sequence. Frame, calculate decision score: Local density : Calculate the first Frame and The average distance between the nearest neighbor frames in the feature space is converted into density values ​​using a Gaussian kernel function:

[0026] in, This indicates that in the feature space, the distance to the th The nearest frame A set of indexes for each frame; Represents the index of a nearest neighbor frame in the set; Separation distance : Calculate the minimum distance from this frame to any frame with a higher density than itself:

[0027] Represents a set of criteria for selection; Selection decision: Calculate the overall score. Based on the time index of the retained token, extract the corresponding features from the input tensor and output the pruned sparse feature tensor. ; Step 2.5.2: Receive the sparse features of the output of the last layer of time-series Mamba blocks. It employs a cross-attention mechanism: Initialize query: Define a set of learnable recovery tokens And inject position encoding; Context aggregation: For querying, with sparse features Using keys and values, full-length sequence features are reconstructed through multi-head cross-attention, leveraging sparse keyframe features. Details:

[0028] in, This represents the feature dimension of each attention head in a multi-head attention mechanism. This indicates the normalized exponential function operation.

[0029] The specific process of step 3 is as follows: The 2D keypoint sequences from the training set are input into the model to obtain the predicted 3D pose sequences. And obtain the corresponding 3D pose ground truth value. Three key loss terms are selected to construct a comprehensive loss function. The overall loss function L is defined as:

[0030] in, , , These are the weighting coefficients for each loss term; Mean joint position error loss The calculation formula is as follows: (twenty four) Normalized scaling loss First, the optimal scalar scaling factor for each frame is calculated using the least squares method. This minimizes the distance between the scaled predicted pose and the true pose, and then the minimum distance is calculated:

[0031]

[0032] Weighted position error loss The calculation formula is as follows: (27) in, For the number of frames in the sequence, For the number of joints, This represents the L2 norm.

[0033] The beneficial effects of this invention are as follows: (1) The overall network structure is a spatiotemporal heterogeneous decoupling framework. This framework utilizes spatial adaptive symmetric graph convolution and Mamba to perform customized modeling for the characteristics of spatial structure and time series, respectively. By combining long-range temporal modeling with linear complexity with structure-aware fine-grained spatial reasoning, the framework effectively aligns the architectural inductive bias with the essential attributes of the data, avoiding the limitations of traditional homogeneous models in processing spatiotemporally variable data, thereby achieving more accurate 3D pose estimation.

[0034] (2) By designing the Spatial Adaptive Symmetric Graph Convolutional Subnetwork (ASGCN), a node adaptive filter synthesis mechanism is introduced, and joint embedding is used to dynamically generate a convolutional kernel for each joint. This design breaks through the limitation of global weight sharing in traditional GCN and realizes differentiated and refined modeling of the functions of different joints in the human body. At the same time, the algebraic degree normalization strategy is adopted to preserve the adversarial motion dependencies between joints and ensure the numerical stability of multi-hop spectrum propagation, thereby enabling more effective capture of potential non-local dependencies.

[0035] (3) A linear complexity temporal modeling and token pruning-recovery strategy based on Mamba was introduced. By utilizing the selective mechanism and bidirectional scanning strategy of the state-space model, long-range motion dynamics are captured efficiently, solving the bottleneck of quadratic growth in computational complexity of the Transformer architecture in long video sequence processing. Combined with a density-based dynamic token pruning mechanism, highly redundant temporal frames are automatically filtered out in the middle layer of the network, and deep feature extraction is performed only on key frames. While maintaining high accuracy, the floating-point operation and inference latency are significantly reduced, achieving a superior balance between accuracy and efficiency. Attached Figure Description

[0036] Figure 1 This is the model framework designed in the 3D human pose estimation method of adaptive graph convolution and state space model of this invention; Figure 2 This is a structural diagram of the spatial modeling module in the 3D human pose estimation method based on adaptive graph convolution and state space model of this invention. Figure 3 This is a visualization comparison of the 3D human pose estimation method based on adaptive graph convolution and state space model of this invention with existing advanced methods. Detailed Implementation

[0037] The following detailed description is provided in conjunction with specific implementation methods.

[0038] Example 1 This invention relates to a 3D human pose estimation method based on adaptive graph convolution and a state-space model, such as... Figure 1 As shown, the specific steps are as follows: Step 1: Obtain the human pose estimation dataset, process the training and test sets in the dataset, and perform online augmentation on the training set data; Step 2: Construct a 3D human pose estimation network model with adaptive graph convolution and state space model, which maps the input 2D human keypoint coordinate sequence to a 3D human keypoint coordinate sequence and outputs it. The network model includes a linear projection layer, a spatial adaptive symmetric graph convolution sub-network, a spatiotemporal position coding module, a temporal Mamba sub-network, a token pruning-recovery module, and a regression head. Step 3: Use the training set to train the model built in Step 2 end-to-end; calculate the comprehensive loss function, and use the AdamW optimizer to update the network parameters through backpropagation, iterating until the model converges to obtain the optimal model after training. Step 4: Use the test set to predict the optimal model trained in Step 3; input the 2D noisy keypoint coordinate sequence from the test set into the model, and the model outputs a high-fidelity and spatiotemporally consistent 3D human pose coordinate sequence.

[0039] Example 2 Step 1 is implemented in the following steps: Step 1.1: Extract video frame sequences from the Human3.6M dataset. For both the training and test sets, use an existing 2D pose detector (Stacked Hourglass in this embodiment) to obtain the 2D human keypoint coordinates for each frame. As input, obtain the corresponding 3D human body key point coordinates. The label serves as the ground truth value.

[0040] Step 1.2: To eliminate the absolute translational differences of the human body in the video frame or 3D space, and to allow the model to focus on the relative structural learning of the human body posture, root joint relativization processing is performed on the input 2D data and 3D label data; for the 3D human body keypoint coordinate labels, the original coordinates are denoted as... If we set the joint with index 0 as the root joint, then the first joint... The first frame Relative 3D coordinates of each joint It is calculated using the following formula:

[0041] in, Indicates the first Frame number The original absolute coordinates of each joint Indicates the first The original absolute coordinates of the frame root joint.

[0042] For 2D human keypoint coordinate input, let the original coordinates be denoted as . Similarly, normalization is performed relative to the root joint, the first... The first frame Relative 2D coordinates of each joint It is calculated using the following formula:

[0043] in, Indicates the first Frame number The original coordinates of each joint in the image coordinate system Indicates the first The original coordinates of the root joint of each frame. After processing with the above formula, the coordinates of the root joint of each frame are all returned to zero, eliminating the influence of global displacement.

[0044] Step 1.3: To enhance the model's generalization ability and robustness, online augmentation of the training set data is performed only during training. Augmentation strategies include: Horizontal Flip: The 2D input keypoint coordinate sequence and 3D labels are horizontally flipped with a probability of 0.5, and the joint indices of the left and right sides are swapped; Noise Injection: Random Gaussian noise is added to the 2D keypoint coordinates to simulate detection jitter in real scenes; Masking Operation: Parts of joints or frames are randomly occluded, forcing the model to use context information to fill in the missing data.

[0045] Example 3 Step 2 is implemented in the following steps: Step 2.1, Linear Projection Layer: Receives the 2D human keypoint coordinate sequence from Step 1, performs linear projection, and outputs the initial feature tensor. The specific process is as follows: Step 2.1.1: Receive the 2D human keypoint coordinate sequence output from Step 1 (preprocessing) as the input for this step, denoted as... The dimension of the sequence is defined as ,in The number of frames in the time series. This represents the number of human joints, and 2 represents the two-dimensional Euclidean coordinates of each keypoint on the image plane. .

[0046] Step 2.1.2: Apply a linear projection layer to the input sequence. Perform a linear affine transformation to map it to a feature dimension of The feature space yields the initial feature tensor. As the input to the subsequent spatially adaptive symmetric graph convolutional sub-network, the specific calculation formula is as follows:

[0047] in, , This is the weight matrix. The bias vector is used; in this embodiment, the latent feature dimension is set to . ; Step 2.2, Spatial Adaptive Symmetric Graph Convolutional Sub-Network, receives the initial feature tensor output from Step 2.1. Input space adaptive symmetric graph convolutional subnetwork outputs feature tensor The specific process is as follows: In this embodiment, the initial feature tensor output in step 2.1 is... (dimension is) Input space adaptive symmetric graph convolutional subnetwork, such as Figure 2 As shown. The specific sub-steps are as follows: Step 2.2.1, Adaptive adjacency matrix generation: First, the network initializes a learnable joint embedding matrix. Each row represents a 128-dimensional semantic vector of a human joint, and the matrix... It is independent of the input feature tensor The globally learnable parameters do not change with variations in the input video content. The original adjacency matrix is ​​calculated via matrix multiplication. The formula is as follows:

[0048] get The original adjacency matrix A. Subsequently, the following is introduced. identity matrix An enhanced adjacency matrix is ​​obtained to preserve the feature information of the joint itself.

[0049] Step 2.2.2, Algebraic normalization: Based on the original adjacency matrix obtained in step 2.2.1 Introducing the identity matrix (dimension is) The enhanced adjacency matrix is ​​calculated. The calculation formula is:

[0050] Subsequently, the degree matrix is ​​calculated. In order to avoid due to The presence of negative weights in the matrix results in a summation of zero; therefore, a small threshold is introduced. Truncation, degree matrix The diagonal elements are calculated using the following formula:

[0051] in, Degree matrix The Middle Line number The diagonal elements of the column; In the enhanced adjacency matrix, the first... Line number The element values ​​of the column; The small constant representing the prevention of the denominator being zero is set as follows in this invention. ; This indicates taking the maximum of the two values ​​within the parentheses.

[0052] Finally, symmetric normalization is performed to obtain the final normalized support matrix. The calculation formula is as follows:

[0053] in, Degree matrix The inverse square root matrix.

[0054] Step 2.2.3, Construction of multi-scale graph propagation operator: The order of the Chebyshev polynomial is set as follows: . use Chebyshev polynomials recursively generate graph propagation operators at three different scales. :

[0055] in, ( Aggregate its own node information, ( Aggregate information from directly adjacent joints. Aggregate the next nearest neighbor joint information. Stack these three operators to obtain the multi-scale support tensor.

[0056] Step 2.2.4, Node-Adaptive Graph Filtering and Feature Aggregation: First, we construct two globally shared learnable parameter libraries: a weighted basis tensor. (Corresponding dimension: Embedded dimension) × Scale × Number of input channels × Number of output channels and bias basis matrix Using the embedding vectors of the 17 joints defined in step 2.2.1 As modulation coefficients, they are used to synthesize node-specific filters; Receive the initial feature tensor output in step 2.1 And expand it along the time dimension, let... Indicates the first time steps ( The feature matrix of the spatial adaptive symmetric graph convolutional subnetwork is fused with the spatial topology operator generated in step 2.2.3. The node features are updated according to the following formula, along with the dynamically synthesized node-specific weights:

[0057] in, It is a time step Updated features; Represents the current frame in the 1st position. Aggregated features at various scales, aggregated features after graph operator diffusion; Item Indicates by along the embedding dimension The set of node-specific weight matrices generated by tensor contraction; operator Represents node-wise matrix multiplication: for each node The aggregated feature vector Only with the set of node-specific weight matrices Extracted from the first Each dedicated weight matrix slice is linearly transformed and multiplied to achieve differential feature extraction for different joints; Represents the embedding matrix With global bias base Node-specific biases for synthesis.

[0058] Input features It will evolve dynamically over time, but the structural parameter library ( , ) and node embedding Maintain sharing throughout the entire time series.

[0059] Finally, after independently applying a spatially adaptive symmetric graph convolutional subnetwork to each frame of the input sequence, the output results are combined along the time dimension to obtain the spatially augmented sequence tensor:

[0060] This feature tensor It will then be sent to the spatiotemporal location encoding module.

[0061] Step 2.3, the spatiotemporal location coding module receives the spatial enhancement features output from step 2.2. Spatiotemporal sensing features are output through the spatiotemporal location coding module. The specific process is as follows: Step 2.3.1, Initialize the learnable location embedding: Construct two independent sets of learnable parameter matrices: spatial location embedding and time location embedding .

[0062] Step 2.3.2, Broadcast Injection and Feature Overlay: Receive the spatial augmentation features output from Step 2.2. Its dimensions are Using broadcast mechanism to Copy and expand 243 times along the timeline; Copy and expand 243 times along the time axis, each time resulting in a shape of tensor; Subsequently, the expanded position embedding is used in conjunction with the input features. By adding elements one by one, we can obtain the position-aware features. :

[0063] Step 2.3.3, Layer Normalization and Output: Features after superposition Layer normalization is performed using the following formula:

[0064] Finally, spatiotemporal perception features are obtained. This serves as the input for subsequent time-series Mamba subnetworks.

[0065] Step 2.4, Temporal Mamba sub-network, uses the spatiotemporal awareness features output from Step 2.3. Input to by N (In this embodiment) N In a temporal Mamba subnetwork consisting of 10 temporal Mamba blocks, the deep temporal feature tensor is output. The specific process is as follows: Receive the spatiotemporal awareness features output from step 2.3 (dimension) Input a temporal Mamba subnetwork. To focus on uncovering the temporal evolution of individual joints, the dimension of the feature tensor is reshaped to... Input to 10 layers ( In the subnet of time-series Mamba block stacking.

[0066] Step 2.4.1, Parameter Discretization and Content Adaptation: Within each temporal Mamba block, parameter generation is first performed for each time step in the input keypoint coordinate sequence. ( ), receive the input features of the current frame Using a linear projection layer according to Dynamically generate time scale parameters and input projection matrix and output projection matrix Based on the zero-order preservation principle, the continuous state transition matrix is... Discretization, the calculation formula is as follows:

[0067]

[0068] (16) (17) in, For the input feature vector, Represents a learnable linear projection function; Represents globally learnable scalar baseline parameters; This indicates tensor dimension expansion; Represents a smooth, non-linear activation function; Indicates time step The dynamic time scale parameters; This represents the globally shared and input-independent state transition matrix in a continuous-time state-space model. , They represent the input respectively Dynamically generated continuous-time input and output projection matrices; This represents the state transition matrix obtained after zero-order preserved discretization; This represents the input projection matrix obtained after discretization; Represents the identity matrix; This represents the matrix exponential function.

[0069] Step 2.4.2, Bidirectional Temporal SSM Module State Update: Each bidirectional temporal SSM module executes a bidirectional scanning strategy: Forward Scan: From frame 1 to frame 243, using the state equation Recursively update the latent state and utilize the output equation. The first step is to capture the causal dependencies of motion; the second step is to perform a backward scan: from frame 243 to frame 1, the same recursive operation is performed to capture the future expectations and anti-causal dependencies of motion. Indicates time step The potential state vector; This represents the latent state vector of the previous time step. The recursive process is accelerated in hardware implementation using a parallel scanning algorithm, achieving linear computational complexity. After scanning, the forward and backward output features are added and fused to obtain the global temporal feature response of the current Mamba block, which is then fed into the subsequent gating mechanism and residual network.

[0070] Step 2.4.3, the complete computation process of a time-series Mamba block: The complete forward propagation of each temporal Mamba block includes local feature enhancement, bidirectional state-space modeling (SSM) modeling, and nonlinear transformation. The specific process is as follows: With the first Taking a time-series Mamba block as an example, first input the first... Timing Mamba block output characteristics After layer normalization, the model is activated by depthwise separable convolution (DSConv) and SiLU. Then, it enters the bidirectional temporal SSM module described in step 2.4.2 above for global temporal modeling. Subsequently, intermediate features are obtained through residual connections and MLP (multilayer perceptron) layers. ,at last After layer normalization and MLP layers, residual connections are added to obtain the output features of this time-series Mamba block. The mathematical expression is as follows:

[0071] in, Represents SiLU activation. This represents a bidirectional temporal SSM module, which, after being processed by stacking 10 layers of temporal Mamba blocks, outputs a deep temporal feature tensor.

[0072] Step 2.5, Token Pruning-Recovery Module: The pruning module receives the output characteristics of the Layer 4 time-series Mamba block. Output sparse feature tensors after token pruning The recovery module receives the sparse characteristics of the last layer output of the temporal Mamba subnetwork. After token recovery, output The specific process is as follows: A token pruning and recovery strategy is introduced in the time-series Mamba subnet. Specifically, the pruning module is scheduled to execute before the 5th time-series Mamba block, and the number of tokens to be retained is set to [value missing]. frame.

[0073] Step 2.5.1, Token Pruning: Features of the output of the fourth-level temporal Mamba block Before entering the next-level temporal Mamba block, the feature tensor is pruned, and the output features of the fourth-level temporal Mamba block are received. Along the joint dimension Adaptive average pooling is performed to obtain frame-level feature sequences. ; Subsequently, a density-based clustering algorithm (DPC-kNN) was used to select the most representative keyframes. For the first keyframe in the sequence... Frame, calculate its decision score: Local density ( ): Calculate the first Frame and The average distance between the nearest neighbor frames in the feature space is converted into density values ​​using a Gaussian kernel function: (20) This indicates that in the feature space, the distance to the th The nearest frame A set of indexes for each frame; This represents the index of a nearest neighbor frame in the set.

[0074] Separation distance ( ): Calculate the minimum distance from this frame to any frame with a higher density: (twenty one) Describe a conditional selection set, that is, a set whose local density is strictly greater than a certain value. Iterate through the candidate frames.

[0075] Screening decision: Calculate the overall score The 81 frames with the highest scores are selected as retention tokens, and the remaining 162 redundant frames are pruned and removed. Based on the temporal index of the retention tokens, the corresponding features are extracted from the input tensor, and the pruned sparse feature tensor is output. This sparse feature is then fed into subsequent temporal Mamba blocks from layer 5 to layer 10 for computation.

[0076] Step 2.5.2, Token Recovery: Receive the sparse characteristics of the last layer's time-series Mamba block output. To restore the original 243 frame resolution, a cross-attention mechanism is used: Initialize query: Define a set of learnable recovery tokens of length 243. And inject position encoding; Context aggregation: For queries, with sparse features These are the key and value. Through multi-head cross-attention, the full-length sequence features are reconstructed using sparse keyframe features. Details:

[0077] in, This represents the feature dimension of each attention head in a multi-head attention mechanism; This represents the normalization exponential function operation. The final result is a feature tensor containing complete temporal information from all 243 frames. .

[0078] Step 2.6, Regression Output: Receive the full-length temporal features recovered in Step 2.5. 3D human pose sequence is output through the regression head. The specific process is as follows: Receive the full-length timing features recovered in step 2.5 First, regarding the full-length time series characteristics... Perform layer normalization; then, pass a linear fully connected layer to convert the feature dimension from the latent dimension. Project back into 3D coordinates (i.e., 3D). The final output is the predicted 3D human pose sequence. It includes 17 joints in 243 frames. The predicted coordinates.

[0079] Example 4 Step 3 is implemented according to the following steps: Step 3.1, forward propagation and overall loss calculation: The 2D keypoint sequences from the training set are input into the model to obtain the predicted 3D pose sequences. And obtain the corresponding 3D pose ground truth value. To ensure both positioning accuracy and geometric consistency with the human body structure, this invention selects three key loss terms to construct a comprehensive loss function. The overall loss function L is defined as:

[0080] In this embodiment, the weighting coefficients for each loss term are configured as follows: , , .

[0081] The specific definitions and calculation processes for each item of loss are as follows: (1) Average joint position error loss ( This is the most basic regression objective, used to measure the absolute distance between the predicted and true coordinates in Euclidean space. The calculation formula is as follows:

[0082] in, For the number of frames in the sequence, For the number of joints, This represents the L2 norm.

[0083] (2) Normalized scaling loss ( To eliminate global scale differences in the human skeleton under different movements and focus on the recovery of pose structure, this loss is introduced. Its calculation process is as follows: First, the optimal scalar scaling factor for each frame is calculated using the least squares method. This minimizes the distance between the scaled predicted pose and the true pose, and then the minimum distance is calculated:

[0084] (3) Weighted position error loss ( Considering the large range of motion and high prediction difficulty of the human extremities, while the trunk is relatively stable, a joint weight vector is introduced to enhance the learning of difficult-to-learn joint samples. The calculation formula is as follows:

[0085] In this embodiment, based on human anatomy, 17 joints are assigned different weight values. This strengthens the focus on difficult samples. Specifically, the left elbow, left wrist, right elbow, and right wrist are assigned the highest weights. The right knee, right ankle, left knee, left ankle, neck, and head are assigned the second highest weight. Weights are assigned to the pelvis, right hip, left hip, spine, and left and right shoulders. .

[0086] Step 3.2: Calculate the error gradient direction and update the parameters using the Adam optimizer.

[0087] After calculating the comprehensive loss Then, its gradient with respect to the network parameters is calculated, and the model weights are updated using the optimizer. The specific training parameter settings in this embodiment are as follows: Hardware environment: Training was performed on a single NVIDIA RTX 5090 GPU.

[0088] Optimizer: The AdamW optimizer is used, with an initial learning rate of [value missing]. The weight decay coefficient is 0.01.

[0089] Learning rate scheduling: An exponential decay strategy is adopted, and the learning rate and decay factor are updated after each epoch. .

[0090] Training cycle: Batch size is set to 4, and total training rounds are set to 120 rounds.

[0091] Data augmentation: During training, horizontal flip augmentation is applied to the input data in real time to increase data diversity.

[0092] Example 5 This embodiment uses the Human3.6M dataset to train and evaluate the model, verifying the effectiveness of the method of the present invention in long sequence 3D pose estimation tasks.

[0093] 1. Dataset and Evaluation Metric: Human3.6M is currently the largest and most representative indoor human motion capture dataset. This example uses Subjects 1, 5, 6, 7, and 8 for training and Subjects 9 and 11 for testing. The evaluation metric is MPJPE (Mean Joint Position Error, in mm). This metric calculates the average Euclidean distance between the predicted joint coordinates and the actual joint coordinates after aligning the root node; a lower value indicates higher accuracy.

[0094] 2. Experimental setup: The length of the input sequence is set to... To accommodate different computing resource requirements, this embodiment constructs three model variants of different scales: Table 1: Model variants.

[0095]

[0096] in, Representing feature dimension, This indicates the number of keyframe tokens retained after the token pruning operation. This indicates the level index at which token pruning is performed (i.e., before which temporal Mamba block pruning is performed), Param indicates the number of model parameters, and MACs indicates the number of multiply-accumulate operations for processing a single input sequence.

[0097] 3. Experimental Results and Analysis: The method of this invention is compared with current mainstream Transformer methods (such as MotionBERT, MixSTE) and GCN methods (such as GLA-GCN), and the results are as follows: Table 2: Comparison of metrics between the method of this invention and existing methods on the Human 3.6M dataset.

[0098] Where CE represents the model output strategy. √ indicates the model only estimates the pose of the center frame of the input sequence; × indicates the model estimates the pose of all frames of the input sequence; MACs / frame represents the average computational cost per frame (M); P1↓ / P2↓: represent the evaluation metrics on the Human3.6M dataset. P1 (Protocol 1): Mean Joint Position Error (MPJPE), in millimeters (mm). A smaller value indicates higher accuracy (↓); P2 (Protocol 2): ​​Mean Joint Position Error after Prototype alignment (P-MPJPE), in millimeters (mm), used to evaluate the structural accuracy after eliminating rigidity transformations. A smaller value indicates higher accuracy (↓); P1*↓: represents the P1 error when using 2D ground truth as input, used to evaluate the theoretical upper limit accuracy of the model under ideal input conditions.

[0099] Accuracy Performance: our-B achieved an MPJPE of 39.3 mm with 243 frames of input. This result outperforms MixSTE (40.9 mm) and GLA-GCN (44.4 mm), and is roughly on par with the state-of-the-art model MotionBERT (39.2 mm) with its large number of parameters. our-L further reduced the error to 38.6 mm, setting a new accuracy record for this configuration and demonstrating the scalability of the architecture.

[0100] Efficiency Analysis: Parameters: our-B has only 9.4 M parameters, while MotionBERT with equivalent precision has as many as 42.3 M parameters. This invention uses only about 22% of the parameters. Computational Costs (MACs): our-B's single-frame inference computation cost is only 27 M. In contrast, MixSTE's single-frame computation cost is as high as 572 M. This means that the inference speed of this invention can theoretically be improved by more than 20 times.

[0101] Example 6 This embodiment uses the MPI-INF-3DHP dataset to train and evaluate the model, thereby verifying its generalization ability. This dataset contains more diverse scenes and significant variations in action amplitude, posing a crucial test to the model's robustness.

[0102] 1. Implementation details adjustments: Considering the short video clips and fast-paced action in this dataset, this embodiment adaptively adjusts the input sequence length. Specifically, the number of input frames is set to... Frames. Other parameters in the network structure remain consistent with the our-B and our-L variants. This demonstrates the flexibility of the method of the present invention at different temporal resolutions.

[0103] 2. Implementation details adjustments: On the MPI-INF-3DHP test set, the MPJPE metric was used to compare with state-of-the-art methods, and the results are as follows: Table 2: Comparison of metrics between the method of this invention and existing methods on the MPI-INF-3DHP dataset

[0104] our-B: Achieved an MPJPE of 15.0 mm, which is superior to many Transformer models with larger parameter sets (such as MotionAGFormer-B's 18.2 mm). our-L: Further reduced the error to 14.1 mm, setting a new record for this dataset and surpassing PoseMamba-B (14.5 mm) and MotionAGFormer-L (16.2 mm) of the same period.

[0105] Experimental results show that the proposed method not only performs excellently in controlled indoor environments (Human3.6M), but also possesses strong generalization ability in complex outdoor and variable lighting scenarios (MPI-INF-3DHP). Thanks to the spatially adaptive symmetric graph convolutional sub-network's adaptive learning of the skeleton structure and the temporal Mamba's robust capture of motion dynamics, the model can effectively resist the influence of background interference and unknown actions, generating stable and reliable 3D poses.

Claims

1. A 3D human pose estimation method based on adaptive graph convolution and state-space model, characterized by: Specifically, the steps include the following: Step 1: Obtain the human pose estimation dataset, process the training and test sets in the dataset, and perform online augmentation on the training set data; Step 2: Construct a 3D human pose estimation network model based on adaptive graph convolution and state space model; Step 3: Use the training set to perform end-to-end training on the model built in Step 2 to obtain the optimal model after training; Step 4: Use the test set to predict the optimal model trained in Step 3, and output a spatiotemporally consistent 3D human pose sequence.

2. The 3D human pose estimation method based on adaptive graph convolution and state-space model according to claim 1, characterized in that: The specific process of step 1 is as follows: Step 1.1: Extract video frame sequences from the human pose dataset. For both the training and test sets, use a 2D pose detector to obtain the coordinates of 2D human key points for each frame. As input, obtain the corresponding 3D human body key point coordinates. Labels are ground truth values; Step 1.2, for the 3D human body key point coordinate labels, denoted as the original coordinates. If the joint with index 0 is set as the root joint, then the first joint... The first frame Relative 3D coordinates of each joint It is calculated using the following formula: （1） in, Indicates the first Frame number The original absolute coordinates of each joint Indicates the first The original absolute coordinates of the frame root joint; For 2D human keypoint coordinate input, let the original coordinates be denoted as . , No. The first frame Relative 2D coordinates of each joint It is calculated using the following formula: （2） in, Indicates the first Frame number The original coordinates of each joint in the image coordinate system Indicates the first The original coordinates of the frame root joint; Step 1.3 involves performing online augmentation operations on the training data, including horizontal flipping, noise injection, and masking.

3. The 3D human pose estimation method based on adaptive graph convolution and state-space model according to claim 2, characterized in that: In step 2, the 3D human pose estimation network model based on adaptive graph convolution and state space model includes a linear projection layer, a spatial adaptive symmetric graph convolution sub-network, a spatiotemporal position coding module, a temporal Mamba sub-network, a token pruning-recovery module, and a regression head.

4. The 3D human pose estimation method based on adaptive graph convolution and state-space model according to claim 3, characterized in that: The specific process of step 2 is as follows: Step 2.1: After linear projection, the 2D human keypoint coordinate sequence output in Step 1 is used to output the initial feature tensor. ; Step 2.2, convert the initial feature tensor output in Step 2.1 into a single value. Input space adaptive symmetric graph convolutional subnetwork outputs feature tensor : Step 2.3, extract the spatial augmentation features output from Step 2.

2. Spatiotemporal sensing features are output through the spatiotemporal location coding module. ; Step 2.4: Develop the spatiotemporal awareness features output from Step 2.

3. Input to by N In a temporal Mamba subnetwork composed of temporal Mamba blocks, the deep temporal feature tensor is output. Step 2.5, in the token pruning-recovery module, the pruning module receives the output characteristics of the Layer 4 time-series Mamba block. Output sparse feature tensors after token pruning ; The recovery module receives the sparse characteristics of the last layer output of the temporal Mamba subnetwork. After token recovery, output ; Step 2.6: Recover the full-length temporal features from Step 2.

5. 3D human pose sequence is output through the regression head. .

5. The 3D human pose estimation method based on adaptive graph convolution and state-space model according to claim 4, characterized in that: The specific process of step 2.1 is as follows: Step 2.1.1: Take the 2D human keypoint coordinate sequence output from Step 1 as input, denoted as... The dimension of the sequence is defined as , where T represents the number of frames in the time series and J represents the number of human joints; Step 2.1.2: Apply a linear projection layer to the input sequence. Perform a linear affine transformation, mapping to a feature dimension of d. m The feature space yields the initial feature tensor. : in, , This is the weight matrix. This is the bias vector.

6. The 3D human pose estimation method based on adaptive graph convolution and state-space model according to claim 5, characterized in that: The specific process of step 2.2 is as follows: Step 2.2.1: Initialize a learnable joint embedding matrix for the network. The original adjacency matrix is ​​calculated by matrix multiplication. The formula is as follows: Step 2.2.2, based on the original adjacency matrix obtained in Step 2.2.1 Introducing the identity matrix The enhanced adjacency matrix is ​​calculated. The calculation formula is: （5） Degree matrix The diagonal elements are calculated using the following formula: in, Degree matrix The Middle Line 1 The diagonal elements of the column; In the enhanced adjacency matrix, the first... Line 1 The element values ​​of the column; The small constant representing the prevention of the denominator being zero is set as follows in this invention. ; This indicates that the value within the parentheses should be the larger of the two values. Finally, symmetric normalization is performed to obtain the final normalized support matrix. The calculation formula is as follows: in, Degree matrix The inverse square root matrix; Step 2.2.3, using Chebyshev polynomials recursively generate graph propagation operators at three different scales. : in, ( Aggregate its own node information, ( Aggregate information from directly adjacent joints. Aggregate information from next nearest neighbor joints; Step 2.2.4, convert the initial feature tensor output in step 2.1 into... Expanding along the time dimension, let's assume Indicates the first The feature matrices at each time step, and the spatial adaptive symmetric graph convolutional subnetwork, are fused with the spatial topology operators generated in step 2.2.

3. The node features are updated according to the following formula, along with the dynamically synthesized node-specific weights: After independently applying a spatially adaptive symmetric graph convolutional subnetwork to each frame of the input sequence, the outputs are combined along the time dimension to obtain the spatially augmented sequence tensor: in, It is a time step Updated features; Represents the current frame in the 1st position. Aggregation features at various scales; Indicates by along the embedding dimension The set of node-specific weight matrices generated by tensor contraction; operator This represents node-by-node matrix multiplication.

7. The 3D human pose estimation method based on adaptive graph convolution and state-space model according to claim 6, characterized in that: The specific process of step 2.3 is as follows: Step 2.3.1: Construct two independent sets of learnable parameter matrices: spatial location embedding. and time location embedding ; Step 2.3.2, Spatial Augmentation Features Based on the Output of Step 2.2 Using a broadcast mechanism to separately and Copy and expand along the time axis, then embed the expanded position into the input features. By adding elements one by one, we can obtain the position-aware features. : Step 2.3.3, location-aware features Layer normalization is performed using the following formula: Finally, spatiotemporal perception features are obtained. .

8. The 3D human pose estimation method based on adaptive graph convolution and state-space model according to claim 7, characterized in that: The specific process of step 2.4 is as follows: Step 2.4.1: Within each time-series Mamba block, parameter generation is first performed for each time step in the input keypoint coordinate sequence. Receive the input features of the current frame Using a linear projection layer according to Dynamically generate time scale parameters and input projection matrix and output projection matrix Based on the zero-order preservation principle, the continuous state transition matrix is... Discretization, the calculation formula is as follows: （16） （17） in, The input feature vector; Represents a learnable linear projection function; Represents globally learnable scalar baseline parameters; This indicates tensor dimension expansion; Represents a smooth, non-linear activation function; Indicates time step The dynamic time scale parameters; In a continuous-time state-space model, , They represent the input respectively Dynamically generated continuous-time input and output projection matrices; This represents the state transition matrix obtained after zero-order preserved discretization; This represents the input projection matrix obtained after discretization; Represents the identity matrix; Represents the matrix exponential function; Step 2.4.2, each bidirectional time SSM module internally executes a bidirectional scan strategy: Forward scan: utilizing state equations Recursively update the latent state and utilize the output equation. Capture the causal dependencies of motion; Backward scan: Perform the same recursive operations as the forward scan, where, Indicates time step The potential state vector; This represents the potential state vector of the previous time step; after the scan is completed, the forward scan output features and the backward scan output features are added and fused to obtain the global temporal feature response of the current Mamba block; Step 2.4.3, for the first A sequence of Mamba blocks, first input the first one. Temporal Mamba block output characteristics After layer normalization, depthwise separable convolution and SiLU activation are applied. Then, global temporal modeling is performed using the bidirectional temporal SSM module described in step 2.4.2 above. Subsequently, intermediate features are obtained through residual connections and MLP layers. ,at last After layer normalization and MLP layers, residual connections are added to obtain the output features of this time-series Mamba block. : in, Represents SiLU activation. This represents a bidirectional time-based SSM module.

9. The 3D human pose estimation method based on adaptive graph convolution and state-space model according to claim 8, characterized in that: The specific process of step 2.5 is as follows: Step 2.5.1, features output from the fourth-layer temporal Mamba block. Before entering the next-level temporal Mamba block, the feature tensor is pruned, and the output features of the fourth-level temporal Mamba block are received. Along the joint dimension Adaptive average pooling is performed to obtain frame-level feature sequences. ; Subsequently, a density-based clustering algorithm was used to filter keyframes, specifically the first keyframe in the sequence. Frame, calculate decision score: Local density : Calculate the first Frame and The average distance between the nearest neighbor frames in the feature space is converted into density values ​​using a Gaussian kernel function: in, In the feature space, the distance to the th The nearest frame A set of indexes for each frame; Represents the index of a nearest neighbor frame in the set; Separation distance : Calculate the minimum distance from this frame to any frame with a higher density than itself: Represents a set of criteria for selection; Selection decision: Calculate the overall score. Based on the time index of the retained token, extract the corresponding features from the input tensor and output the pruned sparse feature tensor. ; Step 2.5.2: Receive the sparse features of the output of the last layer of time-series Mamba blocks. It employs a cross-attention mechanism: Initialize query: Define a set of learnable recovery tokens And inject position encoding; Context aggregation: For querying, with sparse features Using keys and values, full-length sequence features are reconstructed through multi-head cross-attention, leveraging sparse keyframe features. Details: in, This represents the feature dimension of each attention head in a multi-head attention mechanism. This indicates the operation of the normalized exponential function.

10. The 3D human pose estimation method based on adaptive graph convolution and state-space model according to claim 9, characterized in that: The specific process of step 3 is as follows: The 2D keypoint sequences from the training set are input into the model to obtain the predicted 3D pose sequences. And obtain the corresponding 3D pose ground truth value. Three key loss terms are selected to construct a comprehensive loss function. The comprehensive loss function L is defined as: in, , , These are the weighting coefficients for each loss term; Mean joint position error loss The calculation formula is as follows: （24） Normalized scaling loss First, the optimal scalar scaling factor for each frame is calculated using the least squares method. This minimizes the distance between the scaled predicted pose and the true pose, and then the minimum distance is calculated: Weighted position error loss The calculation formula is as follows: （27） in, For the number of frames in the sequence, For the number of joints, This represents the L2 norm.

Images