3D human pose estimation method and system based on physical constraints and space-time decoupling

Abstract

This invention discloses a 3D human pose estimation method and system based on physical constraints and spatiotemporal decoupling. The method includes constructing a 3D human pose estimation model containing a spatiotemporally decoupled dual-branch encoder and training the model; constructing a multi-dimensional loss function and dynamically adjusting the weights of different dimensions during training to ensure the output pose conforms to physical laws; acquiring human motion image information, extracting 2D pose sequences, and preprocessing and encoding their spatiotemporal positions; inputting the 2D pose sequences into the trained model and extracting features from both temporal and spatial branches; fusing the temporal and spatial branch features through an adaptive fusion module using cross-modal attention, integrating the bidirectional attention results based on dynamic weights, and outputting fused features; and mapping the obtained fused features to 3D space through a linear decoding layer. This method achieves end-to-end physically reasonable pose generation, improving the robustness of pose estimation in complex scenes.

Description

Technical Field

[0001] This invention relates to the field of human pose estimation technology, and in particular to a 3D human pose estimation method and system based on physical constraints and spatiotemporal decoupling. Background Technology

[0002] In recent years, deep learning-based 3D human pose estimation technology has made significant progress, especially with the application of the Transformer architecture (such as MotionBERT and PoseFormer), which has improved pose estimation accuracy through powerful sequence modeling capabilities. However, existing technologies still have two major limitations that restrict their reliability in real-world scenarios: 1. Insufficient utilization of spatiotemporal modeling coupling and skeletal topology: Existing methods (such as MotionBERT's DSTformer and PoseFormer's pure temporal attention) mostly employ spatiotemporally coupled attention mechanisms, making it difficult to specifically mine the inherent hierarchical topology of the human skeleton (such as parent-child joint motion constraints and limb symmetry). For example, PoseFormer only focuses on inter-frame temporal dependencies, completely ignoring intra-frame joint spatial relationships; although MixSTE attempts to decouple joints from the temporal dimension, it does not design a dedicated module for skeletal topology, causing the model to repeatedly learn known anatomical priors from the data, increasing the learning difficulty and data requirements, and easily generating poses that do not conform to the skeletal structure in data-sparse regions.

[0003] 2. Lack of physical plausibility: Existing data-driven methods lack explicit modeling of the physical laws governing human movement, often resulting in physical anomalies in the generated postures. Inconsistent bone length: The length of the same bone (such as the humerus and femur) varies by more than 10% in different frames, which violates the fixed structure of human anatomy. Joint angle violations: knees bending backward (angle < 0°), elbows hyperextending (angle > 180°), etc., exceeding the anatomical range of motion; Dynamic anomaly: Center of gravity (COM) acceleration exceeds 0.5 m / s². 2 (Far exceeding the normal range of human movement), the foot sliding distance exceeds 5mm per frame; Temporal jitter: Joint position changes exceeding 8mm between adjacent frames result in a lack of smoothness in the motion trajectory. Although some studies have attempted to supplement physical constraints (such as adding bone length constraints to the loss function and post-processing correction of joint angles), these are all limited to the "loss function level" or "post-processing steps," failing to deeply embed physical constraints into the feature learning process. Furthermore, most studies target a single physical dimension (such as only bone length or only joint angles), lacking a systematic constraint framework from local to global perspectives. This makes it difficult to organically integrate with the Transformer architecture and fundamentally solve the problem of physical rationality. Summary of the Invention

[0004] In order to overcome the above-mentioned problems in the prior art, the present invention proposes a 3D human pose estimation method and system based on physical constraints and spatiotemporal decoupling.

[0005] The technical solution adopted by this invention to solve its technical problem is: a 3D human pose estimation method based on physical constraints and spatiotemporal decoupling, comprising the following steps: Step 1: Construct a 3D human pose estimation model that includes a spatiotemporally decoupled dual-branch encoder and train the model; construct a multi-dimensional loss function and dynamically adjust the weights of different dimensions of loss during training to ensure that the output pose conforms to physical laws. Step 2: Collect human movement image information, extract 2D pose sequences, and preprocess and encode them in spatiotemporal position. Step 3: Input the 2D pose sequence obtained in Step 2 into the 3D human pose estimation model trained in Step 1, and extract features for the temporal branch and the spatial branch respectively. Step 4: The temporal branch features and spatial branch features from Step 3 are fused across modal attention using an adaptive fusion module. The bidirectional attention results are integrated based on dynamic weights, and the fused features are output. Step 5: Map the fused features obtained in Step 3 to 3D space through a linear decoding layer; In the spatiotemporal decoupled dual-branch encoder, the temporal branch is used to identify periodic motion patterns and predict motion trends to ensure the smoothness of 3D pose in the time dimension. The spatial branch learns anatomically compliant spatial relationships through the skeletal topology perception module and the hierarchical physical constraint embedding module. The multi-dimensional loss function is specifically as follows: in , , and These represent the dynamic weighting coefficients for basic pose loss, bone length consistency loss, joint angle loss, and temporal smoothness loss, respectively. Indicates the basic attitude loss. This indicates loss of bone length consistency. This indicates a loss of joint angle. This represents the loss of temporal smoothness.

[0006] The aforementioned 3D human pose estimation method based on physical constraints and spatiotemporal decoupling, specifically step 2 of the preprocessing and spatiotemporal position encoding method, involves mapping low-dimensional 2D coordinates to a high-dimensional feature space through a linear projection layer, preserving the original motion information while expanding the representational capability. The formula is as follows: in, For learnable projection weights, , C is the bias term, C is the feature dimension, and X is the 2D pose sequence. T is the sequence length, and J is the number of joints; This is a 2D pose sequence after projection of high-dimensional features; Introducing learnable temporal position coding Spatial location coding Embedded representation is: .

[0007] In the aforementioned 3D human pose estimation method based on physical constraints and spatiotemporal decoupling, step 3 involves the skeletal topology perception module constructing joint features for all joint pairs. BF (Batch-Frame Product) learns different types of joint relationships through four independent encoders; it defines body part hierarchies and constructs a hierarchical relationship matrix. Encoding into hierarchical features through linear layers Incorporating multi-relational encoding results; by randomly initializing weights Weighted aggregation of relational features is performed, and then joint dependencies are captured using spatial multi-head attention with the original features and aggregated relational features. By concatenating the original features and attention features to generate a gating coefficient g, the original information and the aggregated context are balanced, as shown in the formula: in, For bias terms, For the gated layer activation function, Features generated for global awareness based on relational attention. This is the final fused feature output after gating and balancing. After layer normalization and residual connection, the skeletal topology perception module features are output. .

[0008] The aforementioned 3D human pose estimation method based on physical constraints and spatiotemporal decoupling includes a hierarchical physical constraint embedding module comprising a joint constraint unit, a limb coordination unit, and a whole-body consistency unit. The joint constraint unit constrains the range of motion of joints based on anatomical limitations, specifically by calculating angular deviations using parent and child joint vectors. Define the range of motion of the joint and calculate the constraint loss. : ,in This represents the lower anatomical limit of the range of motion of joint j. This represents the anatomical upper limit of the range of motion of joint j; Mapping angle deviation and constraint loss to feature adjustment amounts Spatial features are incorporated through linear layers: ,in, To enhance spatial features by incorporating joint constraint information, These are learnable mapping weights.

[0009] The aforementioned 3D human pose estimation method based on physical constraints and spatiotemporal decoupling ensures the coordination of joint movements within the same limb through a limb coordination module. Specifically, it groups the joints and achieves intra-limb joint interaction through grouped multi-head attention. Let |g| represent the joint characteristics of a certain limb, and |g| represent the number of joints in that limb. For limb aggregation features, B is the batch size, and C is the feature dimension; calculate the representative features of each limb. By capturing global coordination relationships through multi-head attention mechanisms between limbs, output .

[0010] The aforementioned 3D human pose estimation method based on physical constraints and spatiotemporal decoupling ensures that the overall motion conforms to dynamic laws through a whole-body consistency module, specifically by using human body mass distribution weights. Weighted calculation of centroid position: Through the loss function The focus of punishment shifted drastically: Where B is the batch size and F is the total number of frames. Let b be the centroid coordinates of the b-th sample in frame t. Let be the centroid coordinates of the b-th sample at frame t-1; Joint acceleration was calculated using second-order difference. The formula for punishing unreasonable acceleration is: .

[0011] In the aforementioned 3D human pose estimation method based on physical constraints and spatiotemporal decoupling, step 4 specifically comprises: Step 4.1: Project the bi-branch features to a unified space using a non-linear mapping to ensure fusion compatibility. Step 4.2: Generate global weights based on the overall characteristics of the sequence. The formula is: This is the result of global feature aggregation; Local weights are generated based on joint-level differences. To adapt to the spatiotemporal dependency requirements of different joints; Calculate the time-aware factor using time-series features. Adjust the temporal components of the local weights, and combine them with the joint importance factor. Integrating global and local weights: ,in The adjusted local weights, To incorporate dynamic weights; Step 4.3: Through cross-modal attention fusion, bidirectional interaction of spatiotemporal features is achieved. Based on the dynamic weights obtained in Step 4.2, the bidirectional attention results are integrated to output the fused features. .

[0012] The aforementioned 3D human pose estimation method based on physical constraints and spatiotemporal decoupling, the basic pose loss Specifically: ,in As the weight of key joints, For the true attitude coordinates, To predict attitude coordinates; Loss of bone length consistency Specifically: in This indicates the total number of bone connections. Indicates the training batch size. The number of frames in the input sequence. , Indicates the predicted 3D coordinates of the pose. , 3D coordinates representing the actual pose, subscript Index batches, time frames, parent joints, and child joints respectively; Joint angle loss Specifically: in, Let S be the total number of joint constraints, and S be the set of joint constraints. and These represent the lower and upper anatomical limits of a specific joint's range of motion, respectively. Indicates in batch and time Calculated actual joint angles Temporal smoothness loss Specifically: in, This represents the total number of joints in the human body. Representative batch ,time ,joint 3D coordinate position.

[0013] A 3D human pose estimation system based on physical constraints and spatiotemporal decoupling is used to implement the 3D human pose estimation method based on physical constraints and spatiotemporal decoupling as described above. It includes an input module, a preprocessing and embedding module, a spatiotemporal decoupling dual-branch encoder, an adaptive fusion module, an output module, and a physical perception loss module. The input module is used to input a 2D pose sequence, and the preprocessing and embedding module is used to perform high-dimensional feature projection and spatiotemporal position encoding on the 2D pose sequence. The spatiotemporal decoupled dual-branch encoder includes a temporal branch composed of a 5-layer Transformer encoder and a spatial branch composed of a skeletal topology perception module and a hierarchical physical constraint embedding module. The temporal branch captures long-term motion dependence and continuity through alternating spatiotemporal attention. The skeletal topology perception module avoids the limitations of a fixed adjacency matrix by implicitly learning the dynamic relationships between joints. The hierarchical physical constraint embedding module specifically includes a joint constraint unit, a limb coordination unit, and a whole-body consistency unit. The joint constraint unit constrains the range of motion of joints based on anatomical limitations. The limb coordination module ensures the coordination of joint movements of the same limb. The whole-body consistency module ensures that the overall motion conforms to the laws of dynamics. The adaptive fusion module is used to extract temporal and spatial features from the spatiotemporally decoupled dual-branch encoder and perform cross-modal attention fusion. The physical perception loss module ensures that the 3D posture output by the output module conforms to physical laws by dynamically adjusting the dynamic weight coefficients of the basic posture loss, bone length consistency loss, joint angle loss, and temporal smoothness loss.

[0014] The beneficial effects of this invention are as follows: it optimizes temporal dynamic dependencies and spatial skeletal topology relationships through a decoupled architecture, making full use of prior knowledge of human anatomy to reduce the difficulty of model learning; it constructs a multi-level constraint system from joints to the whole body, deeply embedding physical laws into the feature representation level to achieve end-to-end physically reasonable pose generation; it designs a dynamic weight fusion mechanism to balance temporal dynamic accuracy and spatial structural rationality, improving the robustness of pose estimation in complex scenarios (such as rapid movement and occlusion); and it establishes an evaluation system that includes bone length consistency, joint angle compliance, and dynamic smoothness to ensure the practicality and reliability of the technical solution. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall architecture of the present invention; Figure 2 This is a schematic diagram of the skeleton topology sensing module (STAM) structure of the present invention; Figure 3 This is a schematic diagram of the structure of the hierarchical physical constraint embedding module (PCEM) of the present invention; Figure 4 This is a schematic diagram of the adaptive fusion module (AFM) structure of the present invention. Detailed Implementation

[0016] To enable those skilled in the art to better understand the technical solution of the present invention, the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0017] This invention proposes the PhysMoBERT (Physics-Guided MotionBERT) framework, such as... Figure 1 As shown, through four core modules—"spatiotemporal decoupling dual-branch encoder + hierarchical physical constraint embedding + adaptive fusion + physical perception loss"—the synergistic improvement of accuracy and physical rationality is achieved. The specific technical solution is as follows: I. Overall Architecture Design PhysMoBERT adopts an encoder-decoder paradigm. The overall process is as follows: 2D pose sequence input → preprocessing and embedding → dual-branch feature extraction → adaptive fusion → 3D pose output. Each module works together to achieve physically reasonable 3D pose estimation.

[0018] II. Detailed Design of Core Modules 1. Input preprocessing and embedding module for input 2D pose sequences (T is the sequence length, J is the number of joints, default J=17), the preprocessing steps are as follows: (1) High-dimensional feature projection: The low-dimensional 2D coordinates are mapped to the high-dimensional feature space through a linear projection layer, which preserves the original motion information while expanding the representation capability. The formula is as follows: in (C=512 is the feature dimension) represents the learnable projective weights. This is a bias term.

[0019] (2) Spatiotemporal location coding: Introducing learnable temporal location coding (Capturing inter-frame temporal relationships) and spatial location coding (Encoding joint topology), the final embedding representation is: This step provides initial features, including spatiotemporal priors, for the two-branch processing.

[0020] 2. The spatiotemporally decoupled dual-branch encoder decouples the traditional coupled architecture into two parallel branches, focusing on temporal dynamics and spatial topology learning respectively: (1) Temporal Branch: Based on the DSTformer architecture of MotionBERT, it retains the advantages of temporal modeling and captures long-term motion dependencies and continuity through alternating spatiotemporal attention. Structure: 5-layer Transformer encoder, each layer contains multi-head attention (8 attention heads), layer normalization, GELU activation and residual connections; Core functions: Identify periodic movement patterns (such as the limb swing cycle during walking), predict movement trends, ensure the smoothness of 3D pose over time, and output temporal features.

[0021] (2) SpatialBranch: A newly designed spatial feature extraction module learns anatomically consistent spatial relationships through Skeletal Topology Awareness (STAM) and Physical Constraint Embedding (PCEM). ① Skeletal Topology Awareness Module (STAM): Implicitly learns the dynamic relationships between joints, avoiding the limitations of a fixed adjacency matrix, such as... Figure 2 As shown.

[0022] Multi-relation encoding: Constructing joint features for all joint pairs BF (Batch-Frame Product) learns different types of joint relationships (such as parent-child joints, symmetrical joints) through four independent encoders. The formula is: Hierarchical Prior Enhancement: Based on the H36M dataset, a hierarchical relationship matrix is ​​constructed by defining body parts (trunk, head, limbs) using 17 joints. (Joints of the same limb have a weight of 1.0, joints of the trunk have a weight of 0.5, and the rest have a weight of 0), encoded into hierarchical features through a linear layer. Incorporating multi-relational coding results; Adaptive spatial aggregation: by randomly initializing weights (Adaptive optimization during training) Weighted aggregation of four relation features is performed, and then the original features are used as the Query and the aggregated relation features are used as the Key / Value. Joint dependencies are captured through spatial multi-head attention, as shown in the formula: ( (Result of sequence format conversion for aggregation features). Gated fusion: By concatenating the original features and attention features, a gating coefficient g is generated to balance the original information and the aggregated context. The formula is as follows: Finally, after layer normalization and residual connection, the STAM feature is output. .

[0023] ② Hierarchical Physical Constraint Embedding Module (PCEM): Embeds physical constraints at three levels: joints, limbs, and the whole body. The structure is as follows: Figure 3 As shown.

[0024] Joint Constraint Module (JCM): Based on anatomical limitations, it constrains the range of motion of joints. (a) Joint angle calculation: using parent-child joint vectors (Normalization eliminates scale effects), calculate angular deviation. ; (b) Anatomical Constraint Modeling: Defining the range of motion of joints (e.g., elbow joint) Calculate the constraint loss: (c) Feature layer embedding: Mapping angle deviation and loss to feature adjustment amount Spatial features are incorporated through linear layers, as shown in the formula: Limb coordination unit (LCM): Ensures the coordination of joint movements within the same limb. (a) Limb grouping: The 17 joints are divided into 6 groups (left arm, right arm, left leg, right leg, trunk, head); (b) Intralimb coordination: Achieving joint interaction within the same limb through grouped multi-head attention (MHA), the formula is as follows: ( (where |g| represents the joint characteristics of a limb, and |g| represents the number of joints in that limb). (c) Inter-limb interaction: Calculate the representative features of each limb By capturing global coordination relationships through interlimb MHA, output .

[0025] Body Coherence Module (BCM): Ensures that overall movement conforms to the laws of dynamics. (a) Center of gravity (COM) calculation: based on human body mass distribution weights (Predefined physiological proportions, such as pelvic weight 0.678, head weight 0.1), weighted calculation of center of gravity position: (b) Center of gravity stability constraint: Penalizes drastic changes in the center of gravity; the loss formula is: (c) Dynamic constraints: Joint acceleration is calculated using second-order difference. The formula for punishing unreasonable acceleration is: 3. The Adaptive Fusion Module (AFM) dynamically balances the feature contributions of the two branches. The structure of the Adaptive Fusion Module is as follows: Figure 4 As shown. The steps are as follows: (1) Feature alignment: The bi-branch features are projected onto a unified space through non-linear mapping to ensure fusion compatibility. Dynamic weight generation: Global weights: Global weights are generated based on the overall characteristics of the sequence (such as motion speed). The formula is: ( (This is the result of global feature aggregation). Local weights: Generating local weights based on joint-level differences To adapt to the spatiotemporal dependency requirements of different joints; Time-aware conditioning: Calculating time-aware factors based on temporal characteristics Adjust the temporal components of the local weights, and combine them with the joint importance factor. Integrating global and local weights: Cross-modal attention fusion: Achieving bidirectional interaction of spatiotemporal features. Temporal to Spatial Attention: Temporal features are used to query spatial information, with the following formula: Spatial-to-temporal attention: Spatial features are used to query temporal information, the formula is: Final fusion: Based on dynamic weights, the bidirectional attention results are integrated, and the fused features are output via MLP and residual connections. .

[0026] 4. Output Decoding and Physical Perception Loss (1) 3D pose output: The fused features are mapped back to 3D space through a linear decoding layer, as shown in the formula: (for bias).

[0027] (2) Physical perception loss function: Construct a multi-dimensional loss system to balance accuracy and physical rationality: Base pose loss (weighted MPJPE): assigns higher weights to key joints (such as the pelvis and spine). The formula is: Physics Constraint Loss: In PhysMoBERT, the physics-aware loss function is a key component in ensuring that the generated pose conforms to physical laws. The bone length consistency loss aims to constrain the bone length of the predicted pose to be consistent with the actual bone length; its mathematical expression is a specific set of bone pairs. The average of the sum of the absolute values ​​of the length differences of all bones in the middle. In this formula, Represents the total number of skeletal connections. Indicates the training batch size. The number of frames in the input sequence, two Represent the 3D coordinates of the predicted pose and the true pose, respectively, with subscripts... The batch, time frame, parent joint, and child joint are indexed separately, and the L2 norm is used to calculate the Euclidean length of the bone vector.

[0028] Joint angle loss is based on strict prior knowledge of human anatomy, ensuring that the range of motion of each joint is within physiologically permissible limits. The loss function is calculated by summing the constraints in all predefined joint constraint sets C, where... and These represent the lower and upper anatomical limits of a specific joint's range of motion, respectively. Indicates in batch and time The calculated actual joint angles. When the joint angles exceed the reasonable range... The function will impose a penalty, thereby guiding the model to generate motions that conform to biomechanical principles.

[0029] Temporal smoothness loss focuses on ensuring the continuity of generated motion in the time dimension, and achieves motion smoothness constraints by calculating the L2 norm of joint position acceleration. In this formula, This represents the total number of joints in the human body. Representative batch ,time ,joint The 3D coordinate position. This loss function is essentially a second-order differential constraint on the motion trajectory. It suppresses jitter in the generated sequence by penalizing unreasonable acceleration changes, making the motion trajectory more natural and smooth.

[0030] The total loss function organically integrates the above losses through an adaptive weight adjustment mechanism, forming a complete optimization objective system.

[0031] in , , and These represent dynamic weight coefficients for the base pose loss, bone length consistency loss, joint angle loss, and temporal smoothness loss, respectively. These weights are intelligently adjusted during training using a course learning strategy, enabling the model to automatically balance the influence of various constraints according to the characteristics of different training stages, thereby ensuring the physical plausibility of the generated poses while maintaining high accuracy.

[0032] III. The training strategy design employs a three-stage progressive training approach (50 rounds in total) to ensure training stability and optimal performance: 1. Basic Phase (Rounds 0-10): Primarily based on MPJPE losses ( =1.0), with low physical constraint weights ( =0.05), prioritize learning basic pose mapping; 2. Constraint Introduction Phase (10-25 rounds): Gradually increase the weight of physical constraints to 0.2, balancing accuracy and physical rationality; 3. Fine-tuning phase (rounds 25-50): Physical constraint weights are increased to 0.5, AdamW optimizer is used (initial learning rate 1e-4, cosine annealing scheduling), weight decay is 1e-4, and gradient clipping norm is 1.0.

[0033] This document details the specific implementation process of PhysMoBERT, including hardware environment, data preparation, model training, performance verification, and deployment, ensuring the repeatability and practicality of the technical solution.

[0034] I. Hardware and Software Environment Preparation 1. Hardware Configuration Training equipment: 4 servers equipped with NVIDIA RTX 3090 GPUs (24GB of video memory per card) to support distributed parallel training; each server is equipped with 128GB of DDR4 memory (3200MHz) and 2TB SSD solid-state drive (read / write speed 3500MB / s+) to ensure efficient data loading and parameter exchange.

[0035] Inference testing equipment: Server-side: NVIDIA RTX 3060 GPU (12GB VRAM), Intel i9-11900K CPU (8 cores, 16 threads, 3.5GHz). Edge devices: NVIDIA Jetson AGX Xavier (GPU computing power 32TOPS), Intel NUC 12 Pro (i7-1260P CPU); Storage device: 8TB HDD (for backing up datasets and experiment logs) to prevent data loss.

[0036] 2. Software Environment Operating system: Ubuntu 20.04LTS (server / edge device), kernel version 5.4.0-150-generic; Deep learning framework: PyTorch 2.0.1, with CUDA 11.8 and CuDNN 8.6; Data processing: OpenCV 4.8.0 (image reading), NumPy 1.24.3 (numerical computation), Pandas 2.0.3 (experimental data statistics); Model building: torchvision 0.15.2 (Transformer module), einops 0.6.1 (tensor operations); Inference acceleration: TensorRT 8.6 (GPU acceleration), ONNX Runtime 1.15 (CPU inference); Visualization: Matplotlib 3.7.1 (experimental plotting), TensorBoard 2.13.0 (training monitoring).

[0037] II. Dataset Preparation and Preprocessing 1. Core dataset: Human 3.6M Dataset specifications: Contains 15 categories of daily activities (such as Walking, Eating, Phoning) for 7 actors, totaling 3.6 million frames, providing high-precision 3D Motion Capture annotations (17 joints). Partitioning method: According to the standard protocol, Subjects1, 5, 6, 7, and 8 are the training set (approximately 2.8 million frames), and Subjects9 and 11 are the test set (approximately 800,000 frames). Preprocessing steps: (1) 2D pose extraction: 2D joint coordinates are extracted from the original video using a pre-trained HRNet model to ensure the output... (T=243 frames, default sequence length); (2) Coordinate normalization: Map the joint coordinates to the range [0,1], the formula is as follows Eliminate scale differences; (3) Data augmentation: Temporal enhancement: Randomly trim the sequence length (180-243 frames) and shuffle the frame interval (±2 frames) to simulate changes in motion speed; Spatial enhancement: random horizontal flipping (probability 0.5, synchronous adjustment of joint x-coordinate) and Gaussian noise addition (standard deviation 0.01) to enhance robustness; (4) Format conversion: The processed data is stored as a TFRecord file in the format of "frame index-joint ID-x coordinate-y coordinate" to accelerate training reading.

[0038] 2. Auxiliary Validation Dataset - MPI-INF-3DHP: Contains 8 scene categories (indoor and outdoor mixed), with 150,000 training frames and 50,000 test frames, used to evaluate cross-scene generalization ability; the preprocessing steps are the same as Human3.6M, only the sequence length is adjusted to 180 frames.

[0039] 3DPW: Outdoor scene dataset (60 outdoor sequences), including complex backgrounds and occlusions, with 40,000 frames in the training set and 20,000 frames in the test set; during preprocessing, "illuminance normalization" (adjusting brightness to [0.8, 1.2] times) is added to adapt to changes in outdoor lighting.

[0040] III. Model Construction and Training Implementation 1. Model parameter configuration Embedding module: feature dimension C=512, temporal / spatial encoding dimension 512, linear projection weight initialization (He normal distribution).

[0041] Temporal branch: 5 Transformer layers, 8 attention heads, hidden layer dimension 512, dropout=0.1.

[0042] STAM module: 4-way relation encoder, hierarchical matrix weights (1.0 for the same limb, 0.5 for trunk association), gating layer activation function Sigmoid.

[0043] PCEM module: JCM joint range (elbow [0°, 150°], knee [0°, 140°]), BCM quality weights (Predefined).

[0044] AFM module: Global weights, MLP layer number 2, Local weights, Joint importance factor. =0.3.

[0045] Decoding layer: Linear layer output dimension 3, bias initialized (constant 0).

[0046] 2. The training process adopts a three-stage progressive training approach, totaling 50 rounds, with each round training 2000 batches (batchsize=32): Phase 1: Basic Posture Learning (0-10 rounds) Optimization objective: Prioritize learning the basic 2D to 3D mapping, and set the loss weights accordingly. Learning rate scheduling: The initial learning rate is 1e-4, which is linearly increased to 1e-4 in the first 5 rounds, and then kept constant in the last 5 rounds; Monitoring metrics: Print the MPJPE loss every 50 batches, evaluate it on the validation set every 2 rounds, and save "best_mpjpe_model.pth".

[0047] Phase 2: Introduction of Physical Constraints (Rounds 11-25) Optimization goal: Gradually increase physical constraints, increasing the physical weight by 0.03 every 3 rounds (finally). =0.2); Regularization strategies: Enable weight decay (1e-4) and gradient clipping (norm 1.0) to prevent overfitting; Dynamic adjustment: If physical losses occur in a certain round ( The weight increase exceeded 10%, so the weight increase was paused and training was stabilized.

[0048] Phase 3: Fine-tuning (26-50 rounds) Optimization goal: Balance accuracy and physical plausibility, increase the physical weight to 0.5, and enable adaptive weight learning. Learning rate scheduling: Cosine annealing is used to reduce the learning rate from 1e-4 to 1e-5; Model saving: The model is saved every 5 rounds. After training, the model with the lowest MPJPE on the validation set and the lowest physical loss is selected as the final model ("physmobet_best.pth").

[0049] 3. Training Monitoring and Log Recording - Real-time Monitoring: Record the following metrics using TensorBoard: Training metrics: MPJPE per round, physical loss, and learning rate change curve; Validation metrics: MPJPE test set, P-MPJPE, bone length error, angle violation rate; Log storage: Experimental data is saved in the format of "date_round_metric.csv", which includes parameter update records for each round and hardware resource usage (GPU memory, CPU utilization) to facilitate reproduction and debugging.

[0050] IV. Model Deployment and Application Implementation 1. Server-side deployment (taking sports motion analysis as an example) - Deployment goal: Real-time processing of athlete training videos (1080p, 30fps), outputting physically accurate 3D poses for motion correction; Deployment process: 1. Model Conversion: Convert the PyTorch model to ONNX format (input dimensions 1×243×17×2), and then convert it to an FP16 precision engine file (“physmobet_trt_fp16.engine”) using TensorRT, which improves inference speed by 2 times.

[0051] 2. Video processing pipeline: Video reading: OpenCV is used to read the RTSP stream (real-time camera streaming), and 2D pose is extracted for each frame (HRNet model, time <10ms). Sequence buffer: Buffer 243 frames of 2D pose to form the input sequence X; Model inference: The TensorRT engine is called, and the inference time per frame is <30ms (GPU side), outputting 3D pose Y; Results analysis: Calculate the athlete's joint angles (e.g., whether the knee joint angle is within [0°, 140°]) and bone length changes (e.g., femur length fluctuation <5%), and generate a movement assessment report.

[0052] 3. Visualization output: 3D pose skeleton (drawn using Matplotlib) is overlaid on the original video and streamed to the monitoring screen in real time with a latency of <100ms.

[0053] Deployment results: Supports simultaneous processing of 4 video feeds, with motion analysis accuracy >92%, and can accurately identify problems such as "excessive knee flexion" and "abnormal arm movement trajectory".

[0054] 2. Edge device deployment (taking medical rehabilitation assessment as an example) Deployment device: NVIDIA Jetson AGXXavier (edge ​​GPU device); Optimization measures: 1. Lightweight Model: Channel pruning was performed on the PCEM module (the number of channels was reduced from 512 to 256), reducing parameters by 40% and increasing inference speed to 15 FPS; 2. Quantization processing: INT8 quantization (TensorRT tool) was used, reducing the model size from 88MB to 22MB, making it suitable for edge device storage.

[0055] Application process: 1. Patient motion capture: Use a tablet camera to record the patient's rehabilitation movements (such as elbow flexion and extension) and extract 2D sequences (T=180 frames). 2.3D pose estimation: Edge device inference generates 3D pose and calculates joint range of motion (e.g., elbow flexion-extension angle difference <3°, meeting the accuracy requirements for medical assessment). 3. Rehabilitation Report Generation: By comparing the patient's movements with the standard rehabilitation movements (MPJPE <5mm is acceptable), a rehabilitation progress chart is automatically generated and uploaded to the hospital system simultaneously. - Deployment Results: Single device power consumption <15W, continuous operation >8 hours, meeting the portability and battery life requirements of outpatient rehabilitation assessment.

[0056] V. Implementation Precautions and Troubleshooting 1. Training Notes - Gradient Explosion Handling: If the gradient norm > 1.0 during training, automatically reduce the learning rate (×0.5) and restart the current round; Data imbalance handling: Oversampling (sampling rate × 1.5) is used for action categories (such as SittingDown, which has fewer samples) to avoid the model being biased towards the majority class.

[0057] Checkpoint management: Save intermediate models every 5 rounds. If training is interrupted, it can be restored from the most recent checkpoint (loading parameters, learning rate, optimizer state).

[0058] 2. Reasoning Fault Handling High inference latency: Check if the TensorRT engine is generating correctly. If the CPU inference latency is >50ms, enable MKL-DNN acceleration in ONNXRuntime.

[0059] Anomaly output: If the joint angle violation rate is >10%, reload the pre-trained weights and check whether the input 2D pose extraction is accurate.

[0060] Hardware compatibility issues: When deploying edge devices, if there is a CUDA version mismatch, use Docker containerization for deployment (with CUDA 11.8 pre-installed).

[0061] Compared with existing 3D human pose estimation techniques, the PhysMoBERT framework proposed in this application has the following significant advantages, all of which are verified based on experimental data in the technical documentation. As shown in Table 1, Table 1 presents a quantitative comparison of MPJPE and P-MPJPE based on detecting 2D pose input on the Human3.6M dataset. Under the more challenging condition of detecting 2D pose input (T=243), PhysMoBERT achieves an MPJPE score of 38.2mm, significantly outperforming traditional methods such as VideoPose3D (46.8mm) and PoseFormer (44.3mm), and also demonstrating a competitive advantage compared to the current state-of-the-art methods. Specifically, it performs particularly well in dynamic motion categories such as "Walk" (34.6mm) and "WalkT" (26.9mm), which is attributed to the specialized design of spatiotemporal modeling in this embodiment. In terms of P-MPJPE, the method in this embodiment achieves 32.3mm, which is comparable to the best method STCFormer (31.8mm). However, it exhibits more stable performance across multiple motion categories, especially maintaining a lower error in structurally complex seated motions.

[0062] Table 1 Table 2 presents a quantitative comparison of MPJPE based on 2D real keypoint input on the Human3.6M dataset. When the input is accurate 2D real keypoints, PhysMoBERT achieves an MPJPE of 17.5 mm, outperforming MotionBERT (17.8 mm) and BMFormer (18.9 mm). In fine-grained motion analysis, the method in this embodiment (Ours) achieves the best results in actions involving fine upper limb coordination, such as "Eat" (17.2 mm), "Phone" (17.4 mm), and "Photo" (18.1 mm), validating the effectiveness of spatial branching in modeling complex joint relationships. Notably, in long-time sequence processing (T=243), the method in this embodiment shows a significant advantage over methods using only short sequences (T=81), demonstrating the effectiveness of temporal branching in long-term dependency modeling.

[0063] Table 2 Table 3 Table 3 presents the quantitative comparison results on the MPI-INF-3DHP dataset. On the more challenging MPI-INF-3DHP dataset, PhysMoBERT achieves a resolution of 16.1 mm on the MPJPE metric, outperforming KTPFormer (16.7 mm) and MotionAGFormer (16.2 mm). It also performs comparably to the best method in PCK (98.5%) and AUC (85.3%), with PCK matching GLA-GCN and AUC only slightly lower than TCPFormer (86.5%). This result demonstrates that the method in this embodiment exhibits good generalization ability across different datasets and can adapt to complex indoor and outdoor scene variations. Particularly noteworthy is that this embodiment maintains high accuracy while using only 81 frames of input sequence, offering a significant advantage in computational efficiency compared to methods requiring longer sequences (such as D3DP which requires 243 frames).

[0064] Table 4 presents a quantitative comparison of human mesh reconstruction on the 3DPW dataset. On the 3DPW dataset, which depicts real outdoor scenes, PhysMoBERT achieves MPVE and MPJPE scores of 83.5 mm and 70.8 mm, respectively, outperforming most existing methods. Compared to video-based methods, the method in this embodiment uses only 2D motion sequences as input, and outperforms temporal methods such as VIBE, MPS-Net, and MotionBERT in both MPVE and MPJPE scores. Although it is slightly inferior to TokenHMR (44.3 mm) and ScoreHypo (44.5 mm), which are specifically optimized for this metric, in the PA-MPJPE score (45.2 mm), it performs excellently in terms of overall mesh quality (MPVE) and joint accuracy (MPJPE), validating the practical value of the method in real-world complex scenes.

[0065] Table 4 Table 5 shows the ablation experiments of different components of the method in this embodiment. After gradually introducing the STAM, PCEM, and AFM modules, the MPJPE gradually decreased from the baseline of 39.2 mm to 38.2 mm, the P-MPJPE decreased from 32.9 mm to 32.4 mm, and the PA-MPJPE decreased from 17.8 mm to 17.5 mm. This progressive performance improvement clearly demonstrates the effectiveness of each module. The STAM module brings initial improvement through skeletal topology awareness (MPJPE reduction of 0.7 mm), the PCEM module further optimizes through physical constraint embedding (MPJPE further reduced by 0.2 mm), and finally the AFM module achieves optimal performance through adaptive fusion. This complementarity between modules verifies the rationality of the architecture design in this embodiment.

[0066] Table 5 Table 6 Table 6 presents the ablation experiments of the JCM, LCM, and BCM components. Fine-grained analysis of the physical constraint embedding module shows that all three sub-modules—JCM, LCM, and BCM—contribute significantly to the final performance. Introducing the JCM module alone reduced the MPJPE from 39.2 mm to 38.8 mm, primarily improving joint-level motion rationality. Further introduction of the LCM module reduced the MPJPE to 38.3 mm, highlighting the importance of limb coordination modeling. Finally, the introduction of the BCM module achieved the optimal MPJPE of 38.2 mm, validating the necessity of whole-body dynamic constraints. Particularly noteworthy is the significant role of the BCM module in reducing acceleration errors, providing crucial assurance for motion smoothness.

[0067] Table 7 shows a comparison of different fusion strategies. Compared with traditional fusion strategies such as feature concatenation (38.5mm), average pooling (38.4mm), and weighted summation (38.3mm), the AFM module proposed in this embodiment achieves the best results in all three metrics: MPJPE, P-MPJPE, and PA-MPJPE. This advantage is particularly evident in complex action sequences, demonstrating that the AFM module can dynamically adjust the weights of spatiotemporal branches according to input characteristics, achieving more intelligent feature fusion. This adaptive mechanism is of great significance for handling different types of human motion (such as slow daily movements vs. fast sports movements).

[0068] Table 7 The above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the present invention. Those skilled in the art can make various modifications or equivalent substitutions to the present invention within its scope and spirit, and such modifications or equivalent substitutions should also be considered to fall within the scope of protection of the present invention.

Claims

1. A 3D human pose estimation method based on physical constraints and spatiotemporal decoupling, characterized in that, Includes the following steps: Step 1: Construct a 3D human pose estimation model that includes a spatiotemporally decoupled dual-branch encoder and train the model; construct a multi-dimensional loss function and dynamically adjust the weights of different dimensions of loss during training to ensure that the output pose conforms to physical laws. Step 2: Collect human movement image information, extract 2D pose sequences, and preprocess and encode them in terms of spatiotemporal position; Step 3: Input the 2D pose sequence obtained in Step 2 into the 3D human pose estimation model trained in Step 1, and extract features for the temporal branch and the spatial branch respectively. Step 4: The temporal branch features and spatial branch features from Step 3 are fused across modal attention using an adaptive fusion module. The bidirectional attention results are integrated based on dynamic weights, and the fused features are output. Step 5: Map the fused features obtained in Step 3 to 3D space through a linear decoding layer; In the spatiotemporal decoupled dual-branch encoder, the temporal branch is used to identify periodic motion patterns and predict motion trends to ensure the smoothness of 3D pose in the time dimension. The spatial branch learns anatomically compliant spatial relationships through the skeletal topology perception module and the hierarchical physical constraint embedding module. The multi-dimensional loss function is specifically as follows: in , , and These represent the dynamic weighting coefficients for basic pose loss, bone length consistency loss, joint angle loss, and temporal smoothness loss, respectively. Indicates the basic attitude loss. This indicates loss of bone length consistency. This indicates a loss of joint angle. This represents the loss of temporal smoothness.

2. The 3D human pose estimation method based on physical constraints and spatiotemporal decoupling according to claim 1, characterized in that, Step 2, preprocessing and spatiotemporal location encoding, specifically involves mapping low-dimensional 2D coordinates to a high-dimensional feature space using a linear projection layer. This preserves the original motion information while expanding the representational capability. The formula is as follows: in, For learnable projection weights, , Here, C is the bias term, C is the feature dimension, and X is the 2D pose sequence. T is the sequence length, and J is the number of joints; The 2D pose sequence after high-dimensional feature projection; Introducing learnable temporal position coding Spatial location coding Embedded representation is: 。 3. The 3D human pose estimation method based on physical constraints and spatiotemporal decoupling according to claim 1, characterized in that, In step 3, the skeletal topology sensing module constructs the joint features of all joint pairs. BF (Batch-Frame Product) learns different types of joint relationships through four independent encoders; it defines body part hierarchies and constructs a hierarchical relationship matrix. Encoding into hierarchical features through linear layers Incorporating multi-relational encoding results; by randomly initializing weights Weighted aggregation of relational features is performed, and then joint dependencies are captured using spatial multi-head attention with the original features and aggregated relational features. By concatenating the original features and attention features to generate a gating coefficient g, the original information and the aggregated context are balanced, as shown in the formula: in, For bias terms, For the gated layer activation function, Features generated for global awareness based on relational attention. This is the final fused feature output after gating and balancing. After layer normalization and residual connection, the skeletal topology perception module features are output. .

4. The 3D human pose estimation method based on physical constraints and spatiotemporal decoupling according to claim 1, characterized in that, The hierarchical physical constraint embedding module specifically includes a joint constraint unit, a limb coordination unit, and a whole-body consistency unit. The joint constraint unit constrains the range of motion of joints based on anatomical limitations, specifically by calculating angular deviations using parent joint vectors and child joint vectors. Define the range of motion of the joint and calculate the constraint loss. : ,in This represents the lower anatomical limit of the range of motion of joint j. This represents the anatomical upper limit of the range of motion of joint j; Mapping angle deviation and constraint loss to feature adjustment amounts Spatial features are incorporated through linear layers: ,in, To enhance spatial features by incorporating joint constraint information, These are learnable mapping weights.

5. The 3D human pose estimation method based on physical constraints and spatiotemporal decoupling according to claim 4, characterized in that, The limb coordination module ensures the coordination of joint movements within the same limb by grouping joints and achieving intra-limb joint interaction through multi-head attention within these groups. Let |g| represent the joint characteristics of a certain limb, and |g| represent the number of joints in that limb. B represents the aggregated features within the limb, C represents the batch size, and C represents the feature dimension. Calculate the representative features of each limb By capturing global coordination relationships through multi-head attention mechanisms between limbs, output .

6. The 3D human pose estimation method based on physical constraints and spatiotemporal decoupling according to claim 4, characterized in that, The whole-body consistency module ensures that the overall movement conforms to the laws of dynamics, specifically by: based on the weights of human body mass distribution. Weighted calculation of centroid position: Through the loss function The focus of punishment shifted drastically: Where B is the batch size and F is the total number of frames. Let b be the centroid coordinates of the b-th sample in frame t. Let be the centroid coordinates of the b-th sample at frame t-1; Joint acceleration was calculated using second-order difference. The formula for punishing unreasonable acceleration is: .

7. The 3D human pose estimation method based on physical constraints and spatiotemporal decoupling according to claim 1, characterized in that, Step 4 specifically involves: Step 4.1: Project the bi-branch features to a unified space using a non-linear mapping to ensure fusion compatibility. Step 4.2: Generate global weights based on the overall characteristics of the sequence. The formula is: This is the result of global feature aggregation; Local weights are generated based on joint-level differences. To adapt to the spatiotemporal dependency requirements of different joints; Calculate the time-aware factor using time-series features. Adjust the temporal components of the local weights, and combine them with the joint importance factor. Integrating global and local weights: ,in The adjusted local weights, To incorporate dynamic weights; Step 4.3: Through cross-modal attention fusion, bidirectional interaction of spatiotemporal features is achieved. Based on the dynamic weights obtained in Step 4.2, the bidirectional attention results are integrated to output the fused features. .

8. The 3D human pose estimation method based on physical constraints and spatiotemporal decoupling according to claim 1, characterized in that, The basic attitude loss Specifically: ,in As the weight of key joints, For the true attitude coordinates, To predict attitude coordinates; Loss of bone length consistency Specifically: in Indicates the total number of skeletal connections. Indicates the training batch size. The number of frames in the input sequence. , This represents the predicted 3D coordinates of the pose. , 3D coordinates representing the actual pose, subscript Index batches, time frames, parent joints, and child joints respectively; Joint angle loss Specifically: in, Let S be the total number of joint constraints, and S be the set of joint constraints. and These represent the lower and upper anatomical limits of a specific joint's range of motion, respectively. Indicates in batch and time Calculated actual joint angles Temporal smoothness loss Specifically: in, This represents the total number of joints in the human body. Representative batch ,time ,joint 3D coordinate position.

9. A 3D human pose estimation system based on physical constraints and spatiotemporal decoupling, characterized in that, The method for implementing the 3D human pose estimation method based on physical constraints and spatiotemporal decoupling as described in any one of claims 1-8 includes an input module, a preprocessing and embedding module, a spatiotemporal decoupling dual-branch encoder, an adaptive fusion module, an output module, and a physical perception loss module. The input module is used to input a 2D pose sequence, and the preprocessing and embedding module is used to perform high-dimensional feature projection and spatiotemporal position encoding on the 2D pose sequence. The spatiotemporal decoupled dual-branch encoder includes a temporal branch composed of a 5-layer Transformer encoder and a spatial branch composed of a skeletal topology perception module and a hierarchical physical constraint embedding module. The temporal branch captures long-term motion dependence and continuity through alternating spatiotemporal attention. The skeletal topology perception module avoids the limitations of a fixed adjacency matrix by implicitly learning the dynamic relationships between joints. The hierarchical physical constraint embedding module specifically includes a joint constraint unit, a limb coordination unit, and a whole-body consistency unit. The joint constraint unit constrains the range of motion of joints based on anatomical limitations. The limb coordination module ensures the coordination of joint movements of the same limb. The whole-body consistency module ensures that the overall motion conforms to the laws of dynamics. The adaptive fusion module is used to extract temporal and spatial features from the spatiotemporally decoupled dual-branch encoder and perform cross-modal attention fusion. The physical perception loss module ensures that the 3D posture output by the output module conforms to physical laws by dynamically adjusting the dynamic weight coefficients of the basic posture loss, bone length consistency loss, joint angle loss, and temporal smoothness loss.

Images