Video three-dimensional pose estimation method and system based on skeleton graph constraint block timing

Abstract

The application is suitable for the technical field of computer vision, and provides a video three-dimensional pose estimation method and system based on skeleton graph constraint block timing, which comprises the following steps: performing joint linear embedding on a two-dimensional pose sequence, superimposing learnable joint position coding, and generating initial features; feeding the initial features into a preset skeleton graph constraint block timing Transformer model; the skeleton graph constraint block timing Transformer model comprises multiple layers of encoding iteration modules and a three-dimensional coordinate regression head, and each layer of encoding iteration module is processed by a skeleton graph constraint double-path encoder and a block timing interaction module in turn; after layer normalization and feature projection preprocessing are sequentially performed on the updated timing features, the three-dimensional coordinate regression head is input, and three-dimensional human pose coordinates are predicted. The application can maintain good three-dimensional human pose estimation accuracy under lower parameter quantity and computation quantity, balance model performance and deployment cost, and is suitable for various video pose estimation scenes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a method and system for video 3D pose estimation based on skeleton graph constraint block temporal sequence. Background Technology

[0002] Monocular video 3D human pose estimation is a fundamental and core task in the field of computer vision. Its core objective is to extract 2D human keypoints from videos captured by a monocular camera and further infer the 3D spatial coordinates of various joints. Compared to acquisition methods such as multi-view cameras and depth sensors, monocular video acquisition is convenient, has low hardware costs, and has strong practical value. However, 2D images are projections of 3D space, inherently containing depth ambiguity. Furthermore, issues such as human occlusion, limb crossing, rapid movement, and perspective changes in video further amplify the uncertainty of pose estimation.

[0003] Currently, 2D to 3D pose enhancement has become the mainstream technical paradigm for 3D human pose estimation in monocular video. This paradigm decouples image appearance feature extraction from 3D geometric inference, relies on mature 2D keypoint detection algorithms, and focuses on learning skeleton structure and temporal motion patterns. Existing mainstream methods are mainly divided into four technical routes: temporal convolution methods, spatiotemporal Transformer methods, skeleton graph structure prior methods, and efficient long sequence modeling methods.

[0004] Temporal convolution methods rely on convolution kernels to capture local motion in adjacent frames, resulting in high efficiency in temporal modeling, but they struggle to capture long-distance inter-frame dependencies. Spatiotemporal Transformer methods model joint relationships through spatial attention and cross-frame dependencies through temporal attention, demonstrating outstanding long-range context utilization capabilities. However, the pure attention mechanism requires learning joint connection relationships from massive amounts of data and does not incorporate prior knowledge of the inherent physical topology of the human body, which can easily lead to problems such as abnormal limb proportions and joint misalignment. Skeleton graph convolution methods construct graph structures based on the physical connections of human joints, explicitly incorporating structural priors, but the fixed graph structure is insufficient for modeling collaborative actions of distal joints and non-physically related actions. In recent years, efficient modeling schemes such as frequency domain compression, token filtering, and state-space models have reduced the computational overhead of long sequences, but they have not differentiated the modeling of three types of information: static topology of the human body, local short-term motion, and long-range action context.

[0005] While existing graph-Transformer fusion models combine graph structures and attention mechanisms, they still suffer from significant technical bottlenecks: First, they fail to explicitly differentiate between modeling fixed skeleton topology, local short-term motion, and long-term temporal dependencies, and the uniform dense attention mechanism increases unnecessary computational overhead, resulting in high model parameter and computational costs. Second, static skeleton graphs lack short-term motion modeling capabilities and cannot effectively fit the continuous changes in human pose between adjacent frames in a video. Third, global attention modeling for long temporal videos leads to a sharp increase in computational complexity with sequence length, making it difficult to balance pose estimation accuracy, model parameter count, and computational overhead. Fourth, most models do not distinguish the feature contributions of different joints and frames, have a single feature fusion method, and poor adaptability.

[0006] In summary, designing a model architecture that takes into account human physical topology priors, local short-term motion, and long-term temporal dependencies, while controlling the number of parameters and computational load, to improve the accuracy and generalization ability of monocular video 3D human pose estimation, has become a technical challenge to be solved by those skilled in the art. Summary of the Invention

[0007] The purpose of this invention is to provide a video 3D pose estimation method and system based on skeleton graph-constrained block temporal sequence, aiming to solve the problems mentioned in the background art.

[0008] This invention is implemented as follows: a video 3D pose estimation method based on skeleton graph-constrained block temporal sequence includes the following steps:

[0009] Obtain the monocular video to be estimated and its two-dimensional pose sequence;

[0010] Perform joint linear embedding on the 2D pose sequence and superimpose learnable joint position codes to generate initial features;

[0011] The initial features are fed into a preset skeleton graph-constrained block-based temporal Transformer model. This model includes multiple layers of encoding iteration modules and a 3D coordinate regression head. Each layer of encoding iteration modules is processed sequentially by a skeleton graph-constrained dual-path encoder and a block-based temporal interaction module. The skeleton graph-constrained dual-path encoder performs data-driven attention path encoding and fixed skeleton graph path encoding on the initial features to obtain attention features and skeleton graph features. The attention features and skeleton graph features are then fused to output fused features. The block-based temporal interaction module divides the fused features into blocks, sequentially completing local temporal modeling within segments and global context aggregation across segments, outputting updated temporal features and temporal query features.

[0012] After performing layer normalization and feature projection preprocessing on the updated temporal features, the data is input into a 3D coordinate regression head to predict the 3D human pose coordinates.

[0013] Furthermore, the method for obtaining the two-dimensional pose sequence specifically includes:

[0014] Perform 2D keypoint detection or read pre-extracted 2D keypoints from video frames of a monocular video.

[0015] The coordinates of the two-dimensional key points are normalized, and the confidence channels of each joint key point are spliced ​​to obtain a standardized two-dimensional pose sequence.

[0016] Furthermore, the method for generating the initial features specifically includes:

[0017] Long-term two-dimensional pose sequences are divided into fixed-length temporal slices to unify the temporal dimension of network input;

[0018] Joint linear embedding is performed on the sliced ​​2D pose sequence, and learnable joint position codes are superimposed to generate initial features.

[0019] Furthermore, the two-dimensional attitude sequence is represented as follows: Where B is the batch size; T is the input frame length; and J is the number of human joints. The number of channels is specified, and the dimensions of each channel include the two-dimensional x-coordinate, two-dimensional y-coordinate, and keypoint confidence. The joint linear embedding and joint position encoding perform the following operations:

[0020]

[0021] in, Represents a joint linear embedding layer; This represents the learnable joint position encoding; Indicates the initial characteristics.

[0022] Furthermore, the processing flow of the skeleton graph-constrained dual-path encoder specifically includes:

[0023] Data-driven attention path encoding: The current input feature H is sequentially encoded using spatial multi-head self-attention and temporal multi-head self-attention, and the output attention feature is then generated. Its encoding formula is:

[0024] ;

[0025] in, Output features for the attention path. This indicates spatial multi-head self-attention along the joint dimension. This represents temporal multi-head self-attention along the time dimension;

[0026] Fixed skeleton graph path encoding: Based on the adjacency matrix A constructed from the fixed physical connections of human joints, feature extraction is performed by combining graph convolution, and the skeleton graph features are output. Its encoding formula is:

[0027] ;

[0028] in, For graph convolution operations;

[0029] Based on a token-gated fusion mechanism, attention features Features of skeleton diagram By performing fusion, fusion characteristics are obtained. .

[0030] Furthermore, in the fixed skeleton graph path encoding, after feature extraction is completed by combining graph convolution, the method further includes: supplementing the short-term motion features of the features output by graph convolution with temporal convolution, and after feature optimization, outputting skeleton graph features. The temporal convolution is of kernel size. The depthwise separable temporal convolution process is as follows: The features output by the graph convolution are sequentially subjected to depthwise convolution, GELU nonlinear activation, ... Pointwise convolution outputs skeleton map features. .

[0031] Furthermore, the token-level gating fusion mechanism specifically involves: incorporating attention features... Features of skeleton diagram After concatenation, a linear transformation is performed, followed by Softmax normalization to obtain the dynamic weight coefficients. , Attention features are weighted and fused based on the dynamic weight coefficients. Features of skeleton diagram The fusion formula for the two features is:

[0032] ;

[0033] in, Features of fusion; , These are the dynamic weighting coefficients for the two features. .

[0034] Furthermore, the segmented temporal interaction module receives the fused features output by the skeleton graph-constrained dual-path encoder, and simultaneously receives a learnable temporal query vector. Its processing flow specifically includes:

[0035] Along the time dimension, the fusion features with complete temporal sequence are evenly divided into 3 independent sub-segments. Each sub-segment independently executes local temporal attention and feedforward network MLP to complete local temporal modeling within the segment.

[0036] The updated sub-fragments are reassembled to obtain local temporal fusion features;

[0037] The learnable temporal query vector is concatenated with the local temporal fusion feature in the time dimension, and cross-segment context aggregation is completed through a masked temporal attention module to obtain the aggregated long sequence.

[0038] The aggregated long sequence is split, and the first T time positions are used as the updated time-series features. The remaining time positions are used as the time-series query vector for the next layer.

[0039] Furthermore, the video 3D pose estimation method based on skeleton graph-constrained block temporal sequence also includes a training phase for the skeleton graph-constrained block temporal sequence Transformer model, as detailed below:

[0040] A multi-loss joint supervision function is constructed, which integrates the average joint coordinate Euclidean distance loss, the scale normalization constraint loss, and the velocity consistency loss; the total loss formula of the multi-loss joint supervision function is:

[0041] ;

[0042] in, Total loss; The loss is the average Euclidean distance between joint coordinates; For scale normalization constraint loss; This is due to the loss of speed consistency between adjacent frames; and All are weighted coefficients;

[0043] Loss calculation is performed based on the multi-loss joint supervision function, and the network parameters of the skeleton graph constrained block temporal Transformer model are iteratively updated through the backpropagation mechanism to complete the overall model training.

[0044] Based on the trained skeleton graph-constrained block-series Transformer model, the predicted 3D human pose coordinates are used as the final monocular video 3D pose estimation result.

[0045] Another objective of this invention is to provide a video 3D pose estimation system based on skeleton graph-constrained block temporal sequence, for implementing the aforementioned video 3D pose estimation method based on skeleton graph-constrained block temporal sequence, comprising:

[0046] The data acquisition unit is used to acquire the monocular video to be estimated and its two-dimensional pose sequence;

[0047] The initial feature generation unit is used to perform joint linear embedding on the two-dimensional pose sequence and superimpose learnable joint position codes to generate initial features;

[0048] A feature processing unit is used to feed the initial features into a preset skeleton graph-constrained block-based temporal Transformer model. The skeleton graph-constrained block-based temporal Transformer model includes multiple layers of encoding iteration modules and a three-dimensional coordinate regression head. Each layer of encoding iteration modules is processed sequentially by a skeleton graph-constrained dual-path encoder and a block-based temporal interaction module. The skeleton graph-constrained dual-path encoder is used to perform data-driven attention path encoding and fixed skeleton graph path encoding on the initial features respectively to obtain attention features and skeleton graph features, and then fuses the attention features and skeleton graph features to output fused features. The block-based temporal interaction module is used to split the fused features into blocks, sequentially complete local temporal modeling within segments and global context aggregation across segments, and output updated temporal features and temporal query features.

[0049] The pose regression unit is used to perform layer normalization and feature projection preprocessing on the updated temporal features in sequence, and then input the three-dimensional coordinate regression head to predict the three-dimensional human pose coordinates.

[0050] The model training unit is used to construct multiple loss functions and backpropagate to update the parameters of the skeleton graph-constrained block-series Transformer model.

[0051] This invention provides a video 3D pose estimation method based on skeleton graph-constrained block temporal sequences. By fusing prior knowledge of human structure and dynamic spatiotemporal dependencies through dual-path encoding and optimizing long sequence modeling efficiency with a block temporal sequence interaction module, it maintains good 3D human pose estimation accuracy with relatively low parameter and computational costs, balancing model performance and deployment costs. It is applicable to various video pose estimation scenarios. This invention divides the modeling of static human topology, local short-term motion, and long-range temporal dependencies, ensuring 3D human pose estimation accuracy while controlling the number of model parameters and computational costs. It possesses strong scene generalization ability and can be widely applied to motion analysis, rehabilitation assessment, virtual reality, human-computer interaction, intelligent monitoring, and other scenarios. Attached Figure Description

[0052] Figure 1 This is a flowchart illustrating the video 3D pose estimation method based on skeleton graph constraint block temporal sequence provided in an embodiment of the present invention.

[0053] Figure 2 This is a module structure diagram of a skeleton graph constrained dual-path encoder provided in an embodiment of the present invention;

[0054] Figure 3This is a network structure diagram of the block-based timing interaction module provided in an embodiment of the present invention;

[0055] Figure 4 This is a schematic diagram illustrating the construction of an adjacency matrix provided in an embodiment of the present invention;

[0056] Figure 5 A complete flowchart of model training and inference provided for embodiments of the present invention;

[0057] Figure 6 These are visualizations of various real-world scenarios provided in the embodiments of the present invention. Detailed Implementation

[0058] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0059] like Figure 1 As shown, in one embodiment of the present invention, a video 3D pose estimation method based on skeleton graph-constrained block temporal sequence is provided, including the following steps:

[0060] S1. Obtain the monocular video to be estimated and its two-dimensional pose sequence;

[0061] S2. Perform joint linear embedding on the two-dimensional pose sequence and superimpose learnable joint position codes to generate initial features;

[0062] S3. The initial features are fed into a preset skeleton graph-constrained block-based temporal Transformer model. The skeleton graph-constrained block-based temporal Transformer model includes multiple layers of encoding iteration modules and a three-dimensional coordinate regression head. Each layer of encoding iteration modules is processed sequentially by a skeleton graph-constrained dual-path encoder and a block-based temporal interaction module. The skeleton graph-constrained dual-path encoder is used to perform data-driven attention path encoding and fixed skeleton graph path encoding on the initial features respectively to obtain attention features and skeleton graph features. The attention features and skeleton graph features are then fused to output fused features. The block-based temporal interaction module is used to split the fused features into blocks, sequentially completing local temporal modeling within segments and global context aggregation across segments, and outputting updated temporal features and temporal query features.

[0063] S4. After performing layer normalization and feature projection preprocessing on the updated temporal features, input the three-dimensional coordinate regression head to predict the three-dimensional human pose coordinates.

[0064] In a preferred embodiment of the present invention, the method for obtaining the two-dimensional pose sequence specifically includes:

[0065] S11. Input monocular video, perform 2D key point detection on the video frames of the monocular video or directly read the pre-extracted 2D key points;

[0066] S12. Normalize the coordinates of the two-dimensional key points and splice the confidence channels of each joint key point to obtain a standardized two-dimensional pose sequence.

[0067] In a preferred embodiment of the present invention, the method for generating the initial feature specifically includes:

[0068] S21. Divide the long-time two-dimensional pose sequence into fixed-length temporal slices to unify the temporal dimension of the network input;

[0069] S22. Perform joint linear embedding on the sliced ​​two-dimensional pose sequence and superimpose learnable joint position codes to generate initial features (high-dimensional).

[0070] Specifically, the two-dimensional attitude sequence is represented as Where B is the batch size; T is the input frame length; and J=17 is the number of human joints. The number of channels is specified, and the dimensions of each channel include the two-dimensional x-coordinate, two-dimensional y-coordinate, and keypoint confidence. The joint linear embedding and joint position encoding perform the following operations:

[0071]

[0072] in, Represents a joint linear embedding layer; This represents the learnable joint position encoding; This represents the initial features fed into the encoding iteration module.

[0073] like Figure 2 As shown, in a preferred embodiment of the present invention, the skeleton graph-constrained dual-path encoder receives the current input feature H and extracts features in parallel through two branches. The specific processing flow includes:

[0074] The first branch, data-driven attention path encoding, sequentially performs spatial multi-head self-attention and temporal multi-head self-attention encoding on the current input feature H, outputting attention features. Its encoding formula is:

[0075] ;

[0076] in, Output features for the attention path. This indicates spatial multi-head self-attention along the joint dimension. This represents temporal multi-head self-attention along the time dimension;

[0077] The second branch, fixed skeleton graph path encoding: Based on the adjacency matrix A constructed from the fixed physical connections of 17 human joints, feature extraction is completed by combining graph convolution, and the skeleton graph features are output. Its encoding formula is:

[0078] ;

[0079] in, For graph convolution operations;

[0080] Based on a token-gated fusion mechanism, attention features Features of skeleton diagram By performing fusion, fusion characteristics are obtained. .

[0081] In an optional embodiment of the present invention, the fixed skeleton graph path encoding, after feature extraction is completed by graph convolution, further includes: supplementing short-time motion features by temporal convolution on the features output by graph convolution, and after feature optimization, outputting skeleton graph features. The temporal convolution is of kernel size. The depthwise separable temporal convolution process is as follows: The features output by the graph convolution are sequentially subjected to depthwise convolution, GELU nonlinear activation, ... Pointwise convolution outputs skeleton map features. .

[0082] In a preferred embodiment of the present invention, the token-level gating fusion mechanism specifically comprises: incorporating attention features Features of skeleton diagram After concatenation, the data is fed into a linear layer + Softmax layer for linear transformation, and then normalized using Softmax to obtain dynamic weight coefficients. , Attention features are weighted and fused based on the dynamic weight coefficients. Features of skeleton diagram The fusion formula for the two features is:

[0083] ;

[0084] in, Features of fusion; , These are the dynamic weighting coefficients for the two features. .

[0085] like Figure 3 As shown, in a preferred embodiment of the present invention, the block-based temporal interaction module receives the fused features output by the skeleton graph-constrained dual-path encoder. It also receives a learnable temporal query vector Q, and its processing flow specifically includes:

[0086] S31. Divide the fusion features with complete temporal sequence into 3 independent sub-segments along the time dimension. Each sub-segment independently executes local temporal attention and feedforward network MLP to complete local temporal modeling within the segment.

[0087] S32. Reassemble the updated sub-segments to obtain the local temporal fusion feature S;

[0088] S33. The learnable temporal query vector Q is concatenated with the local temporal fusion feature S in the time dimension to obtain the sequence U; subsequently, an upper triangle is constructed as follows: The causal mask M is used to feed the sequence U into the masked temporal self-attention module, and then the feedforward network MLP is used to complete the global cross-segment context aggregation to obtain the aggregated long sequence.

[0089] S34. Split the aggregated long sequence and extract the first T time positions as the updated time-series features, which are the main features of the current layer. The remaining temporal positions are fed into the next layer of encoding iteration; they are used as the temporal query vector for reuse in the next layer. .

[0090] like Figure 4 The diagram shows the construction of an adjacency matrix based on a 17-joint skeleton. There are 17 joints in the human body, numbered 0-16. The joint connection relationship follows the topology of the human physiological skeleton: 0 is the pelvis as the root node, branching downwards to the left lower limb (1 left hip, 2 left knee, 3 left ankle), right lower limb (4 right hip, 5 right knee, 6 right ankle), and trunk branches (7 trunk center, 8 upper trunk); 8 upper trunk extends to 9 neck, 10 head, 11 right shoulder, 14 left shoulder; 11 right shoulder extends to 12 right elbow, 13 right wrist; 14 left shoulder extends to 15 left elbow, 16 left wrist.

[0091] The standardization steps for the adjacency matrix are as follows:

[0092] (1) Initialize the original adjacency matrix A according to the physiological connection relationship of the joints;

[0093] (2) Adding a unit self-loop to the matrix This ensures that the features of each joint can participate in convolution aggregation;

[0094] (3) Based on the node degree matrix Perform row normalization to obtain the standard input matrix for graph convolution. .

[0095] like Figure 5As shown, in a preferred embodiment of the present invention, the video 3D pose estimation method based on skeleton graph constrained block temporal sequence further includes a training phase and an inference phase of the skeleton graph constrained block temporal sequence Transformer model, as detailed below:

[0096] I. Training Phase:

[0097] (1) Input training dataset (containing paired 2D pose sequences and 3D pose ground truths);

[0098] (2) Network forward propagation, based on the model's backbone network output, predicts the 3D human pose sequence;

[0099] (3) Construct a multi-loss joint supervision function that integrates the average joint coordinate Euclidean distance loss, scale normalization constraint loss, and velocity consistency loss; the total loss formula of the multi-loss joint supervision function is:

[0100] ;

[0101] in, Total loss; The loss is the average joint coordinate Euclidean distance, i.e., the MPJPE position loss. For scale normalization constraint loss; This is due to the loss of speed consistency between adjacent frames; and All are weighted coefficients;

[0102] (4) Based on the multi-loss joint supervision function, the loss is calculated, and the network parameters of the skeleton graph constrained block temporal Transformer model are iteratively updated through the backpropagation mechanism to complete the overall model training;

[0103] (5) Save the optimal weight model after iterating until the model converges.

[0104] II. Reasoning Stage:

[0105] (1) Input the two-dimensional pose sequence of the monocular video to be estimated;

[0106] (2) Based on the trained skeleton graph constrained block-series Transformer model, perform positive prediction of the original input and horizontal flip to enhance positive prediction simultaneously;

[0107] (3) Take the mean of the two sets of prediction results and perform scale inverse normalization;

[0108] (4) Output the final stable monocular video 3D pose estimation results.

[0109] In an optional embodiment of the present invention, the above-mentioned multi-loss joint supervision function may further include anatomical constraint losses for bone length and joint angles, then the total loss formula of the multi-loss joint supervision function is:

[0110] ;

[0111] in, The loss is the average Euclidean distance between joint coordinates; The loss is a scale-normalized constraint for the human body. Loss of velocity consistency of joint motion between adjacent frames; For optional bone length and joint angle anatomical constraint loss; , , Preset weighting coefficients for each loss item.

[0112] In another embodiment of the present invention, a video 3D pose estimation system based on skeleton graph constraint block temporal sequence is also provided, for implementing the above-mentioned video 3D pose estimation method based on skeleton graph constraint block temporal sequence, specifically including:

[0113] The data acquisition unit is used to acquire the monocular video to be estimated and its two-dimensional pose sequence;

[0114] The initial feature generation unit is used to perform joint linear embedding on the two-dimensional pose sequence and superimpose learnable joint position codes to generate initial features;

[0115] A feature processing unit is used to feed the initial features into a preset skeleton graph-constrained block-based temporal Transformer model. The skeleton graph-constrained block-based temporal Transformer model includes multiple layers of encoding iteration modules and a three-dimensional coordinate regression head. Each layer of encoding iteration modules is processed sequentially by a skeleton graph-constrained dual-path encoder and a block-based temporal interaction module. The skeleton graph-constrained dual-path encoder is used to perform data-driven attention path encoding and fixed skeleton graph path encoding on the initial features respectively to obtain attention features and skeleton graph features, and then fuses the attention features and skeleton graph features to output fused features. The block-based temporal interaction module is used to split the fused features into blocks, sequentially complete local temporal modeling within segments and global context aggregation across segments, and output updated temporal features and temporal query features.

[0116] The pose regression unit is used to perform layer normalization and feature projection preprocessing on the updated temporal features in sequence, and then input the three-dimensional coordinate regression head to predict the three-dimensional human pose coordinates.

[0117] The model training unit is used to construct multiple loss functions and backpropagate to update the parameters of the skeleton graph-constrained block-series Transformer model.

[0118] To verify the effectiveness and superiority of the method proposed in this invention, current mainstream advanced 3D human pose estimation models were selected as benchmarks. Quantitative and qualitative analyses were performed on the publicly available standard datasets Human3.6M and MPI-INF-3DHP. The results are as follows: Figure 6 As shown in Tables 1 and 2:

[0119] Table 1. Performance of the method of this invention and representative methods under Human 3.6M.

[0120] MHFormer 351 30.9M 7.1G 43.0 34.4 MixSTE 243 33.6M 139.0G 40.9 32.6 P-STMO 243 6.2M 0.7G 42.8 34.4 STCFormer 243 4.7M 19.6G 41.0 32.0 PoseFormerV2 243 14.3M 0.5G 45.2 35.6 GLA-GCN 243 1.3M 1.5G 44.4 34.8 MotionBERT 243 42.3M 174.8G 39.2 32.9 KTPFormer 243 33.7M 69.5G 40.1 31.9 Method of the present invention 81 19.7M 24.8G 41.0 34.1 Method of the present invention 243 19.8M 72.4G 38.7 32.7

[0121] Table 2 Performance of the method of the present invention under MPI-INF-3DHP

[0122] MPI-INF-3DHP 9 22.62 98.5 85.5 MPI-INF-3DHP 27 17.17 98.9 89.3 MPI-INF-3DHP 81 16.00 99.0 89.7

[0123] Figure 6 In each group, the left side shows the input real-world image with two-dimensional joint skeleton annotations, and the right side shows the three-dimensional pose estimation result output by the skeleton graph-constrained block-based temporal Transformer model provided in this embodiment of the invention. This invention, through multi-loss joint supervision combining skeleton graph topological prior constraints, estimates a three-dimensional skeleton that perfectly matches the real human physiological skeletal structure, verifying the effectiveness of this scheme.

[0124] Based on the quantitative data analysis in Table 1, it can be seen that on the Human3.6M dataset, the embodiments of this invention set two temporal input lengths: 81 frames and 243 frames. When the number of input frames is 81, the model MPJPE is 41.0 mm, P-MPJPE is 34.1 mm, the number of parameters is only 19.7M, and the computational complexity (MACs) is 24.8G. When the number of input frames increases to a long sequence of 243 frames, MPJPE decreases to 38.7 mm, P-MPJPE decreases to 32.7 mm, and the number of model parameters only increases slightly to 19.82M. Compared with large-scale spatiotemporal Transformer models such as MixSTE, MotionBERT, and KTPFormer, this invention has significantly lower parameter count and theoretical computational complexity, while achieving better joint position error indices. Compared with lightweight models such as P-STMO and GLA-GCN, this invention achieves a significant improvement in attitude estimation accuracy under the premise of controllable parameter count. In summary, this invention achieves a good trade-off between model parameter count, computational complexity, and estimation accuracy.

[0125] As shown in Table 2, on the MPI-INF-3DHP cross-scene dataset, as the input temporal length increases from 9 frames to 81 frames, the MPJPE metric of this invention continuously decreases, while the PCK and AUC evaluation metrics steadily improve. Even in scenarios with complex backgrounds, varying camera angles, and diverse motion poses, the model maintains excellent performance, demonstrating that this invention possesses good cross-scene generalization capabilities.

[0126] This advantage stems primarily from the improvements made in the embodiments of the present invention in the following aspects:

[0127] First, dual-path parallel feature extraction decouples topological priors from dynamic spatiotemporal dependencies. The skeleton graph path explicitly introduces topological constraints on the human skeleton by relying on the fixed physiological adjacency matrix of the 17 joints of the human body, and is paired with short-term depthwise separable convolution to compensate for the temporal modeling defects of static graph convolution; the attention path freely learns dynamic joint associations and long-distance inter-frame dependencies in the video; and the two-path features are adaptively fused through token-level dynamic gating weights. In occlusion and limb crossing scenes, the skeleton graph path weights can be automatically increased to suppress unreasonable limb structures.

[0128] Second, the segmented temporal interaction module reduces the computational overhead of long sequences while taking into account the global context. The complete long temporal sequence is evenly divided into three sub-segments. First, lightweight local attention is used within each segment to capture short-term continuous motion. Then, a learnable temporal query vector is used to complete the aggregation of global context across segments. This avoids the computational complexity of global multi-head attention, which increases exponentially with the sequence length, and enables the feasibility of real-time inference for long videos.

[0129] Third, a standardized skeleton adjacency matrix fixes the anatomical boundaries of the human body. A normalized graph convolutional adjacency matrix is ​​constructed based on the actual physiological joint connections of the human body, which naturally restricts the range of relative positions of joints in space. This reduces the risk of abnormal bone proportions and reverse bending of joints from the feature extraction stage, and can output postures that conform to the laws of human movement without the need for complex adversarial branches.

[0130] Fourth, the multi-dimensional joint loss function hierarchically constrains the prediction results. The position loss ensures the accuracy of the basic joint coordinates, the scale loss unifies the global human body size range, the velocity loss constrains the smoothness of inter-frame motion, and the optional bone length / angle loss further enhances the anatomical rationality. The combined effect of multiple loss functions significantly reduces the problems of temporal drift and limb scale jumps.

[0131] Fifth, multi-layer encoding iteratively optimizes spatiotemporal feature representation. Each layer of encoding simultaneously completes the skeleton graph spatial constraints and block-based long-term temporal modeling. The multi-layer stacking gradually refines the spatial relationships of joints and long-period motion patterns, making it more capable of fitting complex actions and long video sequences compared to the single-stage Transformer model.

[0132] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods.

[0133] The above embodiments merely illustrate several implementation methods of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this patent should be determined by the appended claims.

Claims

1. A video 3D pose estimation method based on skeleton graph-constrained block temporal sequence, characterized in that, Includes the following steps: Obtain the monocular video to be estimated and its two-dimensional pose sequence; Perform joint linear embedding on the 2D pose sequence and superimpose learnable joint position codes to generate initial features; The initial features are fed into a preset skeleton graph-constrained block-based temporal Transformer model. The skeleton graph-constrained block-based temporal Transformer model includes multiple layers of encoding iteration modules and a 3D coordinate regression head. Each layer of encoding iteration modules is processed sequentially by a skeleton graph-constrained dual-path encoder and a block-based temporal interaction module. The skeleton graph-constrained dual-path encoder is used to perform data-driven attention path encoding and fixed skeleton graph path encoding on the initial features respectively to obtain attention features and skeleton graph features. The attention features and skeleton graph features are then fused to output fused features. The block-based temporal interaction module is used to divide the fused features into blocks, sequentially complete local temporal modeling within a segment and global context aggregation across segments, and output updated temporal features and temporal query features. After performing layer normalization and feature projection preprocessing on the updated temporal features, the data is input into a 3D coordinate regression head to predict the 3D human pose coordinates.

2. The video 3D pose estimation method based on skeleton graph constraint block temporal sequence according to claim 1, characterized in that, The method for obtaining the two-dimensional pose sequence specifically includes: Perform 2D keypoint detection or read pre-extracted 2D keypoints from video frames of a monocular video. The coordinates of the two-dimensional key points are normalized, and the confidence channels of each joint key point are spliced ​​to obtain a standardized two-dimensional pose sequence.

3. The video 3D pose estimation method based on skeleton graph constraint block temporal sequence according to claim 2, characterized in that, The method for generating the initial features specifically includes: Long-term two-dimensional pose sequences are divided into fixed-length temporal slices to unify the temporal dimension of network input; Joint linear embedding is performed on the sliced ​​2D pose sequence, and learnable joint position codes are superimposed to generate initial features.

4. The video 3D pose estimation method based on skeleton graph constraint block temporal sequence according to claim 3, characterized in that, The two-dimensional attitude sequence is represented as follows: Where B is the batch size; T is the input frame length; and J is the number of human joints. The number of channels is specified, and the dimensions of each channel include the two-dimensional x-coordinate, two-dimensional y-coordinate, and keypoint confidence. The joint linear embedding and joint position encoding perform the following operations: in, Represents a joint linear embedding layer; This represents the learnable joint position encoding; Indicates the initial characteristics.

5. The video 3D pose estimation method based on skeleton graph constraint block temporal sequence according to claim 1, characterized in that, The processing flow of the skeleton graph-constrained dual-path encoder specifically includes: Data-driven attention path encoding: The current input feature H is sequentially encoded using spatial multi-head self-attention and temporal multi-head self-attention, and the output attention feature is then generated. Its encoding formula is: ； in, Output features for the attention path. This indicates spatial multi-head self-attention along the joint dimension. This represents temporal multi-head self-attention along the time dimension; Fixed skeleton graph path encoding: Based on the adjacency matrix A constructed from the fixed physical connections of human joints, feature extraction is performed by combining graph convolution, and the skeleton graph features are output. Its encoding formula is: ； in, For graph convolution operations; Based on a token-gated fusion mechanism, attention features Features of skeleton diagram By performing fusion, fusion characteristics are obtained. .

6. The video 3D pose estimation method based on skeleton graph constraint block temporal sequence according to claim 5, characterized in that, In the fixed skeleton graph path encoding, after feature extraction is completed by graph convolution, the method further includes: supplementing the short-term motion features of the features output by graph convolution with temporal convolution, and after feature optimization, outputting skeleton graph features. The temporal convolution is of kernel size. The depthwise separable temporal convolution process is as follows: The features output by the graph convolution are sequentially subjected to depthwise convolution, GELU nonlinear activation, ... Pointwise convolution outputs skeleton map features. .

7. The video 3D pose estimation method based on skeleton graph constraint block temporal sequence according to claim 5 or 6, characterized in that, The token-level gating fusion mechanism specifically involves: incorporating attention features... Features of skeleton diagram After concatenation, a linear transformation is performed, followed by Softmax normalization to obtain the dynamic weight coefficients. , ; Attention features are weighted and fused based on the dynamic weight coefficients. Features of skeleton diagram The fusion formula for the two features is: ； in, Features of fusion; , These are the dynamic weighting coefficients for the two features. .

8. The video 3D pose estimation method based on skeleton graph constraint block temporal sequence according to claim 1, characterized in that, The segmented temporal interaction module receives the fused features output by the skeleton graph-constrained dual-path encoder, and simultaneously receives a learnable temporal query vector. Its processing flow specifically includes: Along the time dimension, the fusion features with complete temporal sequence are evenly divided into 3 independent sub-segments. Each sub-segment independently executes local temporal attention and feedforward network MLP to complete local temporal modeling within the segment. The updated sub-fragments are reassembled to obtain local temporal fusion features; The learnable temporal query vector is concatenated with the local temporal fusion feature in the time dimension, and cross-segment context aggregation is completed through a masked temporal attention module to obtain the aggregated long sequence. The aggregated long sequence is split, and the first T time positions are used as the updated time-series features. The remaining time positions are used as the time-series query vector for the next layer.

9. The video 3D pose estimation method based on skeleton graph constraint block temporal sequence according to claim 1, characterized in that, It also includes the training phase of the skeleton graph-constrained block-based temporal Transformer model, as detailed below: A multi-loss joint supervision function is constructed, which integrates the average joint coordinate Euclidean distance loss, the scale normalization constraint loss, and the velocity consistency loss; the total loss formula of the multi-loss joint supervision function is: ； in, Total loss; The loss is the average Euclidean distance between joint coordinates; For scale normalization constraint loss; This is due to the loss of speed consistency between adjacent frames; and All are weighted coefficients; Loss calculation is performed based on the multi-loss joint supervision function, and the network parameters of the skeleton graph constrained block temporal Transformer model are iteratively updated through the backpropagation mechanism to complete the overall model training. Based on the trained skeleton graph-constrained block-series Transformer model, the predicted 3D human pose coordinates are used as the final monocular video 3D pose estimation result.

10. A video 3D pose estimation system based on skeleton graph constraint block temporal sequence, used to implement the video 3D pose estimation method based on skeleton graph constraint block temporal sequence as described in any one of claims 1-9, characterized in that, include: The data acquisition unit is used to acquire the monocular video to be estimated and its two-dimensional pose sequence; The initial feature generation unit is used to perform joint linear embedding on the two-dimensional pose sequence and superimpose learnable joint position codes to generate initial features; A feature processing unit is used to feed the initial features into a preset skeleton graph-constrained block-based temporal Transformer model. The skeleton graph-constrained block-based temporal Transformer model includes multiple layers of encoding iteration modules and a three-dimensional coordinate regression head. Each layer of encoding iteration modules is processed sequentially by a skeleton graph-constrained dual-path encoder and a block-based temporal interaction module. The skeleton graph-constrained dual-path encoder is used to perform data-driven attention path encoding and fixed skeleton graph path encoding on the initial features respectively to obtain attention features and skeleton graph features, and then fuses the attention features and skeleton graph features to output fused features. The block-based temporal interaction module is used to divide the fused features into blocks, sequentially complete local temporal modeling within a segment and global context aggregation across segments, and output updated temporal features and temporal query features. The pose regression unit is used to perform layer normalization and feature projection preprocessing on the updated temporal features in sequence, and then input the three-dimensional coordinate regression head to predict the three-dimensional human pose coordinates. The model training unit is used to construct multiple loss functions and backpropagate to update the parameters of the skeleton graph-constrained block-series Transformer model.

Images