A video three-dimensional human pose estimation method based on state space recursive model

Abstract

This invention relates to the field of computer vision technology, specifically providing a video 3D human pose estimation method based on a state-space recursive model. The method first extracts human keypoint information from video frames using a 2D pose estimator, and encodes it into frame-level features via a pose embedding module. Then, a state-space recursive network is used to achieve temporal modeling with linear time complexity. A token pruning clustering module based on state density peaks is combined to select keyframes. Representative tokens are input into the state-space recursive network for long-sequence temporal modeling with linear computational complexity. Finally, state interpolation and propagation mechanisms are used to recover the complete sequence, achieving 3D human pose reconstruction. This method has low computational complexity, strong real-time performance, and is suitable for video pose estimation tasks in edge computing and embedded devices.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and artificial intelligence, specifically to a video 3D human pose estimation method based on a state-space recursive model. Background Technology

[0002] This section provides only background information relevant to this disclosure and is not necessarily prior art.

[0003] Monocular video 3D human pose estimation is one of the core tasks in computer vision, aiming to recover the 3D spatial coordinates of key human skeletal joints from 2D video sequences, providing data support for applications such as action recognition, human-computer interaction, and virtual reality. Current mainstream technologies typically employ a two-stage approach: first, detecting 2D human joints from a single frame image; then, mapping the 2D joint sequence to 3D space through temporal context modeling. Although deep learning-based methods have made significant progress, reconstructing 3D pose from 2D sequences remains an ill-posed problem due to the inherent lack of depth information in monocular vision, human self-occlusion, and interference from complex scenes, and still faces significant challenges.

[0004] To address these challenges, existing research mainly focuses on two technical approaches: one is structural modeling methods based on graph convolutional networks, which utilize the natural topology of the human skeleton to model local spatial relationships between joints. However, these methods have limited receptive fields, making it difficult to effectively capture high-order dynamic dependencies. The other is spatiotemporal modeling methods based on the Transformer architecture, which capture global spatiotemporal dependencies through self-attention mechanisms and perform well in long sequence modeling. However, the computational complexity of the self-attention module in the Transformer increases quadratically with the sequence length, resulting in huge computational overhead when processing long videos, severely limiting its deployment in resource-constrained environments.

[0005] In scenarios with limited computing resources, achieving efficient 3D human pose estimation is particularly important. Existing research mainly focuses on two directions: efficient architecture design and data redundancy reduction. Although some works have attempted to reduce the computational burden of Transformers through token pruning or frame sampling, these strategies essentially still rely on attention mechanisms and are local optimizations within the model, failing to overcome the inherent secondary computational complexity bottleneck of Transformers.

[0006] In recent years, State-Space Recursive Models (SSMs) have emerged as a novel architecture to replace Transformers due to their linear computational complexity and strong temporal modeling capabilities. Some studies have attempted to introduce learnable dynamic adjacency matrices or motion-adaptive time scales into SSMs to enhance their ability to represent human structure and motion patterns. Although these methods have achieved some accuracy improvements, they still require processing full-length video sequences, and computational complexity increases linearly with the number of frames. When applied to ultra-long videos or edge computing scenarios, the presence of numerous redundant frames still results in significant computational and memory overhead. Therefore, how to introduce intelligent sequence sparsification and keyframe selection mechanisms on top of the efficient linear modeling capabilities of state-space models, while maintaining the accuracy of 3D pose estimation, to significantly reduce computational and storage requirements, is a pressing technical problem to be solved in this field. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing 3D human pose estimation methods based on the Transformer architecture, such as high computational complexity, severe feature redundancy, and difficulty in real-time operation on edge devices. This invention provides a video 3D human pose estimation method based on a state-space recursive model. This method, through a non-attentional state recursive mechanism combined with dynamic token clustering and state interpolation recovery mechanisms, significantly reduces computational and storage overhead while maintaining estimation accuracy, achieving low-power, high-real-time video pose estimation, which is particularly suitable for mobile devices and edge computing devices.

[0008] A video 3D human pose estimation method based on a state-space recursive model includes the following steps:

[0009] Step 1: Perform two-dimensional human keypoint detection on the input video sequence to obtain a two-dimensional pose sequence, and encode the two-dimensional pose sequence into a spatiotemporal feature token through graph convolutional coding and global feature aggregation;

[0010] Step 2: In order to enhance the feature discrimination ability of subsequent token clustering and pruning clustering stages, the spatiotemporal feature tokens obtained in Step 1 are input into a pre-lightweight state space recursive network to achieve preliminary temporal modeling with a small parameter scale and shallow structure, and extract state response features.

[0011] Step 3: Calculate the frame-level comprehensive score based on the state response features, and perform clustering and pruning operations on the input spatiotemporal feature tokens to obtain a representative set of spatiotemporal feature tokens;

[0012] Step 4: Input the representative spatiotemporal feature token into the main state space modeling network composed of multiple state update units stacked together, complete the long temporal dependency modeling based on the state recursion mechanism, and extract the high-level temporal feature representation;

[0013] Step 5: Based on the adaptive state interpolation and recursive recovery mechanism, recover the feature sequence of the complete time series length according to the high-level features of the representative tokens;

[0014] Step 6: Perform 3D coordinate regression on the recovered complete token sequence to output the 3D coordinate sequence of human key points in each frame.

[0015] In some embodiments, step 1 includes the following steps:

[0016] Step 1-1: For each frame in the input video sequence, extract the two-dimensional coordinates and detection confidence of the human joints using a two-dimensional pose estimator (such as OpenPose, HRNet, ViTPose, etc.) to form a two-dimensional pose sequence. :

[0017] ,

[0018] in For frame number, The number of key points. Let J be the two-dimensional coordinates of the j-th joint in frame t. To test the confidence level;

[0019] The attitude sequence is encoded through the following steps to transform the two-dimensional attitude information into a high-dimensional feature representation with spatial topology and temporal context;

[0020] Step 1-2: Local structure modeling. For each frame of pose data in the two-dimensional pose sequence, model the pose data based on the predefined human skeleton adjacency matrix. A graph convolutional network is used to extract joint coordinate features for each frame to perform spatial structure modeling; the human skeleton adjacency matrix A binary matrix based on human anatomy. Matrix elements. Joint i and joint j are physically directly connected if and only if they are not. ; obtain joint features :

[0021]

[0022] in Let be the coordinate matrix of the key points in frame t. For learnable weight parameters, It is a non-linear activation function;

[0023] Steps 1-3: Global frame-level aggregation, processing the joint features extracted by graph convolution. Aggregate according to spatial weights to generate frame-level feature vectors. :

[0024]

[0025] in Let be the learnable weight coefficient of the j-th joint;

[0026] Steps 1-4: Temporal fusion and projection: Apply lightweight convolution or linear projection to the frame-level feature vector sequence to enhance temporal continuity.

[0027]

[0028] Obtain frame-level embedding sequence As the initial spatiotemporal feature token input to the subsequent state space recursive network, it provides the foundation for temporal modeling and token pruning clustering.

[0029] In some embodiments, step 2 includes the following steps:

[0030] Step 2-1: Input the spatiotemporal feature token sequence obtained in Step 1 into the preceding lightweight state-space recursive network, and calculate and update the hidden state using the state recursive equation consistent with the main state-space modeling network; the spatial model adopts the same state recursive mechanism as the backbone state-space network module.

[0031] ,

[0032] A, B, and C are learnable parameters.

[0033] Step 2-2: Based on the updated hidden state in Step 2-1, obtain the state response features of the current time step through linear mapping. This provides a basic representation for subsequent token pruning and clustering.

[0034] In some embodiments, step 3 includes the following steps:

[0035] Step 3-1: State Response Characteristics As input, spatial dimension aggregation operations (such as average pooling or weighted pooling) are performed on the multi-joint features of each frame to remove spatial redundancy and obtain the frame-level feature vector. Where F is the total number of frames and C is the feature dimension. If the input sequence already has frame-level feature representation, this step can be omitted.

[0036] Step 3-2: Based on the frame-level feature vector sequence, the k-nearest neighbor density peak clustering algorithm is used to calculate the local density and relative separation of each frame state, and the clustering score of each frame is calculated accordingly.

[0037] Step 3-3: To enhance the responsiveness to dynamic actions, calculate the motion intensity score for each frame based on inter-frame feature differences, and then weight and fuse this motion intensity score with the clustering score obtained in Step 3-2 to obtain a comprehensive score.

[0038] Steps 3-4: Sort the frames in descending order of their overall scores, and select the top f frames (f < F, where F is the total number of video frames) as representative frames. While retaining their corresponding spatiotemporal feature tokens, record the temporal position index of these representative frames in the original input sequence. (where 0≤ <F). The tokens corresponding to the remaining frames are pruned, significantly reducing the computational overhead of subsequent networks.

[0039] In some embodiments, step 3-2 specifically includes the following sub-steps:

[0040] Step 3-2-1: Calculate the state distance and motion intensity. The state distance between any two frames is calculated using the following formula. :

[0041]

[0042] in and These are the frame-level feature vectors of the i-th and j-th frames obtained in step 3-1, respectively. The Euclidean distance norm is represented; the state distance measures the static similarity between any two frames in the pose feature space.

[0043] Step 3-2-2: Calculate the local density of each frame using the following formula. and relative separation :

[0044]

[0045] in, Indicates the first The set of k nearest neighbor frames corresponding to a frame; The bandwidth parameter of the kernel function can be determined by the standard deviation of the frame-level feature vector or by cross-validation.

[0046] Relative separation Indicates the first Minimum distance between a frame and its higher-density states in its feature space:

[0047]

[0048] Step 3-2-3: Calculate the cluster score using the following formula. :

[0049]

[0050] In some embodiments, step 3-3 specifically includes the following sub-steps:

[0051] Step 3-3-1: To capture motion change information, calculate the motion intensity score for each frame. Its value is defined by the following formula:

[0052]

[0053] in Let J represent the state response features of the j-th joint in the i-th frame, where J is the total number of joints. For the initial frame of the sequence, its motion intensity score... Set to 0 or a preset value. This score quantifies the intensity of instantaneous pose changes by measuring the difference in feature vectors between adjacent frames, complementing the static clustering score mentioned above;

[0054] Step 3-3-2: Score the exercise intensity Normalization is performed as follows and compared with the cluster score obtained in step 3-2-3 A weighted fusion is performed to obtain a comprehensive score:

[0055]

[0056] Here, γ is a positive fusion weight hyperparameter used to adjust the importance of motion information.

[0057] In some embodiments, step 4 specifically includes the following steps:

[0058] Step 4-1: State Update. Input the representative spatiotemporal feature tokens obtained in Step 3 into the main state space modeling network. This model consists of multiple stacked state update units, each modeled based on the state recursion equation:

[0059]

[0060] in Let be the hidden state of frame t. Let A be the input token features, and B be the learnable parameter matrices.

[0061] Step 4-2: State-response mapping, using the hidden states of each layer in Step 4-1. Using the input as input, a linear mapping is performed on the outputs of each layer to obtain the instantaneous state response:

[0062]

[0063] in The state response is a learnable projection matrix, carrying timing context information and input to the gating unit;

[0064] Step 4-3: Gated propagation mechanism, based on the instantaneous state response obtained in step 4-2. A gated linear unit (GLU) is introduced to selectively propagate the state in order to suppress redundant feature information and obtain the filtered effective state output. :

[0065]

[0066] in For linear transformation weights, It is a non-linear activation function. This indicates element-wise multiplication;

[0067] Step 4-4: Residual connection and normalization, combined with the initial hidden state from Step 4-1. Compared with the output after filtering in step 4-3 The state update of the current cell is completed through residual connection and normalization:

[0068]

[0069] Steps 4-5: Temporal aggregation. Repeat steps 4-1 to 4-4 to stack multiple state update units to complete the full sequence modeling. At the end of the model, the temporal aggregation layer summarizes the features of the states across multiple frames and outputs a high-level temporal feature representation. It is then passed to the token reconstruction step to achieve full-frame feature recovery and 3D pose regression.

[0070] In some embodiments, step 5 specifically includes the following steps:

[0071] Step 5-1: Index mapping. Based on the high-level temporal features of the representative tokens output in Step 4, and according to the temporal position index recorded in Step 3, determine the temporal correspondence between the representative frame and the missing frame to be recovered.

[0072] Step 5-2: State interpolation. Based on the temporal correspondence determined in Step 5-1, and using the hidden states of representative frames adjacent to the missing frame, calculate the initial feature estimate of the missing frame through linear weighted interpolation or exponential weighted interpolation; interpolation weight coefficients. The calculation is based on the relative temporal position of the missing frame as follows:

[0073]

[0074] in and These are the temporal indices of adjacent representative frames. The index of the missing frame to be recovered, and < < .thus, .

[0075] To make the reconstruction process more consistent with human kinematics, an adaptive adjustment mechanism based on motion perception is introduced into the interpolation process. First, the rate of change of state between adjacent representative frames is calculated:

[0076]

[0077] And through a lightweight learnable function (Such as linear layers or lightweight convolutional networks) map it to a scaling factor ;

[0078] Interpolation calculations can be performed using any of the following methods:

[0079] 1) Linear interpolation:

[0080]

[0081] 2) Exponentially weighted interpolation

[0082]

[0083] Where 𝜆 is the attenuation coefficient, which is a positive adjustable hyperparameter.

[0084] This design makes localized movements more intense ( When >1), the interpolation is more biased towards the later frame. Or use faster exponential decay for a quicker response to attitude changes; when the motion is smooth ( When <1), it tends to retain the previous frame. The continuity of the state is maintained. Interpolation weights or attenuation coefficients are dynamically adjusted based on the motion characteristics between adjacent representative frames.

[0085] Step 5-3: Using the initial estimate of the missing frame obtained in Step 5-2 as input, perform time recursive optimization through the state space recursive equation to recover the continuous state trajectory of the pruned frame:

[0086]

[0087] in This represents the state at the previous moment. A represents the input features at the current moment, and B represents the learnable parameter matrices that are independently learned within the token reconstruction module. These parameters are independent of those in the pre-lightweight state space recursive network described in step 2 and the main state space modeling network described in step 4.

[0088] Step 5-4: Convert the continuous state optimized in Step 5-3 into frame-level features using linear projection;

[0089]

[0090] Where C is the projectible matrix that this module learns independently.

[0091] Then, lightweight one-dimensional convolutional or linear smoothing layers are introduced to smooth the feature sequence. The fusion process is performed to finally output a full-length feature token sequence. .

[0092] In some embodiments, step 6 specifically includes the following steps:

[0093] Step 6-1: Temporal regression. Perform a three-dimensional pose regression on the complete token sequence output from Step 5, and use one-dimensional convolution or multilayer perceptron to extract consistent dynamic temporal features across frames.

[0094]

[0095] Where k represents the radius of the convolution time window.

[0096] Step 6-2: Spatial coordinate decoding, processing the dynamic temporal features obtained in Step 6-1. By linearly transforming the output mapping matrix and the bias vector, it is converted into a set of three-dimensional coordinates of human joints in the t-th frame;

[0097]

[0098] in, To output the mapping matrix, For bias vectors, Indicates the first A set of three-dimensional coordinates of J joints of a human body.

[0099] Step 6-3: Joint loss function optimization. The regression results are optimized using the joint loss function to obtain the optimized 3D joint coordinates, thereby improving the physical rationality and temporal stability of the output posture.

[0100] In some embodiments, step 6-3 specifically includes the following steps:

[0101] Step 6-3-1: Calculate the coordinate error loss respectively Skeleton constraint loss and time-series smoothing loss :

[0102]

[0103] Where F is the total number of frames in the input video sequence, and J is the total number of key points for human pose in each frame. The three-dimensional spatial coordinates of the j-th joint in the t-th frame predicted by the model are: For the corresponding The true three-dimensional coordinates This represents the squared Euclidean norm; this loss is used to constrain the deviation between the predicted coordinates and the true coordinates, resulting in the average position error per joint (MPJPE).

[0104]

[0105] in Let the set of edges of the human skeleton be denoted as , and each edge... This represents the index pairs of connected joints. This set is determined based on prior human anatomy and is a fixed quantity throughout the training process. The number of skeleton edges; this loss constraint ensures that the length and orientation of the skeleton in the predicted pose are consistent with the actual pose, so as to ensure the physical rationality of the human body structure;

[0106]

[0107] in The loss is the complete set of 3D pose coordinates predicted by the model for the t-th frame. This loss is used to constrain the smoothness of the pose of adjacent frames, reduce timing jitter, and improve motion coherence.

[0108] Step 6-3-2: Establish a joint optimization objective function based on the above three losses. :

[0109]

[0110] in and A positive, adjustable hyperparameter that is adaptively set based on the performance of the validation set;

[0111] Step 6-3-3: Based on the joint total loss function The network parameters are iteratively optimized until the model converges.

[0112] An efficient video 3D human pose estimation system based on a state-space recursive structure includes the following modules connected in sequence:

[0113] A 2D pose detection module is used to extract the coordinates of key human body points from the input video;

[0114] The pose embedding module is used to encode two-dimensional pose sequences into frame-level spatiotemporal feature tokens;

[0115] A lightweight state-space recursive network module is used to perform preliminary temporal modeling on spatiotemporal feature tokens and generate state response features;

[0116] The token pruning clustering module is used to dynamically select representative tokens based on the state density and motion intensity analysis of state response characteristics, thereby achieving sparsity of the input sequence;

[0117] The main state space modeling network module is used to perform deep temporal dependency modeling on the representative tokens and extract high-level temporal feature representations.

[0118] The token reconstruction module is used to recover the full-length feature token sequence from representative tokens based on interpolation and recursion mechanisms.

[0119] The 3D pose regression module maps the recovered complete feature token sequence to a set of 3D coordinates of human joints in each frame, and outputs the final 3D human pose sequence.

[0120] Beneficial effects

[0121] Compared with the prior art, the present invention has the following significant advantages and beneficial effects:

[0122] 1. High-efficiency time-series modeling capability: This invention adopts a state-space-based linear recursion mechanism, which avoids the secondary computational complexity of the Transformer self-attention structure, realizes linear time modeling of long sequences, and greatly improves computational efficiency.

[0123] 2. Dynamic keyframe selection and input sparsity: By introducing a dual-driven token pruning clustering strategy based on state density peak and motion intensity, keyframes can be adaptively selected, effectively compressing the input timing and reducing redundant computation.

[0124] 3. Continuous temporal feature recovery: By adopting a motion-aware adaptive state interpolation and recursive mechanism, a complete and continuous state feature sequence can be reconstructed from keyframes, taking into account both temporal coherence and information integrity.

[0125] 4. End-to-end Optimizable Structure: The joint loss function designed in this invention integrates pose accuracy, skeleton consistency and temporal smoothness constraints to achieve end-to-end optimization and improve the three-dimensional structural consistency and temporal stability of the model.

[0126] 5. Low power consumption and high real-time performance: The overall architecture of this invention has low computational complexity and small memory footprint, making it particularly suitable for real-time video 3D pose estimation tasks on mobile devices and edge computing devices. Attached Figure Description

[0127] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0128] Figure 1 This is a system structure block diagram of the proposed solution;

[0129] Figure 2 This is a schematic diagram of the token pruning and clustering module structure in this solution;

[0130] Figure 3 This scheme uses a state-space based temporal network structure.

[0131] Figure 4 This is the data flow for the token reconstruction module in this solution. Detailed Implementation

[0132] This example provides an efficient video 3D human pose estimation method based on a state-space recursive structure. The state-space model used in the method can adopt various structural forms, including but not limited to Mamba, Mamba2, etc. The following description uses the Structured State-Space Dual (SSD) architecture of Mamba2 as a preferred embodiment. However, the scope of protection of this invention is not limited to this implementation.

[0133] The model overview is as follows: Figure 1 As shown, the specific steps include:

[0134] Step 1: Video Input and 2D Pose Extraction

[0135] Step 1-1: The system receives a continuous sequence of video frames and uses existing 2D pose estimators (such as OpenPose, HRNet, ViTPose, etc.) to extract the 2D coordinates and confidence scores of the human joints in each frame, forming the input pose sequence:

[0136]

[0137] Where F is the number of video frames (243 frames in this embodiment), J is the number of key points (17), and C is the feature dimension (3).

[0138] The data structure is as follows:

[0139] ,

[0140] in( () represents the normalized two-dimensional coordinates of the j-th joint in the t-th frame; This is the confidence score, used for subsequent weighted fusion.

[0141] The preferred input video frame resolution is 256×256, and the frame rate is 30FPS. Confidence information will be used as feature weights in subsequent encoding.

[0142] Step 1-2: Local structure modeling. For each frame of pose data in the two-dimensional pose sequence, model the pose data based on the predefined human skeleton adjacency matrix. A graph convolutional network is used to extract joint coordinate features for each frame to perform spatial structure modeling; the human skeleton adjacency matrix A binary matrix based on human anatomy. Matrix elements. Joint i and joint j are physically directly connected if and only if they are not. ; obtain joint features :

[0143]

[0144] in Let A be the joint feature matrix of frame t, and let A be the adjacency matrix of the human skeleton (17×17). For learnable graph convolution weights, It is a non-linear activation function.

[0145] Steps 1-3: Global Frame-Level Aggregation: Perform a weighted average of all joint features in each frame to obtain a frame-level feature vector.

[0146]

[0147] in is the learnable weighting coefficient for the j-th joint.

[0148] Steps 1-4: Temporal Fusion and Projection: Mapping frame-level feature sequences to a unified high-dimensional embedding space through linear transformation.

[0149]

[0150] in It is a 256-dimensional linear projection matrix. As a bias term, the final result is a spacetime token sequence of length F. .

[0151] Step 2: Pre-processing a lightweight state-space recursive network

[0152] To improve the feature separability and temporal continuity of the subsequent clustering module and keyframe screening stage, the feature token sequence output in step 1 is input into a lightweight state space modeling module.

[0153] In this embodiment of the feature, the module adopts an SSD-based Mamba2 architecture, and the model parameters are configured as follows:

[0154] State dimension: =16;

[0155] Input feature dimensions: =256;

[0156] Output feature dimensions: =256;

[0157] Number of module layers: L=2;

[0158] The activation function used is GELU;

[0159] Optimizer: AdamW (learning rate 2×10) -4 Weight decay 1×10 -2 ).

[0160] Its core design is as follows:

[0161] SSD architecture features: This architecture constrains the state transition matrix to scalar diagonal form.

[0162]

[0163] in It is the identity matrix. These are learnable scalar parameters that are related to the input.

[0164] Input mapping matrix and output projection moment These are dynamic parameters, generated by a learnable function:

[0165]

[0166] in and It can be implemented using linear layers or small multilayer perceptrons.

[0167] This structure is mathematically equivalent to a causal linear attention mechanism, which maintains the ability to model global dependencies while having linear time complexity.

[0168] Step 2-1: Input the spatiotemporal feature token sequence obtained in Step 1 into the preceding lightweight state-space recursive network, and calculate and update the hidden state using the state recursive equation consistent with the main state-space modeling network; the spatial model adopts the same state recursive mechanism as the backbone state-space network module.

[0169] ,

[0170] Among them This is a diagonal state transition matrix. and For inputting relevant dynamic parameters.

[0171] After 80 epochs of training, the model converged on the Human 3.6M dataset, generating state-response feature sequences with good temporal discriminative properties. }

[0172] Step 2-2: Based on the updated hidden state in Step 2-1, obtain the state response features of the current time step through linear mapping. This serves as the input for the subsequent token pruning and clustering module.

[0173] Step 3: Token Pruning Clustering

[0174] The structure of the token pruning clustering module is as follows: Figure 2 As shown. This module outputs the state response feature sequence from the preceding state-space modeling module { The core of this approach is to dynamically select the most representative keyframes from long sequences based on feature similarity and motion saliency, thereby achieving intelligent sparsification of the input sequence.

[0175] The main parameter settings are as follows:

[0176] kNN neighborhood size k=10;

[0177] Kernel function bandwidth It is 0.5 times the variance of the entire sequence;

[0178] The frame ratio is approximately 1 / 3 (e.g., when F=243, f=81).

[0179] Motion weight coefficient =0.1.

[0180] Step 3-1: Spatial aggregation, state response features As input, spatial weighted pooling is performed on the multi-joint features of each frame to obtain frame-level feature vectors. To eliminate redundancy in spatial dimensions:

[0181]

[0182] in These are learnable or preset spatial weights.

[0183] Step 3-2: Based on the frame-level feature vector sequence, the k-nearest neighbor density peak clustering algorithm and inter-frame motion intensity analysis are used to achieve dynamic clustering of frame-level features and keyframe selection in the state feature space; the local density and relative separation of the state of each frame are calculated, and the comprehensive clustering score of each frame is calculated accordingly; specifically:

[0184] Step 3-2-1: State distance calculation. Calculate the Euclidean distance between features of any two frames in the feature space, as a measure of state difference.

[0185]

[0186] in and These are the frame-level feature vectors of the i-th and j-th frames obtained in step 3-1, respectively. The Euclidean distance norm is represented; the state distance measures the static similarity between any two frames in the pose feature space.

[0187] Step 3-2-2: Density peak clustering score calculation, calculate the local density of each frame using the following formula. and relative separation To characterize the state distribution features of each frame:

[0188]

[0189] in, Indicates the first The set of k nearest neighbor frames corresponding to a frame, where k is 10; The bandwidth parameter of the kernel function can be determined by the standard deviation of the frame-level feature vector or by cross-validation.

[0190] Relative separation Indicates the first Minimum distance between a frame and its higher-density states in its feature space:

[0191]

[0192] Step 3-2-3: Calculate the cluster score using the following formula. :

[0193]

[0194] Step 3-3: To enhance the responsiveness to dynamic actions, calculate the motion intensity score for each frame based on inter-frame feature differences, and then weight and fuse this motion intensity score with the clustering score obtained in Step 3-2 to obtain a comprehensive score; specifically, this includes the following sub-steps:

[0195] Step 3-3-1: To capture motion change information, calculate the motion intensity score for each frame. Its value is defined by the following formula:

[0196]

[0197] in Let J represent the state response features of the j-th joint in the i-th frame, where J is the total number of joints. For the initial frame of the sequence, its motion intensity score... Set to 0. This score quantifies the intensity of instantaneous pose changes by measuring the difference in feature vectors between adjacent frames, complementing the static clustering score mentioned above;

[0198] Step 3-3-2: Score the exercise intensity Normalization is performed as follows and compared with the cluster score obtained in step 3-2-3 A weighted fusion is performed to obtain a comprehensive score:

[0199]

[0200] Where γ is a positive fusion weight hyperparameter, set to 0.1.

[0201] This design ensures that, in addition to the high representativeness of the state features themselves, frames with significant motion receive an extra score boost, making them more likely to be selected as keyframes.

[0202] Steps 3-4: Representative frame selection and index recording:

[0203] Based on the overall score Sort the frames in descending order and select the top 10 frames with the highest scores as representative frames (in this example, when 1=24, 1=81, corresponding to a compression rate of 66%). This forms a representative token sequence. Record the corresponding time-series position index set:

[0204]

[0205] This index is used for time interpolation in subsequent reconstruction modules.

[0206] Unselected frames are processed through a simplified state transfer equation. Perform time updates for context reference in subsequent reconstruction phases.

[0207] Technical effect: This step achieves dynamic sequence compression, which reduces the amount of subsequent computation by about 65% while preserving key information.

[0208] Step 4: Model the network in the main state space

[0209] The representative frame sequence obtained after pruning is input into the main state-space modeling network for deep temporal modeling, and its structure is as follows: Figure 3 As shown. In this embodiment, the main state space modeling network also preferably adopts the SSD-based Mamba2 architecture, which captures global temporal dependencies while maintaining linear complexity by stacking multiple state update units.

[0210] Network parameter configuration:

[0211] State dimension: =32;

[0212] Input feature dimensions: =256;

[0213] Number of module layers: L=4;

[0214] Each layer shares parameters and uses an SSD architecture;

[0215] Gating layer width = =128.

[0216] The core operations for each state update unit are as follows:

[0217] Step 4-1: State Update. Input the representative spatiotemporal feature tokens obtained in Step 3 into the main state space modeling network. This model consists of multiple stacked state update units, each modeled based on the state recursion equation:

[0218]

[0219] in Let be the hidden state of frame t. Let A be the input token features, and B be the learnable parameter matrices.

[0220] Step 4-2: State-response mapping, using the hidden states of each layer in Step 4-1. Using the input as input, a linear mapping is performed on the outputs of each layer to obtain the instantaneous state response:

[0221]

[0222] in The state response is a learnable projection matrix, carrying timing context information and input to the gating unit;

[0223] Step 4-3: Gated propagation mechanism, based on the instantaneous state response obtained in step 4-2. A gated linear unit (GLU) is introduced to selectively propagate the state in order to suppress redundant feature information and obtain the filtered effective state output. :

[0224]

[0225] in For linear transformation weights, It is a non-linear activation function. This indicates element-wise multiplication;

[0226] Step 4-4: Residual connection and normalization, combined with the initial hidden state from Step 4-1. Compared with the output after filtering in step 4-3 The state update of the current cell is completed through residual connection and normalization:

[0227]

[0228] Steps 4-5: Temporal aggregation. Repeat steps 4-1 to 4-4 to stack multiple state update units to complete the full sequence modeling. At the end of the model, the temporal aggregation layer summarizes the features of the states across multiple frames and outputs a high-level temporal feature representation. It is then passed to the token reconstruction step to achieve full-frame feature recovery and 3D pose regression.

[0229] The network consists of multiple layers of stacked units. Thanks to the SSD architecture, the network maintains linear computational complexity while possessing modeling capabilities comparable to the Transformer. Furthermore, the training process is more stable and it is easier to deploy and accelerate on edge devices.

[0230] Technical results: Compared to the Transformer model, the computational complexity is reduced from O(F²) to O(F), training is stable, and it can run in real time on the Jetson Orin NX platform.

[0231] Step 5: Token Reconstruction Module, which includes the following steps:

[0232] The token reconstruction module is responsible for recovering the complete video feature sequence from a finite number of representative frames. The data stream is as follows: Figure 4 As shown, this module consists of four units: index mapping, state interpolation, state propagation, and feature fusion.

[0233] Step 5-1: Index mapping. Based on the high-level temporal features of the representative tokens output in Step 4, and according to the temporal position index recorded in Step 3, determine the temporal correspondence between the representative frame and the missing frame to be recovered.

[0234] Step 5-2: State interpolation. Based on the temporal correspondence determined in Step 5-1, and using the hidden states of representative frames adjacent to the missing frame, calculate the initial feature estimate of the missing frame through linear weighted interpolation or exponential weighted interpolation; interpolation weight coefficients. The calculation is based on the relative temporal position of the missing frame as follows:

[0235]

[0236] in and These are the temporal indices of adjacent representative frames. The index of the missing frame to be recovered, and < < .thus, .

[0237] To make the reconstruction process more consistent with human kinematics, an adaptive adjustment mechanism based on motion perception is introduced into the interpolation process. First, the rate of change of state between adjacent representative frames is calculated:

[0238]

[0239] And through a lightweight learnable function (Such as linear layers or lightweight convolutional networks) map it to a scaling factor ;

[0240] Interpolation calculations can be performed using any of the following methods:

[0241] 1) Linear interpolation:

[0242]

[0243] 2) Exponentially weighted interpolation

[0244]

[0245] Where 𝜆 is the attenuation coefficient, which is taken as 0.7.

[0246] This design makes localized movements more intense ( When >1), the interpolation is more biased towards the later frame. Or use faster exponential decay for a quicker response to attitude changes; when the motion is smooth ( When <1), it tends to retain the previous frame. The continuity of the state is maintained. Interpolation weights or attenuation coefficients are dynamically adjusted based on the motion characteristics between adjacent representative frames.

[0247] Step 5-3: Using the initial estimate of the missing frame obtained in Step 5-2 as input, perform time recursive optimization through the state space recursive equation to recover the continuous state trajectory of the pruned frame:

[0248]

[0249] in This represents the state at the previous moment. A represents the input features at the current moment, and B represents the learnable parameter matrices that are independently learned within the token reconstruction module. These parameters are independent of those in the pre-lightweight state space recursive network described in step 2 and the main state space modeling network described in step 4.

[0250] Step 5-4: Convert the continuous state optimized in Step 5-3 into frame-level features using linear projection;

[0251]

[0252] Where C is the projectible matrix that this module learns independently, and TemporalFusion represents the temporal smoothing operation.

[0253] The reconstructed complete sequence has a dimension of F = 256. After TemporalFusion smoothing, the output is... .

[0254] Technical benefits: Recovery sequence error (MSE) is reduced by 27% compared to traditional linear interpolation, ensuring inter-frame temporal consistency.

[0255] Step 6: 3D pose regression and optimization specifically includes the following steps:

[0256] Step 6-1: Temporal regression. Perform a three-dimensional pose regression on the complete token sequence output from Step 5, and use one-dimensional convolution or multilayer perceptron to extract consistent dynamic temporal features across frames.

[0257]

[0258] Where k represents the radius of the convolution time window.

[0259] Step 6-2: Spatial coordinate decoding, processing the dynamic temporal features obtained in Step 6-1. By linearly transforming the output mapping matrix and the bias vector, it is converted into a set of three-dimensional coordinates of human joints in the t-th frame;

[0260]

[0261] in, To output the mapping matrix, For bias vectors, Indicates the first A set of three-dimensional coordinates of J joints of a human body.

[0262] Step 6-3: Joint Loss Function Optimization. The regression results are optimized using the joint loss function to obtain the optimized 3D joint coordinates, thereby improving the physical plausibility and temporal stability of the output pose. This includes the following steps:

[0263] Step 6-3-1: Calculate the coordinate error loss respectively Skeleton constraint loss and time-series smoothing loss :

[0264]

[0265] Where F is the total number of frames in the input video sequence, and J is the total number of key points for human pose in each frame. The three-dimensional spatial coordinates of the j-th joint in the t-th frame predicted by the model are: For the corresponding The true three-dimensional coordinates This represents the squared Euclidean norm; this loss is used to constrain the deviation between the predicted coordinates and the true coordinates, resulting in the average position error per joint (MPJPE).

[0266]

[0267] in Let the set of edges of the human skeleton be denoted as , and each edge... This represents the index pairs of connected joints. This set is determined based on prior human anatomy and is a fixed quantity throughout the training process. The number of skeleton edges; this loss constraint ensures that the length and orientation of the skeleton in the predicted pose are consistent with the actual pose, so as to ensure the physical rationality of the human body structure;

[0268]

[0269] in The loss is the complete set of 3D pose coordinates predicted by the model for the t-th frame. This loss is used to constrain the smoothness of the pose of adjacent frames, reduce timing jitter, and improve motion coherence.

[0270] Step 6-3-2: Establish a joint optimization objective function based on the above three losses. :

[0271]

[0272] in and Positive, adjustable hyperparameters are adaptively set based on validation set performance; hyperparameters are set based on optimization using the Human 3.6M dataset. =0.1, =0.05.

[0273] Step 6-3-3: Based on the joint total loss function The network parameters are iteratively optimized until the model converges.

[0274] On the Human3.6M dataset, the model achieves MPJPE=40.2mm and P-MPJPE=33.8mm, which is better than similar lightweight Transformer models.

[0275] Step 7: System Operation and Deployment

[0276] The method and its corresponding system described in this invention have excellent cross-platform deployment capabilities. The algorithm model is compatible with mainstream deep learning inference frameworks (including TensorRT, ONNXRuntime, etc.), thus enabling efficient operation on GPU-equipped servers or resource-constrained edge computing platforms (such as NVIDIA Jetson series, Qualcomm mobile SoCs, etc.).

[0277] (1) Video input

[0278] The system acquires raw image sequences via a USB camera, network video stream, or local video file. Input frames can be cached in real time in a video buffer queue for the attitude detection module to read frame by frame.

[0279] (2) Calculate the production line

[0280] The system's computational workflow adopts a modular pipeline structure, including:

[0281] Two-dimensional pose detection module: Implemented by a lightweight pose detection network, which can run in parallel on GPU cores or AI acceleration units;

[0282] State-space temporal reasoning module: performs feature modeling and keyframe selection based on state recursion;

[0283] 3D coordinate regression module: Uses linear regression or lightweight perceptron to complete 3D pose output.

[0284] The modules mentioned above communicate asynchronously via shared memory or message queues, and the entire computation process can be pipelined in parallel on heterogeneous acceleration hardware such as GPUs, NPUs, or DSPs.

[0285] (3) Output of results: The three-dimensional human posture sequence output by the system can be transmitted to the host computer or terminal device in real time through network protocols such as TCP / IP, HTTP or WebSocket, and used for application scenarios such as three-dimensional visualization display, behavior analysis, virtual interactive control or intelligent monitoring.

[0286] System performance metrics (example metrics):

[0287] After hardware optimization and software inference acceleration, the method of this invention exhibits excellent efficiency and accuracy performance under typical conditions. Taking a 243-frame video sequence as an example, the performance indicators are as follows:

[0288] Inference speed: 60 FPS at 243 frames per second on an edge device (Jetson Orin NX);

[0289] Memory usage: When deployed at the edge, memory usage is reduced by approximately 40% compared to similar Transformer models;

[0290] Pose estimation accuracy: On the Human3.6M dataset, MPJPE is approximately 40.2 mm;

[0291] Power efficiency: The overall power consumption is less than 35W on the Jetson NX platform.

[0292] This invention provides an approach and method for video 3D human pose estimation based on a state-space recursive model. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A video 3D human pose estimation method based on a state-space recursive model, characterized in that, Includes the following steps: Step 1: Perform two-dimensional human keypoint detection on the input video sequence to obtain a two-dimensional pose sequence, and encode the two-dimensional pose sequence into a spatiotemporal feature token through graph convolutional coding and global feature aggregation; Step 2: Input the spatiotemporal feature tokens obtained in Step 1 into a pre-lightweight state-space recursive network to achieve preliminary temporal modeling with a small parameter scale and shallow structure, and extract state response features; Step 3: Calculate the frame-level comprehensive score based on the state response features, and perform clustering and pruning operations on the input spatiotemporal feature tokens to obtain a representative set of spatiotemporal feature tokens; Step 4: Input the representative spatiotemporal feature token into the main state space modeling network composed of multiple state update units stacked together, complete the long temporal dependency modeling based on the state recursion mechanism, and extract the high-level temporal feature representation; Step 5: Based on the adaptive state interpolation and recursive recovery mechanism, recover the feature sequence of the complete time series length according to the high-level features of the representative tokens; Step 6: Perform 3D coordinate regression on the recovered complete token sequence to output the 3D coordinate sequence of human key points in each frame.

2. The video 3D human pose estimation method based on a state-space recursive model according to claim 1, characterized in that, Step 1 includes the following steps: Step 1-1: For each frame in the input video sequence, extract the two-dimensional coordinates and detection confidence of the human joints using a two-dimensional pose estimator to form a two-dimensional pose sequence; Steps 1-2: For each frame of pose data in the two-dimensional pose sequence, based on the predefined human skeleton adjacency matrix... A graph convolutional network is used to extract joint coordinate features for each frame to perform spatial structure modeling; the human skeleton adjacency matrix A binary matrix based on human anatomy, matrix elements Joint i and joint j are physically directly connected if and only if they are not. ; Steps 1-3: Aggregate the joint features extracted by graph convolution according to spatial weights to generate frame-level feature vectors; Steps 1-4: Apply lightweight convolution or linear projection to the frame-level feature vector sequence to obtain the final spatiotemporal feature token sequence.

3. The video 3D human pose estimation method based on a state-space recursive model according to claim 1, characterized in that, Step 2 includes the following steps: Step 2-1: Input the spatiotemporal feature token sequence obtained in Step 1 into the preceding lightweight state space recursive network, and calculate and update the hidden state through the state recursive equation consistent with the main state space modeling network. Step 2-2: Based on the updated hidden state in Step 2-1, obtain the state response features of the current time step through linear mapping, which are used for subsequent keyframe clustering and pruning.

4. The video 3D human pose estimation method based on a state-space recursive model according to claim 1, characterized in that, Step 3 includes the following steps: Step 3-1: Perform spatial dimension pooling on the spatiotemporal feature tokens to remove spatial redundancy and obtain frame-level feature vectors; Step 3-2: Based on the frame-level feature vector sequence, the k-nearest neighbor density peak clustering algorithm is used to calculate the local density and relative separation of each frame state, and the clustering score of each frame is calculated accordingly. Step 3-3: To enhance the responsiveness to dynamic actions, calculate the motion intensity score for each frame based on inter-frame feature differences, and then weight and fuse this motion intensity score with the clustering score obtained in Step 3-2 to obtain a comprehensive score. Steps 3-4: Sort the frames in descending order of their comprehensive scores, and select the top f frames (f < F, where F is the total number of video frames) with the highest scores as representative frames. While retaining their corresponding spatiotemporal feature tokens, record the temporal position index of these representative frames in the original input sequence. , where 0≤ <F, the tokens corresponding to the remaining frames are pruned.

5. The video 3D human pose estimation method based on a state-space recursive model according to claim 4, characterized in that, Step 3-2 specifically includes the following sub-steps: Step 3-2-1: Calculate the state distance between any two frames using the following formula. : in and These are the frame-level feature vectors of the i-th and j-th frames obtained in step 3-1, respectively. The Euclidean distance norm is represented; the state distance measures the static similarity between any two frames in the pose feature space. Step 3-2-2: Calculate the local density of each frame using the following formula. and relative separation : in, Indicates the first The set of k nearest neighbor frames corresponding to a frame; The bandwidth parameter of the kernel function can be determined by the standard deviation of the frame-level feature vector or by cross-validation. Step 3-2-3: Calculate the cluster score using the following formula. :

6. The video 3D human pose estimation method based on a state-space recursive model according to claim 4, characterized in that, Step 3-3 specifically includes the following sub-steps: Step 3-3-1: To capture motion change information, calculate the motion intensity score for each frame. The instantaneous change intensity of the quantified attitude is defined by the following formula: in The state response feature of the j-th joint in the i-th frame is given by J, where J is the total number of joints; for the starting frame of the sequence, its motion intensity score is... Set to 0 or the default value; Step 3-3-2: Score the exercise intensity Normalization is performed as follows and compared with the cluster score obtained in step 3-2-3 A weighted fusion is performed to obtain a comprehensive score: Where γ is the fusion weight, which is a positive hyperparameter.

7. The video 3D human pose estimation method based on a state-space recursive model according to claim 1, characterized in that, Step 4 specifically includes the following steps: Step 4-1: Input the representative spatiotemporal feature tokens obtained in Step 3 into the main state space modeling network. This model consists of multiple stacked state update units, each of which models the temporal features based on the state recursion equation. in Let be the hidden state of frame t. For the input token features, A and B are both learnable parameter matrices; Step 4-2: Using the hidden states of each layer in Step 4-1 Using the input as input, a linear mapping is performed on the outputs of each layer to obtain the instantaneous state response: in The projection matrix is ​​learnable; Step 4-3: Instantaneous state response based on step 4-2 By introducing gated linear units to selectively propagate the state, redundant feature information is suppressed, and the filtered effective state output is obtained. : in For linear transformation weights, It is a non-linear activation function. This indicates element-wise multiplication; Step 4-4: Combine the initial hidden state from Step 4-1 Compared with the output after filtering in step 4-3 The state update of the current cell is completed through residual connection and normalization: Steps 4-5: Repeat steps 4-1 to 4-4 to stack multiple state update units to complete the full sequence modeling; and at the end of the model, perform feature aggregation on the states of multiple frames to output a high-level temporal feature representation. .

8. The video 3D human pose estimation method based on a state-space recursive model according to claim 1, characterized in that, Step 5 specifically includes the following steps: Step 5-1: Based on the high-level temporal characteristics of the representative tokens output in Step 4, determine the temporal correspondence between the representative frame and the pruned frame; Step 5-2: Based on the temporal correspondence determined in Step 5-1, and based on the hidden states of representative frames adjacent to the missing frame, calculate the initial feature estimate of the missing frame through linear weighted interpolation or exponential weighted interpolation. Step 5-3: Using the initial estimate of the missing frame obtained in Step 5-2 as input, perform row-time recursive optimization through the state-space recursive equation to recover the state trajectory of the pruned frame and form a continuous temporal state flow: in This represents the state at the previous moment. A represents the input features at the current moment, and B represents the learnable parameter matrices that are independently learned within the token reconstruction module. These parameters are independent of those in the pre-lightweight state space recursive network described in step 2 and the main state space modeling network described in step 4. Step 5-4: Convert the continuous state optimized in Step 5-3 into frame-level features through linear projection; introduce lightweight convolution or linear smoothing filtering to perform temporal fusion of the frame-level features, and output a full-length feature token sequence.

9. The video 3D human pose estimation method based on a state-space recursive model according to claim 1, characterized in that, Step 6 specifically includes the following steps: Step 6-1: Perform a 3D pose regression step on the complete token sequence output in Step 5, and use one-dimensional convolution or multilayer perceptron to extract consistent dynamic temporal features across frames. Step 6-2: The dynamic temporal features obtained in Step 6-1 are transformed into a set of three-dimensional coordinates of human joints in each frame by linear transformation of the output mapping matrix and the bias vector. Step 6-3: Optimize the regression results using the joint loss function to obtain the optimized three-dimensional joint coordinates.

10. The video 3D human pose estimation method based on a state-space recursive model according to claim 1, characterized in that, Step 6-3 specifically includes the following steps: Step 6-3-1: Calculate the coordinate error loss respectively Skeleton constraint loss and time-series smoothing loss : Where F is the total number of frames in the input video sequence, and J is the total number of key points for human pose in each frame. The three-dimensional spatial coordinates of the j-th joint in the t-th frame predicted by the model are: For the corresponding The true three-dimensional coordinates This represents the squared Euclidean norm; this loss is used to constrain the deviation between the predicted coordinates and the true coordinates, resulting in the average position error per joint (MPJPE). in Let the set of edges of the human skeleton be denoted as , and each edge... This represents the index pairs of connected joints. This set is determined based on prior human anatomy and is a fixed quantity throughout the training process. The number of skeleton edges; this loss constraint ensures that the length and orientation of the skeleton in the predicted pose are consistent with the actual pose, so as to ensure the physical rationality of the human body structure; in The loss is the complete set of 3D pose coordinates predicted by the model for the t-th frame. This loss is used to constrain the smoothness of the pose of adjacent frames, reduce timing jitter, and improve motion coherence. Step 6-3-2: Establish a joint optimization objective function based on the above three losses. : in and A positive, adjustable hyperparameter that is adaptively set based on the performance of the validation set; Step 6-3-3: Based on the joint total loss function The network parameters are iteratively optimized until the model converges.

Images