Monocular video three-dimensional human body posture estimation method based on bidirectional state space model

Abstract

The invention discloses a monocular video three-dimensional human body posture estimation method based on a bidirectional state space model, and relates to the technical field of three-dimensional human body posture estimation. The method comprises the following steps: two-dimensional attitude sequence data preprocessing and position embedding: slicing a two-dimensional attitude sequence according to a specified length and performing spatial position embedding; expanding the state space model, designing a bidirectional calculation structure in space and time dimensions, and respectively learning a space relationship and a time relationship in combination with a space-time decoupling bidirectional scanning mechanism; post-processing the spatial-temporal characteristics and estimating the attitude; designing a loss function for model optimization, wherein the loss function comprises position loss and speed loss; and monocular video three-dimensional human body posture estimation is carried out based on the optimized bidirectional state space model. According to the method, the accuracy of 3D attitude estimation is improved by adopting key technologies such as extended bidirectional Mama, a space-time decoupling bidirectional scanning mechanism, a space extended bidirectional Mama block and a time extended bidirectional Mama block.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional human posture estimation, and in particular relates to a method for estimating three-dimensional human posture from a monocular video based on a bidirectional state space model. Background Art

[0002] Estimating 3D human pose from monocular video is a key task in computer vision. It aims to reconstruct the 3D spatial coordinates of human joints from a 2D skeleton. This technology has broad practical applications in areas such as action recognition, virtual reality, human-computer interaction, and autonomous driving. As these applications continue to expand, the demand for highly accurate and computationally efficient 3D pose estimation methods is growing. The recent development of deep learning has made 3D pose estimation methods based on deep neural networks increasingly mainstream.

[0003] Existing 3D pose estimation methods can be categorized based on input data type and methodology. Based on input data type, existing methods can be divided into multi-view and monocular methods. Multi-view methods rely on the synergy of multiple cameras. While they can mitigate depth ambiguity to a certain extent, they require complex equipment deployment, making them unsuitable for practical application. Monocular methods, relying on a single camera, are more practical, but they face challenges such as depth ambiguity and self-occlusion. From a methodological perspective, existing monocular 3D pose estimation methods are primarily divided into direct estimation methods and 2D-to-3D lifting methods. Direct estimation methods estimate the 3D coordinates of joints directly from the image without relying on a 2D pose detector. However, direct estimation methods typically require more complex networks and require greater computational power. Unlike direct estimation methods, 2D-to-3D lifting methods first use an off-the-shelf 2D pose detector to locate joint coordinates in the image and then use a deep network to lift these 2D coordinates into 3D space. This paper focuses on 2D-to-3D lifting methods that use monocular video as input.

[0004] In recent years, the Transformer has rapidly developed, and numerous Transformer-based 2D-to-3D lifting methods have emerged for the task of 3D human pose estimation from monocular video. The Transformer architecture, leveraging its global modeling capabilities, has been introduced to the task, improving pose estimation accuracy to a certain extent. However, these methods generally suffer from high computational complexity. Specifically, because the Transformer relies on a self-attention mechanism, the computational complexity of which increases quadratically with sequence length, resulting in excessive computational overhead when processing long sequences, hindering real-time applications. Furthermore, some methods attempt to reduce the computational burden through methods such as feature pruning and strided convolution, but these methods can compromise accuracy due to the loss of some feature information. Thus, while Transformer-based methods have improved pose estimation accuracy to a certain extent compared to previous work, they still suffer from high computational complexity, making it difficult to achieve a good balance between accuracy and computational efficiency.

[0005] Recently, state-space models have shown significant advantages in processing long sequence data. State-space models are a class of sequence modeling frameworks inspired by continuous systems. Among them, structured state-space models model long sequence dependencies with linear complexity. Mamba further proposes a data-dependent selective state-space layer based on the structured state-space sequence model. Unlike traditional models that rely on linear time-invariant state-space models, Mamba significantly improves the flexibility and adaptability of the model by introducing a selective scanning mechanism. By introducing time-varying parameters, the model allows the model to dynamically adjust parameters according to the input data, thereby enhancing the context sensitivity of the model. The selective scanning mechanism enables Mamba to selectively propagate or forget information along the sequence length dimension based on the current data. In addition, Mamba introduces a hardware-aware parallel algorithm to achieve efficient long sequence dynamic modeling. Mamba is regarded as a powerful alternative to Transformer. However, the application of state-space models in 3D human pose estimation still needs to be further explored. The main problems currently exist are as follows:

[0006] 1) In 3D human pose estimation, 2D-to-3D lifting methods still face numerous challenges. First, when mapping a 2D pose lacking depth information to a 3D pose, a single 2D pose may correspond to multiple different 3D representations, making this process ill-posed. Second, the complex variations in human pose and self-occlusion further complicate the model's ability to locate joint positions. Self-occlusion renders some human joints invisible in the image, making it difficult for the model to accurately estimate their positions, thereby reducing the model's overall accuracy.

[0007] 2) Existing spatiotemporal modeling strategies have limitations. In monocular videos, human joint positions exhibit rich and complex variations across both spatial and temporal dimensions. Existing techniques, most of which employ simple, one-way spatiotemporal modeling, fail to fully capture the complex spatiotemporal contextual dependencies of human posture and fail to fully exploit the underlying deep spatiotemporal relationships within two-dimensional posture sequences. Furthermore, coupled spatiotemporal modeling flattens the spatiotemporal dimensions, potentially confusing spatial joint constraints with temporal motion trajectories. This limits the learning of fine-grained spatiotemporal features and thus hinders model performance.

[0008] The above limitations cannot effectively address challenges inherent in monocular vision, such as depth ambiguity, complex human posture changes, and self-occlusion.

[0009] To this end, the present invention proposes a method for 3D human pose estimation from monocular video based on a bidirectional state-space model. By adopting key technologies such as extended bidirectional Mamba (ExBiMamba), spatiotemporal decoupled bidirectional scanning mechanism (DST-BS), spatially extended bidirectional Mamba block (S-ExBiMB), and temporally extended bidirectional Mamba block (T-ExBiMB), the accuracy of 3D pose estimation is improved. Summary of the Invention

[0010] The purpose of the present invention is to provide a method for estimating three-dimensional human posture from monocular video based on a bidirectional state-space model, so as to solve the problems of depth ambiguity, complex human posture changes and self-occlusion in monocular vision, which make it difficult to accurately estimate the position of human joints.

[0011] To achieve the above objectives, the present invention adopts the following technical solutions:

[0012] In a first aspect, the present invention proposes a method for estimating three-dimensional human posture from a monocular video based on a bidirectional state space model, comprising the following steps:

[0013] S1. 2D posture sequence data preprocessing and position embedding: Slice the 2D posture sequence into slices of specified length and perform spatial position embedding;

[0014] S2. Build a bidirectional state-space model for spatiotemporal feature extraction. Expand the state-space model, design a bidirectional computing structure in both spatial and temporal dimensions, and combine the spatiotemporal decoupling bidirectional scanning mechanism to learn spatial and temporal relationships respectively.

[0015] S3, spatiotemporal feature post-processing and posture estimation; based on spatiotemporal feature processing, a predicted 3D human posture sequence is obtained;

[0016] S4. Design a loss function to optimize the model; the loss function includes position loss and speed loss;

[0017] S5. Perform 3D human pose estimation from monocular video based on the optimized bidirectional state space model.

[0018] Preferably, the S1 is as follows:

[0019] For a 2D pose sequence with confidence scores Where T is the number of frames and J is the number of joints. First, the joints in each time frame are mapped to D through a linear projection layer. m dimensional feature space, generating initial feature representation Then add the spatial position embedding To preserve the location information in the spatial domain.

[0020] Preferably, the S2 is as follows:

[0021] ExBiMamba is an extension of the state-space model. It processes the input sequence forward and backward along the sequence length dimension, applying different linear projection layers and gating layers for the forward and backward processing, and then fuses the obtained forward and backward features through residual connections and addition operations.

[0022] Based on ExBiMamba, we designed a spatially extended bidirectional Mamba block and built a spatial encoder using a spatiotemporal decoupled bidirectional scanning mechanism to extract spatial features.

[0023] A time-extended bidirectional Mamba block is designed based on ExBiMamba, and a time encoder is constructed by combining the spatiotemporal decoupling bidirectional scanning mechanism to extract time features.

[0024] Preferably, the spatiotemporal feature extraction is specifically as follows:

[0025] The spatial encoder treats each joint as an independent token, and then models the spatial relationship between the joints of the human body in each frame to obtain spatial features. The spatial features are input into the temporal encoder for further processing. The temporal encoder treats each frame as an independent token, and then models the motion trajectory of each joint in each frame to obtain temporal features. The processing process is expressed as follows:

[0026] y s =S-ExBiMB(H (i-1) )

[0027] H (i) =T-ExBiMB(y s )

[0028] Combined with the spatiotemporal decoupled bidirectional scanning mechanism, S-ExBiMB and T-ExBiMB are alternately stacked to bidirectionally model the spatial dependencies between all joints in each frame and the temporal motion trajectory of each joint in all frames. The overall process is expressed as:

[0029] H (i) =T-ExBiMB(S-ExBiMB(H (i-1) ))

[0030] Among them, S-ExBiMB represents spatially extended bidirectional Mamba block, and T-ExBiMB represents temporally extended bidirectional Mamba block.

[0031] Furthermore, the processing process of the spatial encoder is as follows:

[0032] In spatial processing, S-ExBiMB treats each joint as an independent token, and the input of the i-th S-ExBiMB is Apply layer normalization and dimension reordering to accommodate processing of spatial blocks:

[0033]

[0034] in, Represents the operation of dimension reordering,

[0035] The spatial encoder adopts a spatiotemporal decoupled bidirectional scanning mechanism. Each S-ExBiMB contains two state-space models that process spatial joint features in two different directions: intra-frame forward and intra-frame backward, performing bidirectional scanning along the joint dimension to implicitly encode the spatial constraints of human joints. Layer normalization, residual connections, and convolutional feedforward neural networks are then used to enhance the fusion of spatial bidirectional scanning features.

[0036] x′ s =S-ExBiMamba(x s )

[0037]

[0038] y s =MLP(Norm(x″) s ))+x″ s

[0039] Among them, MLP represents multi-layer perceptron and ConvFFN represents convolutional feedforward neural network.

[0040] Furthermore, the processing process of the time encoder is as follows:

[0041] In time processing, T-ExBiMB treats each frame as an independent token and outputs the i-th S-ExBiMB Apply layer normalization and dimension reordering to accommodate processing in time chunks;

[0042]

[0043] in, Represents the operation of dimension reordering,

[0044] The temporal encoder uses a spatiotemporal decoupled bidirectional scanning mechanism. Each T-ExBiMB contains two state-space models that process joint-level temporal motion trajectories in two different directions: forward and backward between frames. The motion trajectory of each joint is bidirectionally scanned along the frame dimension, and the joint temporal motion pattern is learned by combining contextual information. Layer normalization, residual connections, and convolutional feedforward neural networks are then used to enhance the temporal bidirectional scanning feature fusion.

[0045] x′ t =T-ExBiMamba(x t )

[0046]

[0047] H (i) =MLP(Norm(x″) t ))+x″ t

[0048] Among them, MLP represents multi-layer perceptron, ConvFFN represents convolutional feedforward neural network; represents the feature embedding extracted by the i-th layer of the model.

[0049] Preferably, the S3 is as follows:

[0050] The spatiotemporal features are post-processed by applying a linear layer followed by a tanh activation function; the resulting feature representation is then fed into a regression head to transform the features into the final predicted 3D human pose sequence.

[0051] Preferably, the loss function in S4 includes the position loss L 3D and speed loss L v ;

[0052] L 3D is the position loss, which is used to measure the spatial error between the predicted 3D pose and the true pose;

[0053] L v is the velocity loss, which is used to constrain the temporal smoothness of the predicted 3D pose; specifically:

[0054]

[0055] in, ΔQ t =Q t -Q t-1 ;

[0056] The final loss function is defined as follows:

[0057] L=L 3D +λL v

[0058] The constant coefficient λ is a balancing factor used to balance position accuracy and time smoothness.

[0059] In a second aspect, the present invention proposes a monocular video 3D human pose estimation system based on a bidirectional state space model, comprising:

[0060] The data preprocessing module is used to slice the input 2D pose sequence into specified lengths and perform spatial position embedding;

[0061] A bidirectional state-space model, including a spatial encoder and a temporal encoder, is used to extract spatial and temporal features;

[0062] The regression head module is used to process the input spatiotemporal features and output a 3D human pose sequence estimated from the 2D pose sequence.

[0063] Compared with the prior art, the present invention has the following beneficial effects:

[0064] (1) The method of the present invention adopts the key technology of Extended Bidirectional Mamba (ExBiMamba). The present invention designs a bidirectional computing structure in the spatial and temporal dimensions, and achieves more comprehensive information modeling by applying different linear projection layers and gating layers to the forward and reverse state space models.

[0065] (2) The method of the present invention adopts the key technology of spatial extended bidirectional Mamba block (S-ExBiMB). During spatial processing, S-ExBiMB regards each joint as an independent token and adopts a spatiotemporal decoupled bidirectional scanning mechanism. There are two state space models in each S-ExBiMB, which process spatial joint features in two different directions: intra-frame forward and intra-frame backward, and perform bidirectional scanning along the joint dimension to implicitly encode the spatial constraints of human joints. Subsequently, methods such as layer normalization, residual connection and convolutional feedforward neural network are used to enhance the spatial bidirectional scanning feature fusion. This design enables the model to understand the human posture structure in each frame more carefully and capture the spatial topology between joints.

[0066] (3) The method of the present invention adopts the key technology of Temporal Extended Bidirectional Mamba Block (T-ExBiMB). In temporal processing, T-ExBiMB treats each frame as an independent token. There are two state space models in each T-ExBiMB, which process joint-level temporal motion trajectories in two different directions: inter-frame forward and inter-frame backward. The motion trajectory of each joint is bidirectionally scanned along the frame dimension, and the joint temporal motion pattern is fully learned in combination with contextual information. Subsequently, layer normalization, residual connection and convolutional feedforward neural network are used to enhance the temporal bidirectional scanning feature fusion. This design enables the model to learn temporal correlation more effectively, thereby better understanding the dynamic changes of human posture.

[0067] (4) The method of the present invention adopts the key technology of the spatiotemporal decoupled bidirectional scanning mechanism (DST-BS). In the spatial dimension, the joint dependencies within the frame are modeled by bidirectional scanning along the joints. In the temporal dimension, bidirectional scanning is performed along the frame, and the joint-level temporal motion trajectory between frames is modeled in combination with context information. In this way, while ensuring the independence of the spatial and temporal domains, the present invention can capture fine-grained spatiotemporal correlations more comprehensively and efficiently, thereby improving the accuracy of 3D pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 Figure 1 shows the overall model architecture of the bidirectional state-space model in the present invention ((a) overall model architecture of the bidirectional state-space model, with L spatiotemporal blocks stacked alternately; (b) model architecture of S-ExBiMB; (c) model architecture of T-ExBiMB; (b) and (c) independently modeling spatiotemporal dependencies);

[0069] Figure 2 This is the overall model architecture diagram of ExBiMamba in the present invention;

[0070] Figure 3 Schematic diagram illustrating the bidirectional modeling of spatiotemporal dependence in the present invention ((a) represents the spatial dimension; (b) represents the temporal dimension);

[0071] Figure 4 A comparison chart of different space-time scanning mechanisms. DETAILED DESCRIPTION

[0072] The following will provide a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0073] Example 1:

[0074] The method for estimating 3D human pose from monocular video based on a bidirectional state space model adopts a bidirectional state space model. The entire model processing process can be divided into four main stages: 2D pose sequence data preprocessing and position embedding, spatial feature extraction, temporal feature extraction, feature post-processing and pose estimation. The overall architecture of the model proposed in this invention is as follows: Figure 1 shown.

[0075] A method for estimating 3D human pose from a monocular video based on a bidirectional state space model comprises at least the following steps:

[0076] Step 1: 2D posture sequence data preprocessing and position embedding.

[0077] The 2D to 3D lifting method uses a 2D key point sequence detected by an existing 2D posture detector as input. The present invention uses the 2D posture detection results of the 2D posture detection network Stacked Hourglass on Human n3.6M.

[0078] First, the two-dimensional posture sequence is sliced into specified lengths, such as 9, 27, 81, 243, to facilitate subsequent model training. Specifically, the present invention uses a two-dimensional sequence with a confidence score As input, where T is the number of frames and J is the number of joints. First, the joints in each time frame are mapped to D through a linear projection layer. m dimensional feature space, generating initial feature representation Then add the spatial position embedding To preserve the position information in the spatial domain. It is worth noting that although the present invention does not add time position embedding, the bidirectional scanning method designed based on the state space model can implicitly preserve the temporal order between posture sequences. No explicit time position embedding is required.

[0079] Step 2: Extract spatiotemporal features.

[0080] A spatiotemporal encoder is used to extract spatiotemporal features. The posture features obtained after the above processing are input into the spatial encoder. The spatial encoder treats each joint as an independent token and then models the spatial relationship between the human joints in each frame. The feature representation obtained from the spatial encoder is input into the temporal encoder for further processing. The temporal encoder treats each frame as an independent token and then models the motion trajectory of each joint in each frame. The spatiotemporal encoder can capture the deep spatiotemporal dependencies in complex posture sequences. The two work closely together, and through a complex and sophisticated computational process, they comprehensively and efficiently capture the complex dependencies of human posture in the spatiotemporal dimensions, providing rich and critical feature information for subsequent posture estimation.

[0081] The present invention first introduces ExBiMamba. The original unidirectional Mamba was mainly used for unidirectional processing of one-dimensional sequence data and has limitations in more complex visual tasks. Vision Mamba introduces a general bidirectional state space model to process flattened image block sequences. Inspired by this, in order to achieve efficient and accurate three-dimensional human pose estimation, the present invention expands the state space model and introduces ExBiMamba. In the task of 3D human pose estimation from monocular video, the position of human joints has rich and complex changes in the spatial and temporal dimensions. The present invention designs a bidirectional computing structure in the spatial and temporal dimensions to more comprehensively learn the spatiotemporal dependencies of the input two-dimensional pose sequence. Unlike ordinary bidirectional state space models, the present invention applies different linear projection layers and gating layers to the forward and reverse state space models, such as Figure 2 shown.

[0082] Specifically, given an input sequence Where B is the batch size, L is the length of the sequence, and D m is the model dimension. ExBiMamba processes the input sequence x forward and backward along the sequence length dimension respectively:

[0083] x f =SSM f (σ(Conv1d f (xW f1 ))

[0084] x b =SSM b (σ(Conv1d b (flip(x)W b1 ))

[0085] in, The input is mapped into the expanded state dimension E through a separate linear layer. flip(·) represents the flip operation along the sequence length dimension. Conv1d f (·) and Conv1d b (·) are forward and backward causal one-dimensional convolution, SSM f (·) and SSM b (·) denotes the computation process of the forward and reverse state space models, respectively. σ(·) represents the SiLU activation function.

[0086] Next, the forward and backward features obtained by the model are conditioned by independent multiplication gate layers, and then the feature dimensions are mapped back to the model dimension D through different linear layers. m :

[0087] x fg =[x f ⊙σ(xWf2 )]W f3

[0088] x bg =flip{[x b ⊙σ(flip(x)W b2 )]W b3}

[0089] in, ⊙ represents the Hadamard product.

[0090] Finally, the forward and backward features are fused through residual connections and addition operations:

[0091] x E =Norm(x+x fg )+Norm(x+x bg )

[0092] in, Norm(·) represents layer normalization.

[0093] Based on ExBiMamba, the present invention further constructs spatial ExBiMamba (S-ExBiMamba) and temporal ExBiMamba (T-ExBiMamba) to learn spatial and temporal relationships respectively by combining the spatiotemporal decoupling bidirectional scanning mechanism, which can be expressed as x′ s =S-ExBiMamba(x s ), x′ t =T-ExBiMamba(x t ),in is the spatial feature of each frame, is the time characteristic of each joint.

[0094] The spatial-temporal decoupled bidirectional scanning mechanism (DST-BS) is introduced. Different from previous spatial-temporal modeling methods, DST-BS performs joint-level bidirectional scanning on the input posture sequence in both spatial and temporal dimensions, such as Figure 3As shown (spatial dimension: forward modeling of human topological constraints along the motion chain from the root joint → the end of the limb (right hand), and reverse strengthening of the feedback correction of the end of the limb to the torso; time dimension: bidirectional scanning of the motion trajectory of each joint from the first frame to the last frame, learning the natural temporal evolution law of the posture and the potential constraints of future information on the current state). Specifically, in the spatial dimension, bidirectional scanning is performed along the joints to model the joint dependencies within the frame. In the time dimension, bidirectional scanning is performed along the frame, and the joint-level temporal motion trajectory between frames is modeled in combination with contextual information. In this way, the present invention can capture fine-grained spatiotemporal correlations more comprehensively and efficiently while ensuring the independence of the spatial and temporal domains, thereby improving the accuracy of 3D posture estimation.

[0095] (1) Construct a spatial extended bidirectional Mamba block (Spatial Extended Bidirectional MambaBlock, S-ExBiMB) for spatial feature extraction.

[0096] Since the state space model of ExBiMamba is mainly based on linear calculations, although it is efficient, it may limit the model's capture of the nonlinear dynamic characteristics of joint motion. Therefore, it is necessary to appropriately introduce additional nonlinear calculations on the basis of ExBiMamba to enhance the model's capture of high-level information such as potential posture structure, and achieve a balance between computational efficiency and estimation accuracy. In addition, in order to further enhance the fusion of bidirectional scanning features, layer normalization, residual connection, convolutional feed-forward neural network (Convolutional Feed-Forward Network, referred to as ConvFFN) and other operations are introduced. Subsequently, nonlinear calculations are introduced by adding a multilayer perceptron (MLP) with residual connection and layer normalization. Among them, ConvFFN is defined as:

[0097] ConvFFN(x E )=x E +Conv1 1×1 (ReLU(Conv1 1×1 (x E )))

[0098] Based on the above analysis, the present invention further designed S-ExBiMB, such as Figure 1 (b) In spatial processing, S-ExBiMB treats each joint as an independent token, and the input of the i-th S-ExBiMB Apply layer normalization and dimension reordering to accommodate processing of spatial blocks:

[0099]

[0100] in, Represents the Rearrange operation,

[0101] It is worth noting that the present invention adopts a time-space decoupled bidirectional scanning mechanism. There are two state space models in each S-ExBiMB, processing spatial joint features in two different directions: intra-frame forward and intra-frame backward, such as Figure 3 As shown in (a), bidirectional scanning is performed along the joint dimension to implicitly encode the spatial constraints of the human joint. Then, layer normalization, residual connection and convolutional feedforward neural network are used to enhance the spatial bidirectional scanning feature fusion:

[0102] x′ s =S-ExBiMamba(x s )

[0103]

[0104] y s =MLP(Norm(x″) s ))+x″ s

[0105] This design enables the model to understand the human body posture structure in each frame more carefully and capture the spatial topology between joints. The processing of S-ExBiMB can be expressed as:

[0106] y s =S-ExBiMB(H (i-1) )

[0107] (2) Construct a temporal extended bidirectional Mamba block (T-ExBiMB) for temporal feature extraction.

[0108] Similar to the spatial dependency bidirectional modeling, the present invention further designs T-ExBiMB based on ExBiMamba. Figure 1 (c) In temporal processing, T-ExBiMB treats each frame as an independent token and outputs the i-th S-ExBiMB Layer normalization and dimension reordering are applied to accommodate processing in time chunks.

[0109]

[0110] in, There are two state-space models in each T-ExBiMB, processing joint-level temporal motion trajectories in two different directions: forward and backward between frames, as shown in Figure 3As shown in (b), the motion trajectory of each joint is bidirectionally scanned along the frame dimension, and the temporal motion pattern of the joint is fully learned by combining contextual information. Subsequently, layer normalization, residual connections, and convolutional feedforward neural networks are used to enhance the temporal bidirectional scanning feature fusion.

[0111] x′ t =T-ExBiMamba(x t )

[0112]

[0113] H (i) =MLP(Norm(x″) t ))+x″ t

[0114] in, represents the feature embedding extracted by the i-th layer of the model.

[0115] This design enables the model to learn temporal correlations more effectively, thereby better understanding the dynamic changes of human posture. The processing of T-ExBiMB can be expressed as:

[0116] H (i) =T-ExBiMB(y s )

[0117] In summary, this paper combines DST-BS with alternating stacking of S-ExBiMB and T-ExBiMB to bidirectionally model the spatial dependencies between all joints in each frame and the temporal motion trajectory of each joint across all frames. The overall process can be expressed as:

[0118] H (i) =T-ExBiMB(S-ExBiMB(H (i-1) ))

[0119] Through this alternating design, the present invention can more comprehensively and efficiently capture fine-grained spatiotemporal correlations while ensuring the independence of spatial and temporal domains, thereby improving the accuracy of 3D pose estimation.

[0120] Step 3: Feature post-processing and pose estimation.

[0121] The rich spatiotemporal features extracted by the spatiotemporal encoder are further processed. First, the features are mapped to a higher dimension by applying a linear layer and a tanh activation function. The resulting feature representation is then input into the regression head, which processes the features and converts them into the final 3D human pose estimation result.

[0122] Step 4: Model training.

[0123] In order to ensure that the predicted 3D posture can achieve ideal results in both spatial position accuracy and temporal smoothness, the present invention introduces the position loss L 3D and speed loss L v . L 3D is the position loss, which is used to measure the spatial error between the predicted 3D pose and the true pose; L v It is a velocity loss that is used to constrain the temporal smoothness of the predicted 3D pose and avoid unnatural jitter in the time series. It is defined as follows:

[0124]

[0125] in, ΔQ t =Q t -Q t-1 .

[0126] The final loss function is defined as follows:

[0127] L=L 3D +λL v

[0128] The constant coefficient λ is a balancing factor used to balance position accuracy and time smoothness.

[0129] During model training, each mini-batch was set to 4 sequences. The AdamW optimizer was used to optimize the network parameters, with 100 epochs trained and a weight decay of 0.01. The initial learning rate was set to 5e-4 with an exponential decay schedule and a decay factor of 0.99.

[0130] Based on the above invention content, the following technical effect evaluation is proposed:

[0131] (1) Performance evaluation of the spatiotemporal decoupled bidirectional scanning mechanism:

[0132] In order to better demonstrate the advantages of the spatiotemporal decoupling bidirectional scanning mechanism of the present invention, the present invention conducts experiments on different spatiotemporal scanning methods, such as Figure 4 shown.

[0133] in, Figure 4 (a) represents a one-way scan of space and time, Figure 4 (b) represents space-first spatiotemporal hybrid bidirectional scanning: spatial joints are processed frame by frame, and the spatiotemporal dimension (TJ) is flattened for bidirectional scanning. Figure 4 (c) represents time-first spatiotemporal hybrid bidirectional scanning: the motion trajectory of each joint is processed in joint order, and the spatiotemporal dimension (JT) is flattened for bidirectional scanning. Figure 4 (d) Represents spatiotemporal hybrid bidirectional scanning: the combination of (b) SF-STH-BS and (c) TF-STH-BS. Figure 4 (e) represents the time-space separation bidirectional scanning proposed and used by the present invention: independent bidirectional scanning is performed in the time and space domains. The results show that the present invention is more superior.

[0134] (2) Performance evaluation of the overall model:

[0135] The present invention is experimented and evaluated on two large-scale 3D human pose estimation datasets, namely Human3.6M and MPI-INF-3DHP.

[0136] Human3.6M is one of the largest indoor datasets for 3D human pose estimation, containing 15 categories of daily activities performed by 11 subjects under 4 camera views, totaling approximately 3.6 million video frames. To ensure the fairness of the evaluation, the present invention follows the standard method, using the data of subjects S1, S5, S6, S7 and S8 to train the model, and testing it on the data of subjects S9 and S11. Referring to previous studies, the present invention uses two protocols for evaluation. The first protocol (called P1) calculates the mean joint position error (MPJPE, unit: mm) after alignment of the root joint (sacrum). The second protocol (called P2) calculates the Procrustes-MP JPE after aligning the predicted pose with the true pose through rigid transformation.

[0137] MPI-INF-3DHP contains 1.3 million video frames covering eight activities of eight subjects, captured in a variety of environments: green screen, non-green screen, and outdoor. Following previous work, reported evaluation metrics include MPJPE, percentage of correct joints within a 150mm threshold (PCK), and area under the precision-recall curve (AUC).

[0138] The present invention constructs four models with different parameter configurations, as shown in Table 1 (L: number of layers, D m : model dimension, T: number of input frames). Among them, ours-B achieves a good balance between accuracy and computational efficiency. Each model variant is named according to its parameter number and computing power requirements, and users can flexibly choose according to specific practical application requirements such as real-time processing and high-precision estimation. In all experiments, the expansion layer ratio of each MLP is α = 4. In addition, the expanded state dimension E in the SSM is set to be equal to the model dimension D m .

[0139] Table 1 Detailed information of the model variants of the present invention

[0140] Method L <![CDATA[D m ]]> T Params MACs ours-XS 12 64 243 2.8M 8.3G ours-S 20 64 243 4.6M 13.7G ours-B 12 128 243 10.4M 32.8G ours-L 16 128 243 13.9M 43.7G

[0141] We compared the performance of our proposed method with state-of-the-art algorithms on the Human3.6M dataset. To ensure a fair comparison, we only used models that were not pre-trained on additional data. The results are shown in Table 2, where T represents the number of input frames; Seq2Seq represents the estimated 3D pose sequence; P1 represents the MPJPE error (mm); P2 represents the P-MPJPE error (mm); represents the P1 error on the 2D ground truth; (*) indicates 2D pose estimation using HRNet.

[0142] Table 2: Quantitative comparison on Human3.6M

[0143]

[0144] As shown in Table 2, ours-L achieves a P1 error of 36.4mm in the estimated 2D pose and 11.7mm in the true 2D pose. Notably, compared to the previous state-of-the-art model, MotionBERT, this model reduces parameters by 67.1%, saves 75% of multiply-accumulate operations (MACs), and improves accuracy by 2.8mm and 2.0mm, respectively. Compared to another advanced model, MotionAGFormer, our model reduces parameters by 26.8%, saves 44.2% of multiply-accumulate operations (MACs), and improves accuracy by 2.0mm and 1.6mm, respectively.

[0145] The generalization capability of the present invention was verified on the MPI-INF-3DHP dataset. Considering the shorter nature of video sequences, the present invention adjusted the model to accommodate inputs of 27 and 81 frames, as shown in Table 3.

[0146] Table 3 Quantitative comparison on MPI-INF-3DHP

[0147] Method T PCK↑ AUC↑ MPJPE↓ MHFormer 9 93.8 63.3 58.0 P-STMO 81 97.9 75.8 32.2 MixSTE 27 94.4 66.5 54.9 PoseFormerV2 81 97.8 78.8 27.8 STCFormer 81 98.7 83.9 23.1 GLA-GCN 9 98.5 79.1 27.7 DualFormer 81 97.8 73.4 40.1 FTCM 81 97.9 79.8 31.2 GLSTE 81 98.2 79.0 27.4 Zhong et al. 81 98.6 71.5 39.4 MoAGFormer-XS 27 98.2 83.5 19.2 MoAGFormer-S 81 98.3 84.5 19.2 MoAGFormer-B 81 98.3 84.5 17.1 MoAGFormer-L 81 98.2 85.3 18.2 KTPFormer 81 98.9 85.9 16.7 PoseMamba 27 98.0 87.1 17.8 TCPFormer 81 99.0 87.7 15.0 ours-XS 27 98.7 85.1 17.0 ours-S 81 98.6 86.0 15.7 ours-B 81 98.6 86.6 14.7 ours-L 81 99.3 87.0 13.5

[0148] As shown in Table 3, our method demonstrates excellent performance across all variants, particularly in the MPJPE metric, where it consistently outperforms other methods. When T = 81, ours-L achieves a P1 error of 13.5 mm and a PCK of 99.3%, respectively, which are 1.5 mm lower and 0.3% higher than the optimal model. Furthermore, the AUC metric reaches 87.0%. Furthermore, when T = 27, our method surpasses other methods in all metrics. These results demonstrate the strong generalization capabilities of our method.

[0149] It can be seen that the present invention achieves the current state-of-the-art performance level with fewer parameters and lower MACs, achieving a good balance between accuracy and computational efficiency.

[0150] The above description is only used to help understand the method and core essence of the present invention, but the scope of protection of the present invention is not limited thereto. For those skilled in the art, equivalent replacements or modifications based on the technical solutions and inventive concepts of the present invention within the technical scope disclosed by the present invention should be included in the scope of protection of the present invention. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A method for estimating 3D human pose from monocular video based on a bidirectional state space model, characterized in that: The steps include: S1. 2D posture sequence data preprocessing and position embedding: Slice the 2D posture sequence into slices of specified length and perform spatial position embedding; S2, construct a bidirectional state space model for spatiotemporal feature extraction; Expand the state space model, design a bidirectional computing structure in the spatial and temporal dimensions, and combine the spatiotemporal decoupling bidirectional scanning mechanism to learn spatial and temporal relationships respectively; S3, spatiotemporal feature post-processing and posture estimation; based on spatiotemporal feature processing, a predicted 3D human posture sequence is obtained; S4. Design a loss function to optimize the model; the loss function includes position loss and speed loss; S5. Perform 3D human pose estimation from monocular video based on the optimized bidirectional state space model.

2. The method for estimating 3D human pose from monocular video according to claim 1, wherein: The S1 is specifically as follows: For a 2D pose sequence with confidence scores Where T is the number of frames and J is the number of joints. First, the joints in each time frame are mapped to D through a linear projection layer. m dimensional feature space, generating initial feature representation Then add the spatial position embedding To preserve the location information in the spatial domain.

3. The method for estimating 3D human pose from monocular video according to claim 1, wherein: The S2 is specifically as follows: ExBiMamba is an extension of the state-space model. It processes the input sequence forward and backward along the sequence length dimension, applying different linear projection layers and gating layers for the forward and backward processing, and then fuses the obtained forward and backward features through residual connections and addition operations. Based on ExBiMamba, we designed a spatially extended bidirectional Mamba block and built a spatial encoder using a spatiotemporal decoupled bidirectional scanning mechanism to extract spatial features. A time-extended bidirectional Mamba block is designed based on ExBiMamba, and a time encoder is constructed by combining the spatiotemporal decoupling bidirectional scanning mechanism to extract time features.

4. The method for estimating 3D human pose from monocular video according to claim 3, wherein: The spatiotemporal feature extraction is specifically as follows: The spatial encoder treats each joint as an independent token and then models the spatial relationship between the joints of the human body in each frame to obtain spatial features. The spatial features are input into the temporal encoder for further processing. The temporal encoder treats each frame as an independent token and then models the motion trajectory of each joint in each frame to obtain temporal features. Combined with the spatiotemporal decoupled bidirectional scanning mechanism, S-ExBiMB and T-ExBiMB are alternately stacked to bidirectionally model the spatial dependencies between all joints in each frame and the temporal motion trajectory of each joint in all frames, respectively. Among them, S-ExBiMB represents the spatially extended bidirectional Mamba block, and T-ExBiMB represents the temporally extended bidirectional Mamba block.

5. The method for estimating 3D human pose from monocular video according to claim 4, wherein: The processing process of the spatial encoder is as follows: In spatial processing, S-ExBiMB treats each joint as an independent token, and the input of the i-th S-ExBiMB is Apply layer normalization and dimension reordering to accommodate processing of spatial blocks: in, Represents the operation of dimension reordering, The spatial encoder adopts a spatiotemporal decoupled bidirectional scanning mechanism. Each S-ExBiMB contains two state-space models that process spatial joint features in two different directions: intra-frame forward and intra-frame backward, performing bidirectional scanning along the joint dimension to implicitly encode the spatial constraints of human joints. Layer normalization, residual connections, and convolutional feedforward neural networks are then used to enhance the fusion of spatial bidirectional scanning features. x′ s =S-ExBiMamba(x s ) y s =MLP(Norm(x″ s ))+x″ s Among them, MLP represents multi-layer perceptron and ConvFFN represents convolutional feedforward neural network.

6. The method for estimating 3D human pose from monocular video according to claim 4, wherein: The processing process of the time encoder is as follows: In time processing, T-ExBiMB treats each frame as an independent token and outputs the i-th S-ExBiMB Apply layer normalization and dimension reordering to accommodate processing in time chunks; in, Represents the operation of dimension reordering, The temporal encoder uses a spatiotemporal decoupled bidirectional scanning mechanism. Each T-ExBiMB contains two state-space models that process joint-level temporal motion trajectories in two different directions: forward and backward between frames. The motion trajectory of each joint is bidirectionally scanned along the frame dimension, and the joint temporal motion pattern is learned by combining contextual information. Layer normalization, residual connections, and convolutional feedforward neural networks are then used to enhance the temporal bidirectional scanning feature fusion. x′ t =T-ExBiMamba(x t ) H (i) =MLP(Norm(x″ t ))+x″ t Among them, MLP represents multi-layer perceptron, ConvFFN represents convolutional feedforward neural network; represents the feature embedding extracted by the i-th layer of the model.

7. The method for estimating 3D human pose from monocular video according to claim 1, wherein: The S3 is as follows: The spatiotemporal features are post-processed by applying a linear layer followed by a tanh activation function; the resulting feature representation is then fed into a regression head to transform the features into the final predicted 3D human pose sequence.

8. The method for estimating 3D human pose from monocular video according to claim 1, wherein: The loss function in S4 includes the position loss L 3D and speed loss L v ; L 3D is the position loss, which is used to measure the spatial error between the predicted 3D pose and the true pose; L v is the velocity loss, which is used to constrain the temporal smoothness of the predicted 3D pose; specifically: in, The final loss function is defined as follows: L=L 3D +λL v The constant coefficient λ is a balancing factor used to balance position accuracy and time smoothness.

9. A monocular video 3D human pose estimation system based on a bidirectional state space model applied to the method of claim 1, characterized in that: include: The data preprocessing module is used to slice the input 2D pose sequence into specified lengths and perform spatial position embedding; A bidirectional state-space model, including a spatial encoder and a temporal encoder, is used to extract spatial and temporal features; The regression head module is used to process the input spatiotemporal features and output a 3D human pose sequence estimated from the 2D pose sequence.

Images