Three-dimensional human body posture estimation method based on hybrid architecture space-time modeling

Abstract

The invention discloses a three-dimensional human body posture estimation method based on hybrid architecture space-time modeling, and the method comprises the steps: obtaining historical human body posture data, and generating a three-dimensional human body posture estimation network training set; constructing a three-dimensional human body posture estimation network model, wherein the network model comprises a space-time position embedding module, a space-time Mama block, a space-time self-attention block, a full connection layer and a regression layer; based on the estimation network training set and a loss function training network model, learning a mapping relation from a two-dimensional attitude joint point sequence to a three-dimensional attitude joint point sequence; and inputting two-dimensional attitude joint point sequence data to be estimated into the trained network model, and outputting a three-dimensional attitude joint point sequence. According to the method, the global spatio-temporal features are rapidly extracted by using Mama, then the global spatio-temporal feature information is further supplemented by using Transform, and the spatio-temporal features are fused and complemented more effectively by using a hybrid architecture.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and artificial intelligence technology, and specifically to a method for estimating three-dimensional human body posture based on deep learning, and more particularly to a method for estimating three-dimensional human body posture based on hybrid architecture spatiotemporal modeling. Background Art

[0002] Human 3D pose estimation is a core task in computer vision and artificial intelligence. It aims to accurately recover the 3D position of human joints from images or videos, and has important applications in intelligent monitoring, human-computer interaction, motion analysis, and autonomous driving. Currently, deep learning has become the mainstream research method. While convolutional neural networks have improved the accuracy of 2D pose estimation, 3D pose estimation requires the effective use of temporal information. Existing temporal modeling methods all have limitations: recurrent neural networks and their variants suffer from vanishing and exploding gradients, making it difficult to model long-range dependencies, and sequential computations restrict parallelism, resulting in low efficiency. Convolutional neural networks have a limited receptive field and require multiple layers to capture long-range spatiotemporal relationships, resulting in complex models and a large number of parameters. While the Transformer can efficiently model global dependencies, the computational complexity of its self-attention mechanism is proportional to the square of the sequence length, resulting in high computational complexity and memory usage when processing long videos. Furthermore, its "fully connected" nature easily introduces redundant associations, reducing the efficiency of long sequence processing.

[0003] In recent years, Mamba, a structured state-space sequence model, has shown great potential in modeling long sequences. It incorporates a selective state-space and hardware-aware parallel scanning algorithm, whose computational complexity scales linearly with sequence length. This offers significant computational and memory efficiency advantages when processing extremely long sequences, and it can effectively capture long-range dependencies. However, its capabilities may be inferior to the Transformer's self-attention mechanism in modeling global, fine-grained interactions (such as the precise coordinated motion of joints in human posture).

[0004] The deep integration of Transformers and Mamba for 3D human pose estimation is still in its exploratory stages, and there is a lack of effective hybrid mechanisms designed to address the spatiotemporal characteristics of human pose, such as hierarchical structures and joint motion constraints. Achieving real-time or near-real-time 3D pose estimation while maintaining high accuracy and leveraging Mamba's efficiency remains a challenge.

[0005] Existing pure Transformer-based temporal modeling methods suffer from high computational complexity, high resource consumption, and poor real-time performance when processing long video sequences. While the emerging Mamba model is highly efficient, it lacks the ability to capture the complex global spatial structure of human poses and the fine-grained coordination of joints. Therefore, innovative model architectures are urgently needed to combine the advantages of both approaches, improve the accuracy and efficiency of 3D pose estimation, and better adapt to practical application scenarios. This invention aims to address these issues. Summary of the Invention

[0006] The present invention aims to solve the technical problems existing in the related art at least to a certain extent.

[0007] The purpose of this invention is to address the problems and shortcomings of existing methods and propose a 3D human pose estimation method based on hybrid architecture spatiotemporal modeling, which fully combines the Transformer and Mamba models to achieve high-precision human pose estimation while maintaining a low parameter count.

[0008] In order to achieve the above-mentioned object, the present invention provides a method for estimating 3D human posture based on hybrid architecture spatiotemporal modeling, comprising the following steps:

[0009] S1. Acquire historical human posture data and generate a 3D human posture estimation network training set, wherein the human posture data includes a 2D posture joint coordinate sequence and a corresponding 3D posture joint coordinate sequence true value;

[0010] S2. Construct a 3D human pose estimation network model, wherein the 3D human pose estimation network model includes a sequentially arranged spatiotemporal position embedding module, a plurality of alternately stacked spatiotemporal Mamba blocks, a spatiotemporal self-attention block including a Transformer spatiotemporal unit, a fully connected layer, and a regression layer;

[0011] S3. Train the network model based on the 3D human pose estimation network training set and loss function, learn the mapping relationship from the 2D pose joint point sequence to the 3D pose joint point sequence, and continuously adjust the network model parameters through the back-propagation algorithm to minimize the loss function;

[0012] S4. Input the 2D posture joint point sequence data to be estimated into the trained network model, and output the 3D posture joint point sequence to complete the 3D human posture estimation task.

[0013] Preferably, the three-dimensional human posture estimation network training set is ,in is the 2D posture joint coordinate sequence, is the true value of the corresponding three-dimensional posture joint coordinate sequence, is the sequence length, is the number of joints.

[0014] Preferably, the spatiotemporal position embedding module is used to integrate prior knowledge of human skeleton structure and human kinematics, and is composed of a spatial learnable topological coding layer and a temporal learnable topological coding layer connected in series;

[0015] The spatiotemporal Mamba block is used to preliminarily extract global spatiotemporal features, and is composed of a number of Mamba spatiotemporal units connected in series, wherein each Mamba spatiotemporal unit is composed of a spatial Mamba block and a temporal Mamba block connected in series;

[0016] The spatiotemporal self-attention block is used to further supplement the global spatiotemporal feature information and is composed of several Transformer spatiotemporal units in series, where each Transformer spatiotemporal unit is composed of a spatial Transformer block and a temporal Transformer block in series;

[0017] The fully connected layer is used to expand and deepen the features by mapping the input features to a higher dimension to increase the nonlinear capacity of the model. The input feature dimension is , the output latent feature dimension is , the activation function adopts Tanh;

[0018] The regression layer is used to convert the feature vector output by the fully connected layer into a three-dimensional coordinate sequence prediction value.

[0019] Preferably, the spatially learnable topological coding layer is used to integrate prior knowledge of the human skeleton structure, and the specific steps are as follows:

[0020] Step 1: Based on the two-dimensional posture joint point sequence X, the output dimension is The linear layer maps it to a high-dimensional space and obtains features ;

[0021] Step 2: Based on the above features , using the element-by-element multiplication method to obtain the spatial learnable topological coding layer output features , the specific calculation method is as follows:

[0022]

[0023] in, express The features at the tth time step, , represents element-wise multiplication, Represents the learnable weights, all T time steps Merge , Represents the spatial adjacency matrix, and its construction formula is as follows:

[0024]

[0025] Among them, the spatial topological structure matrix Represents the human skeleton model The connection relationship between the key joints, the matrix for The square matrix, the rows and columns correspond to the serial numbers of the corresponding joints. According to the human body structure prior, when the and When there is an anatomical connection between the joints, , when the and When there is no anatomical connection between the joints, ; Learnable matrix for The internal elements of the matrix are updated through network training to express the implicit connections between joints.

[0026] Preferably, the temporal learnable topological coding layer is used to integrate human kinematics priors and is constructed by the following steps:

[0027] Step 1: Output features based on spatially learnable topological coding layer , and exchange its space-time dimension to , and then define the inter-frame similarity matrix of the j-th joint and the temporal adjacency matrix of the j-th joint :

[0028]

[0029]

[0030] in, express The temporal features of the j-th joint in , for The transpose of is the time topology matrix, the matrix The rows and columns correspond to the serial numbers of the corresponding time series; select the matrix The first two largest elements of each row are recorded in The position in the time topology matrix The elements at the corresponding positions are set to 1, and the rest of the elements are set to 0; the matrix can be learned The internal elements of are dynamically updated through training to represent the implicit connections between different joints in the temporal sequence;

[0031] Step 2: Calculate the output features of the temporal learnable topological coding layer based on the temporal adjacency matrix:

[0032]

[0033] in, are learnable weights, Represents element-wise multiplication; all J joint dimensions After merging, the time and space dimensions are exchanged to obtain the output features of the time-learnable topological coding layer .

[0034] Preferably, the spatial Mamba block is used to process the spatial features of J joints in a single time frame, and its specific operation steps are as follows:

[0035] Step 1: Get the input features, normalize them and process them with Mamba TokenMixer to get :

[0036]

[0037] Where n is the sequence number of the Mamba space-time unit in the space-time Mamba block, n=1,2,3....N, N is the number of Mamba space-time units in the space-time Mamba block, is the input feature, is the input feature Normalized features;

[0038] The Mamba token mixer is used to mix features and includes two branches;

[0039] Branch 1: Features First, through the linear projection layer of branch one , the input features Projected to dimension The output is obtained by passing through the one-dimensional convolution layer and the activation function SiLU and finally through the selective scanning state space model layer SSML. , the specific process can be expressed as:

[0040] ;

[0041] Branch 2, First pass the linear projection layer , and then the output is obtained by the same one-dimensional convolution layer and SiLU activation function , the specific process can be expressed as:

[0042] ;

[0043] The outputs of branch one and branch two are then concatenated and passed through the final linear projection layer to obtain the output of the token mixer. , the specific formula is:

[0044] ;

[0045] Step 2: After normalization, the input is the multi-layer perceptron MLP layer, and the output result is added through the residual connection As the output of the final spatial Mamba block , the specific formula is:

[0046] ;

[0047] The MLP layer includes two linear transformation layers, the activation function is GELU, and Dropout is added to prevent network overfitting.

[0048] Preferably, the temporal Mamba block is used to process the temporal features of a single joint in T frames in the temporal dimension, and the specific steps are as follows:

[0049] Step 1: Get input features , and exchange its space-time dimension, becomes ;

[0050] Step 2: Input features First normalize, then pass through the token mixer, and then add it to the input feature to get the intermediate result. The intermediate result is normalized and used as the input of the multi-layer perceptron MLP layer. The output of the MLP layer is added to the intermediate result through the residual connection as the extracted time feature;

[0051] Step 3: Exchange the extracted time features with the time and space dimensions. The dimensions are consistent, and the temporal Mamba block output features are obtained.

[0052] Preferably, the spatial Transformer block is used to process the spatial features of J joints in a single time frame, and the specific steps are: the input features are first normalized, then processed by the token mixer, and then added to the input features to obtain an intermediate result, the intermediate result is normalized as the input of the multi-layer perceptron MLP layer, the output of the MLP layer is added to the intermediate result through the residual connection, and is used as the output of the spatial Transformer block in the e-th Transformer spatiotemporal unit , where e=1,2,3....E, E is the number of Transformer spatiotemporal units in the spatiotemporal self-attention block;

[0053] The spatial Transformer block uses a spatial multi-head self-attention module as a token mixer to calculate the global spatial correlation of key points in the same frame, that is, to linearly map its input features to a query vector , key vector , and the value vector , used to calculate K self-attention heads, the features after the self-attention heads are concatenated are transformed by the linear transformation matrix After projection, a Dropout layer is applied to prevent the network from overfitting, and the output of the token mixer is obtained:

[0054]

[0055] in, is the kth self-attention head, the linear transformation matrix It is implemented by a fully connected layer with both input and output feature dimensions of 128, which is used to fuse the features of multiple self-attention heads.

[0056] Preferably, the temporal Transformer block is used to process the temporal features of a single joint in T frames in the temporal dimension, and the specific steps are as follows:

[0057] Step 1: Input features Exchange time and space dimensions, becomes ;

[0058] Step 2: Input features First normalize, then pass through the token mixer, and then add it to the input feature to get the intermediate result. The intermediate result is normalized and used as the input of the multi-layer perceptron MLP layer. The output of the MLP layer is added to the intermediate result through the residual connection as the extracted time feature;

[0059] Step 3: Exchange the extracted time features with the time and space dimensions. The dimensions are consistent, and the temporal Transformer block output features are obtained.

[0060] As a preference, in step S3, the network model parameters are continuously adjusted by the back propagation algorithm, including: using the gradient descent method to perform back propagation training, and the loss function constrains the difference between the predicted value and the true value of the three-dimensional human posture coordinate; the loss function includes the joint position loss and motion smoothing loss :

[0061]

[0062]

[0063] Among them, the parameters is the length of the sequence, is the number of joints, For the Frame, The predicted coordinate values ​​of the joints, For the Frame, The true value of the coordinates of the joints, , , and the final total loss function is:

[0064]

[0065] Among them, the hyperparameters Used to balance posture accuracy and motion smoothness.

[0066] Beneficial effects: 1. The present invention supplements the human skeleton structure prior and human kinematics prior through the spatiotemporal position embedding module, assisting the network to efficiently and high-quality model the three-dimensional posture characteristics of the human body.

[0067] 2. This paper leverages the advantages of both models. Mamba is used to rapidly extract global spatiotemporal features, and the Transformer is used to further supplement this global spatiotemporal feature information. This hybrid architecture enables more effective integration and complementarity of spatiotemporal features. Through spatiotemporal modeling based on this hybrid architecture, this paper achieves a good balance between computational efficiency and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 Flowchart of an embodiment of the present invention.

[0069] Figure 2 This is a diagram of the overall network structure of an embodiment of the present invention.

[0070] Figure 3 This is a structural diagram of a spatial Mamba block according to an embodiment of the present invention.

[0071] Figure 4 This is a specific structural diagram of a token mixer in a spatial Mamba block according to an embodiment of the present invention. DETAILED DESCRIPTION

[0072] In order to make the purpose, technical solutions and advantages of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the embodiments described are part of the embodiments of the present invention, not all of the embodiments, and they should not be understood as limitations on the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In the description of the present invention, it should be understood that the terms used are only for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0073] The following combination Figure 1-Figure 4 The present invention describes a three-dimensional human posture estimation method based on hybrid architecture spatiotemporal modeling.

[0074] Example 1: Figure 1 As shown, this embodiment provides a 3D human posture estimation method based on hybrid architecture spatiotemporal modeling, comprising the following steps:

[0075] S1. Acquire historical human posture data and generate a 3D human posture estimation network training set, wherein the human posture data includes a 2D posture joint coordinate sequence and a corresponding 3D posture joint coordinate sequence true value;

[0076] The training set in this example uses the public human posture dataset Human3.6M dataset, which contains 3.6 million video frames and includes 15 daily activities performed by 11 subjects in 4 different perspectives, involving complex actions such as walking, sitting, making phone calls, and taking pictures. ,in is the 2D posture joint coordinate sequence, is the true value of the corresponding three-dimensional posture joint coordinate sequence, is the sequence length, is the number of joints.

[0077] S2. Construct a 3D human pose estimation network model, wherein the 3D human pose estimation network model includes a sequentially arranged spatiotemporal position embedding module, a plurality of alternately stacked spatiotemporal Mamba blocks, a spatiotemporal self-attention block including a Transformer spatiotemporal unit, a fully connected layer, and a regression layer;

[0078] like Figure 2 As shown in Figure 2, the 3D human pose estimation network is sequentially configured with a spatiotemporal position embedding module, two alternately stacked spatiotemporal Mamba blocks and spatiotemporal self-attention blocks, a fully connected layer, and a regression layer.

[0079] The spatiotemporal position embedding module is used to integrate prior knowledge of human skeleton structure and human kinematics. It is composed of a spatial learnable topological encoding layer (spatial position embedding module) and a temporal learnable topological encoding layer (temporal position embedding module) connected in series.

[0080] The spatially learnable topological encoding layer is used to integrate the prior knowledge of the human skeleton structure. The specific steps are: the input two-dimensional posture joint point sequence X is transformed into an output dimension of The linear layer maps to the high-dimensional feature space to obtain features , and then perform element-wise multiplication on The features of T time steps are respectively given learnable weights , and the spatial adjacency matrix Perform matrix multiplication and finally obtain the output features of the spatially learnable topological encoding layer , the specific calculation method is as follows:

[0081] First, express At t( ) time step features, and then calculate ,symbol Represents element-by-element multiplication, all T time steps Merge ; Spatial adjacency matrix Constructed by the following formula:

[0082]

[0083] Among them, the spatial topological structure matrix Represents the human skeleton model The connection relationship between the key joints, the matrix for The square matrix, the rows and columns correspond to the serial numbers of the corresponding joints. According to the human body structure prior, if the and There is an anatomical connection between the joints. ,otherwise ; Learnable matrix Also for The internal elements of the matrix are updated through network training to express the implicit connections between joints.

[0084] The temporal learnable topological encoding layer is used for human kinematics prior and is constructed by the following steps: First, the input features Exchange time and space dimensions, becomes ,make express The jth ) joint temporal features, and define the inter-frame similarity matrix of the j-th joint as:

[0085]

[0086] in for The transpose of ; define the time adjacency matrix of the j-th joint :

[0087]

[0088] in is the time topology matrix, the matrix The rows and columns correspond to the serial numbers of the corresponding time series; For each row, select the first two largest elements of the row and record the two elements in The position in the time topology matrix The elements at the corresponding positions are set to 1, and the rest of the elements are set to 0; the matrix can be learned The internal elements of are dynamically updated through training to represent the implicit connections between different joints in time sequence; then the following is calculated:

[0089]

[0090] in is the learnable weight, symbol Represents element-wise multiplication; all J joint dimensions After merging, the time and space dimensions are exchanged to obtain the output features of the time-learnable topological coding layer .

[0091] like Figure 2 As shown in Figure 1, the spatiotemporal Mamba block is used to preliminarily extract global spatiotemporal features. It consists of 10 Mamba spatiotemporal units connected in series, where each Mamba spatiotemporal unit is composed of a spatial Mamba block and a temporal Mamba block connected in series.

[0092] The spatial Mamba block in the nth Mamba spatiotemporal unit in the spatiotemporal Mamba block processes the spatial features of J joints in a single time frame according to the following process , where n=1,2,3....10: Figure 3 As shown, first, the input features Normalized to get , The result after processing by Mamba token mixer and Add together to get , After normalization, it is used as the input of the multi-layer perceptron MLP layer. The MLP contains two linear transformation layers, the activation function is GELU, and Dropout is added to prevent the network from overfitting. The output of the MLP layer is connected through the residual As the output of the final spatial Mamba block , the specific process can be expressed as:

[0093]

[0094]

[0095] in Representative layer normalization method;

[0096] The Mamba block uses the Mamba TokenMixer to mix features, such as Figure 4 As shown, it contains two branches;

[0097] Branch 1, Characteristics First, through the linear projection layer of branch one , the input features Projected to dimension The image is then processed through a one-dimensional convolution layer with a kernel size of 4 and an activation function SiLU, and finally the output is obtained through the selective scanning state space model layer SSML. , the specific process of branch one can be expressed as:

[0098] ;

[0099] For branch 2, First pass the linear projection layer , and then the output is obtained by passing through a one-dimensional convolution layer with a convolution kernel size of 4 and a SiLU activation function , the specific process of branch two can be expressed as:

[0100] ;

[0101] The outputs of branch one and branch two are then concatenated and passed through the final linear projection layer to obtain the output of the token mixer. , the specific process can be expressed as:

[0102] .

[0103] The temporal Mamba block in the nth Mamba spatiotemporal unit in the spatiotemporal Mamba block processes the temporal features of a single joint in T frames in the temporal dimension. First, the input features Exchange time and space dimensions, becomes , and then the same processing flow as the nth spatial Mamba block is performed, and finally the extracted time features are exchanged with the spatial and temporal dimensions to make them The dimensions are consistent, and the temporal Mamba block output features are obtained.

[0104] like Figure 2 As shown in Figure 2, the spatiotemporal self-attention block further supplements the global spatiotemporal feature information and is composed of 10 Transformer spatiotemporal units in series, where each Transformer spatiotemporal unit is composed of a spatial Transformer block and a temporal Transformer block in series;

[0105] The spatial Transformer block in the e-th Transformer spatiotemporal unit in the spatiotemporal self-attention block processes the spatial features of J joints in a single time frame in a process that is basically the same as the spatial Mamba block, that is, the input features are first normalized, then processed by the token mixer, and then added to the input features to obtain the intermediate result. The intermediate result is normalized and used as the input of the multi-layer perceptron MLP layer. The output of the MLP layer is added to the intermediate result through the residual connection as the output of the e-th spatial Transformer block , where e=1,2,3....10;

[0106] The spatial Transformer block uses a spatial multi-head self-attention module as a token mixer to calculate the global spatial correlation of key points in the same frame, that is, to linearly map its input features to a query vector , key vector , and the value vector , used to calculate 8 self-attention heads, the features after the self-attention heads are concatenated are transformed by the linear transformation matrix After projection, a Dropout layer is applied to prevent the network from overfitting, and the output of the token mixer is obtained:

[0107]

[0108] in, is the kth self-attention head, the linear transformation matrix It is implemented by a fully connected layer with both input and output feature dimensions of 128, which is used to fuse the features of multiple self-attention heads;

[0109] The temporal Transformer block in the e-th Transformer spatiotemporal unit in the spatiotemporal self-attention block processes the temporal features of a single joint in T frames in the temporal dimension. First, the input features Exchange time and space dimensions, becomes , and then the same processing flow as the spatial Transformer block is performed, and finally the extracted time features are exchanged with the spatial and temporal dimensions to make them The dimensions are consistent, and the temporal Transformer block output features are obtained.

[0110] The fully connected layer is used to expand and deepen the features by mapping the input features to a higher dimension to increase the nonlinear capacity of the model. The input feature dimension of the fully connected layer is , the output latent feature dimension is , the activation function uses Tanh.

[0111] The feature tensor processed by the fully connected layer is input into the regression layer to obtain the predicted value of the three-dimensional coordinate sequence.

[0112] In some embodiments, the number of spatiotemporal Mamba blocks and spatiotemporal self-attention blocks is specifically selected according to actual conditions; the number of Mamba spatiotemporal units and the number of Transformer spatiotemporal units can be specifically selected according to actual conditions.

[0113] S3. Train the network model based on the 3D human pose estimation network training set and loss function, learn the mapping relationship from the 2D pose joint point sequence to the 3D pose joint point sequence, and continuously adjust the network model parameters through the back-propagation algorithm to minimize the loss function;

[0114] The network model parameters are continuously adjusted through the back-propagation algorithm, including: back-propagation training using the gradient descent method, and the loss function constraining the difference between the predicted value and the true value of the 3D human pose coordinates, including the joint position loss. and motion smoothing loss , respectively defined as:

[0115]

[0116]

[0117] Among them, the parameters is the length of the sequence, is the number of joints, For the Frame, The predicted coordinate values ​​of the joints, For the Frame, The true value of the coordinates of the joints, , , and the final total loss function is:

[0118]

[0119] Among them, the hyperparameters Used to balance posture accuracy and motion smoothness.

[0120] This example sets the time series length , number of joints , the training batch size is 4, and the AdamW optimizer is used for 60 cycles with a weight decay coefficient of 0.01. The initial learning rate is , an exponential decay strategy is adopted with a decay factor of 0.99.

[0121] S4. Input the 2D posture joint point sequence data to be estimated into the trained network model, and output the 3D posture joint point sequence to complete the 3D human posture estimation task.

[0122] To verify the 3D human posture estimation effect of this embodiment, the following further illustrates the effect evaluation of the embodiment in conjunction with Table 1.

[0123] In this embodiment, the mean per joint position error P1 (MPJPE) and the Procrustes-MPJPE (P-MPJPE) are used as objective evaluation indicators, with units in millimeters. GT represents the use of real 2D joint point sequences during the training phase.

[0124] Table 1 Performance comparison table

[0125]

[0126] Table 1 compares the results of this embodiment with other advanced 3D human pose estimation methods. As can be seen, regardless of whether the predicted 2D joint point sequence or the real 2D joint point sequence is used during the training phase, the method of this embodiment achieves the lowest P1 error, and the P2 error is on par with the second-best method. Furthermore, this example simultaneously minimizes the number of model parameters and lowers the computational complexity compared to most other methods.

[0127] Compared with MotionAGFormer, one of the most advanced human pose estimation methods, the P1 error is reduced by 0.5 mm and the P2 error is reduced by 0.6 mm. At the same time, the number of parameters is reduced by 40.5% and the computational complexity is reduced by 34.6%.

[0128] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A 3D human pose estimation method based on hybrid architecture spatiotemporal modeling, characterized in that: The steps include: S1. Acquire historical human posture data and generate a 3D human posture estimation network training set, wherein the human posture data includes a 2D posture joint coordinate sequence and a corresponding 3D posture joint coordinate sequence true value; S2. Construct a 3D human pose estimation network model, wherein the 3D human pose estimation network model includes a sequentially arranged spatiotemporal position embedding module, a plurality of alternately stacked spatiotemporal Mamba blocks, a spatiotemporal self-attention block including a Transformer spatiotemporal unit, a fully connected layer, and a regression layer; S3. Train the network model based on the 3D human pose estimation network training set and loss function, learn the mapping relationship from the 2D pose joint point sequence to the 3D pose joint point sequence, and continuously adjust the network model parameters through the back-propagation algorithm to minimize the loss function; S4. Input the 2D posture joint point sequence data to be estimated into the trained network model, and output the 3D posture joint point sequence to complete the 3D human posture estimation task.

2. The method for 3D human pose estimation based on hybrid architecture spatiotemporal modeling according to claim 1, characterized in that: The 3D human pose estimation network training set is ,in is a two-dimensional posture joint coordinate sequence, is the true value of the corresponding three-dimensional posture joint coordinate sequence, is the sequence length, is the number of joints.

3. The method for 3D human pose estimation based on hybrid architecture spatiotemporal modeling according to claim 2, characterized in that: The spatiotemporal position embedding module is used to integrate the prior knowledge of human skeleton structure and human kinematics, and is composed of a spatial learnable topological coding layer and a temporal learnable topological coding layer in series; The spatiotemporal Mamba block is used to preliminarily extract global spatiotemporal features, and is composed of a number of Mamba spatiotemporal units connected in series, wherein each Mamba spatiotemporal unit is composed of a spatial Mamba block and a temporal Mamba block connected in series; The spatiotemporal self-attention block is used to further supplement the global spatiotemporal feature information and is composed of several Transformer spatiotemporal units in series, where each Transformer spatiotemporal unit is composed of a spatial Transformer block and a temporal Transformer block in series; The fully connected layer is used to expand and deepen the features by mapping the input features to a higher dimension to increase the nonlinear capacity of the model. The input feature dimension is , the output latent feature dimension is , the activation function adopts Tanh; The regression layer is used to convert the feature vector output by the fully connected layer into a three-dimensional coordinate sequence prediction value.

4. The method for 3D human pose estimation based on hybrid architecture spatiotemporal modeling according to claim 3, characterized in that: The spatially learnable topological encoding layer is used to integrate prior knowledge of the human skeleton structure, and the specific steps are as follows: Step 1: Based on the two-dimensional posture joint point sequence X, the output dimension is The linear layer maps it to a high-dimensional space and obtains features ; Step 2: Based on the above features , using the element-by-element multiplication method to obtain the spatial learnable topological coding layer output features , the specific calculation method is as follows: ； in, express The features at the tth time step, , represents element-wise multiplication, Represents the learnable weights, all T time steps Merge , Represents the spatial adjacency matrix, and its construction formula is as follows: ； Among them, the spatial topological structure matrix Represents the human skeleton model The connection relationship between the key joints, the matrix for The square matrix, the rows and columns correspond to the serial numbers of the corresponding joints. According to the human body structure prior, when the and When there is an anatomical connection between the joints, , when the and When there is no anatomical connection between the joints, ; Learnable matrix for The internal elements of the matrix are updated through network training to express the implicit connections between joints.

5. The method for 3D human pose estimation based on hybrid architecture spatiotemporal modeling according to claim 4, characterized in that: The temporal learnable topological encoding layer is used to integrate human kinematics priors and is constructed by the following steps: Step 1: Output features based on spatially learnable topological coding layer , and exchange its space-time dimension into , and then define the inter-frame similarity matrix of the j-th joint and the temporal adjacency matrix of the j-th joint : ； ； in, express The temporal features of the j-th joint in , for The transpose of is the time topology matrix, the matrix The rows and columns correspond to the serial numbers of the corresponding time series; select the matrix The first two largest elements of each row are recorded in The position in the time topology matrix The elements at the corresponding positions are set to 1, and the rest of the elements are set to 0; the matrix can be learned The internal elements of are dynamically updated through training to represent the implicit connections between different joints in the temporal sequence; Step 2: Calculate the output features of the temporal learnable topological coding layer based on the temporal adjacency matrix: ； in, are learnable weights, Represents element-wise multiplication; all J joint dimensions After merging, the time and space dimensions are exchanged to obtain the output features of the time-learnable topological coding layer .

6. The method for 3D human pose estimation based on hybrid architecture spatiotemporal modeling according to claim 5, characterized in that: The spatial Mamba block is used to process the spatial features of J joints in a single time frame. The specific operation steps are as follows: Step 1: Get the input features, normalize them and process them with Mamba TokenMixer to get : ； Where n is the sequence number of the Mamba space-time unit in the space-time Mamba block, n=1,2,3....N, N is the number of Mamba space-time units in the space-time Mamba block, is the input feature, is the input feature Normalized features; The Mamba token mixer is used to mix features and includes two branches; Branch 1: Features First, through the linear projection layer of branch one , the input features Projected to dimension The output is obtained by passing through the one-dimensional convolution layer and the activation function SiLU and finally through the selective scanning state space model layer SSML. , the specific process can be expressed as: ； Branch 2, First pass the linear projection layer , and then the output is obtained by the same one-dimensional convolution layer and SiLU activation function , the specific process can be expressed as: ； The outputs of branch one and branch two are then concatenated and passed through the final linear projection layer to obtain the output of the token mixer. , the specific formula is: ； Step 2: After normalization, the input is the multi-layer perceptron MLP layer, and the output result is added through the residual connection As the output of the final spatial Mamba block , the specific formula is: ； The MLP layer includes two linear transformation layers, the activation function is GELU, and Dropout is added to prevent network overfitting.

7. The method for 3D human pose estimation based on hybrid architecture spatiotemporal modeling according to claim 6, characterized in that: The temporal Mamba block is used to process the temporal features of a single joint in T frames in the temporal dimension. The specific steps are as follows: Step 1: Get input features , and exchange its space-time dimension, becomes ; Step 2: Input features First normalize, then pass through the token mixer, and then add it to the input feature to get the intermediate result. The intermediate result is normalized and used as the input of the multi-layer perceptron MLP layer. The output of the MLP layer is added to the intermediate result through the residual connection as the extracted time feature; Step 3: Exchange the extracted time features with the time and space dimensions. The dimensions are consistent, and the temporal Mamba block output features are obtained.

8. The method for 3D human pose estimation based on hybrid architecture spatiotemporal modeling according to claim 6, characterized in that: The spatial Transformer block is used to process the spatial features of J joints in a single time frame. The specific steps are as follows: the input features are first normalized, then processed by the token mixer, and then added to the input features to obtain an intermediate result. The intermediate result is normalized and used as the input of the multi-layer perceptron MLP layer. The output of the MLP layer is added to the intermediate result through a residual connection and used as the output of the spatial Transformer block in the e-th Transformer spatiotemporal unit. , where e=1,2,3....E, E is the number of Transformer spatiotemporal units in the spatiotemporal self-attention block; The spatial Transformer block uses a spatial multi-head self-attention module as a token mixer to calculate the global spatial correlation of key points in the same frame, that is, to linearly map its input features to a query vector , key vector , and the value vector , used to calculate K self-attention heads, the features after the self-attention heads are concatenated are transformed by the linear transformation matrix After projection, a Dropout layer is applied to prevent the network from overfitting, and the output of the token mixer is obtained: ； in, is the kth self-attention head, the linear transformation matrix It is implemented by a fully connected layer with both input and output feature dimensions of 128, which is used to fuse the features of multiple self-attention heads.

9. The method for 3D human pose estimation based on hybrid architecture spatiotemporal modeling according to claim 8, characterized in that: The temporal Transformer block is used to process the temporal features of a single joint within T frames in the temporal dimension. The specific steps are as follows: Step 1: Input features Exchange time and space dimensions, becomes ; Step 2: Input features First normalize, then pass through the token mixer, and then add it to the input feature to get the intermediate result. The intermediate result is normalized and used as the input of the multi-layer perceptron MLP layer. The output of the MLP layer is added to the intermediate result through the residual connection as the extracted time feature; Step 3: Exchange the extracted time features with the time and space dimensions. The dimensions are consistent, and the temporal Transformer block output features are obtained.

10. The method for 3D human pose estimation based on hybrid architecture spatiotemporal modeling according to claim 9, characterized in that: In step S3, the network model parameters are continuously adjusted by the back propagation algorithm, including: using the gradient descent method to perform back propagation training, and the loss function constrains the difference between the predicted value and the true value of the three-dimensional human posture coordinate; the loss function includes the joint position loss and motion smoothing loss : ； ； Among them, the parameters is the length of the sequence, is the number of joints, For the Frame, The predicted coordinate values ​​of the joints, For the Frame, The true value of the coordinates of the joints, , , and the final total loss function is: ； Among them, the hyperparameters Used to balance posture accuracy and motion smoothness.

Images