A three-dimensional human pose estimation method based on a hybrid architecture spatio-temporal modeling

Abstract

The application discloses a three-dimensional human body posture estimation method based on a mixed architecture space-time modeling, comprising the following steps: acquiring historical human body posture data, and generating a three-dimensional human body posture estimation network training set; constructing a three-dimensional human body posture estimation network model, wherein the network model comprises a space-time position embedding module, a space-time Mamba block, a space-time self-attention block, a full connection layer and a regression layer; training the network model based on the estimation network training set and a loss function, and learning a mapping relationship from a two-dimensional posture joint node sequence to a three-dimensional posture joint node sequence; inputting two-dimensional posture joint node sequence data to be estimated into the trained network model, and outputting a three-dimensional posture joint node sequence. The application uses Mamba to quickly extract global space-time features, and then uses a Transformer to further supplement global space-time feature information, so that the mixed architecture makes the space-time features more effectively fused and complementary.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and artificial intelligence, in particular to a human three-dimensional pose estimation method based on deep learning, and especially to a three-dimensional human pose estimation method based on a hybrid architecture spatio-temporal modeling. BACKGROUND

[0002] Human three-dimensional pose estimation is a core task in the field of computer vision and artificial intelligence, aiming to accurately recover the three-dimensional positions of human joints from images or videos, and has important applications in intelligent monitoring, human-computer interaction, motion analysis, and autonomous driving. Currently, deep learning has become the mainstream research method. Convolutional neural networks have improved the accuracy of two-dimensional pose estimation, but three-dimensional pose estimation needs to effectively utilize temporal information. Existing temporal modeling methods have limitations: recurrent neural networks and their variants have gradient vanishing and explosion problems, making it difficult to model long-range dependencies and limiting parallelism due to sequential computation, resulting in low efficiency; the receptive field of convolutional neural networks is limited, and multiple layers need to be stacked to capture long-range spatio-temporal relationships, resulting in complex models and large parameter quantities; although Transformer can efficiently model global dependencies, the computational complexity of the self-attention mechanism is proportional to the square of the sequence length, making it computationally intensive and memory-intensive when processing long videos, and the "fully connected" nature easily introduces redundant relationships, reducing the efficiency of long sequence processing.

[0003] In recent years, Mamba, as a structured state space sequence model, has great potential in long sequence modeling. It introduces selective state space and hardware-aware parallel scanning algorithms, with a linear increase in computational complexity with sequence length, making it computationally and memory-efficient when processing ultra-long sequences, and effectively capturing long-range dependencies. However, its ability to model global, fine-grained interaction relationships (such as accurate joint motion coordination in human pose) may not be as good as the self-attention mechanism of Transformer.

[0004] Fusing Transformer and Mamba for human three-dimensional pose estimation is still in the exploratory stage, and there is a lack of effective hybrid mechanisms designed for the spatio-temporal characteristics of human pose (such as hierarchical structure and joint motion constraints). It is still a challenge to achieve real-time or near-real-time three-dimensional pose estimation using the efficiency of Mamba while ensuring high accuracy.

[0005] Existing pure Transformer-based temporal modeling methods have high computational complexity, high resource consumption, and poor real-time performance when processing long video sequences. Although the emerging Mamba model is efficient, it lacks the ability to capture complex global spatial structures and fine joint coordination relationships in human pose. Therefore, there is an urgent need to innovate model architectures to combine the advantages of both and improve the accuracy and efficiency of three-dimensional pose estimation to better adapt to real-world application scenarios. The present application aims to solve these problems. SUMMARY

[0006] The present application aims to at least partially solve the technical problems existing in the related art.

[0007] The present application aims to at least partially solve the technical problems existing in the related art.

[0008] In order to achieve the above-mentioned purpose, the present application provides a three-dimensional human pose estimation method based on a hybrid architecture spatio-temporal modeling, comprising the following steps:

[0009] S1, obtaining historical human pose data to generate a three-dimensional human pose estimation network training set, wherein the human pose data includes a two-dimensional pose joint coordinate sequence and a corresponding three-dimensional pose joint coordinate sequence true value;

[0010] S2, constructing a three-dimensional human pose estimation network model, wherein the three-dimensional human pose estimation network model comprises a spatio-temporal position embedding module, a plurality of alternately stacked spatio-temporal Mamba blocks, a spatio-temporal self-attention block comprising a Transformer spatio-temporal unit, a full connection layer and a regression layer arranged in sequence;

[0011] S3, training the network model based on the three-dimensional human pose estimation network training set and the loss function, learning the mapping relationship from the two-dimensional pose joint sequence to the three-dimensional pose joint sequence, and continuously adjusting the network model parameters through the back propagation algorithm to minimize the loss function;

[0012] S4, inputting the two-dimensional pose joint sequence data to be estimated into the trained network model, and outputting the three-dimensional pose joint sequence to complete the three-dimensional human pose estimation task.

[0013] Preferably, the three-dimensional human pose estimation network training set is , wherein is a two-dimensional pose joint coordinate sequence, is a corresponding three-dimensional pose joint coordinate sequence true value, is a sequence length, is the number of joints.

[0014] Preferably, the spatio-temporal position embedding module is used to fuse the human skeleton structure priori knowledge and the human kinematics priori, and is composed of a spatial learnable topological coding layer and a time learnable topological coding layer in series.

[0015] The spatio-temporal Mamba block is used for preliminary extraction of global spatio-temporal features and is composed of a plurality of Mamba spatio-temporal units connected in series, wherein each Mamba spatio-temporal unit is composed of a spatial Mamba block and a temporal Mamba block connected in series;

[0016] The spatio-temporal self-attention block is used for further supplementing global spatio-temporal feature information and is composed of a plurality of Transformer spatio-temporal units connected in series, wherein each Transformer spatio-temporal unit is composed of a spatial Transformer block and a temporal Transformer block connected in series;

[0017] The fully connected layer is used for expanding and deepening features, and by mapping the input features to a higher dimension, the non-linear capacity of the model is increased, wherein the input feature dimension is , the output hidden feature dimension is , and the activation function adopts Tanh;

[0018] The regression layer is used for converting the feature vector output by the fully connected layer to obtain a three-dimensional coordinate sequence prediction value.

[0019] As a preferred, the spatial learnable topology encoding layer is used for fusing human skeleton structure prior knowledge, and the specific steps are as follows:

[0020] Step one, based on the two-dimensional pose joint sequence X, through a linear layer with an output dimension of , it is mapped to a high-dimensional space and a feature is obtained;

[0021] Step two, based on the above feature , the element-wise point multiplication method is used to obtain the output feature of the spatial learnable topology encoding layer, and the specific calculation method is as follows:

[0022]

[0023] , wherein represents the feature at the tth time step, , , represents element-wise multiplication, represents a learnable weight, and of all T time steps are combined to obtain , represents a spatial adjacency matrix, and the construction formula is as follows:

[0024]

[0025] , wherein the spatial topology structure matrix represents the human skeleton model The connection relationship between the key nodes, the matrix is a square matrix, and the rows and columns correspond to the serial numbers of the corresponding joints in turn. According to the human body structure priori, when the first and the first joints are anatomically connected, when the first and the first joints are not anatomically connected, the learnable matrix is a square matrix, and the internal elements are updated through network training, used to express the implicit connection between the joints.

[0026] As preferred, the time learnable topological coding layer is used to fuse the human body kinematics priori, and is constructed by the following steps:

[0027] Step one, based on the spatial learnable topological coding layer output feature , its space-time dimension is exchanged to , and then the inter-frame similarity matrix of the jth joint and the time adjacency matrix of the jth joint are defined:

[0028]

[0029]

[0030] wherein, represents the time sequence feature of the jth joint in , , is the transpose of ; is the time topology matrix, and the rows and columns of the matrix correspond to the serial numbers of the corresponding time sequences in turn; the matrix selects the first two largest elements of each row, records the positions of the two elements in , and sets the elements of the corresponding positions of the time topology matrix to 1 and the remaining elements to 0; the internal elements of the learnable matrix are dynamically updated through training, used to represent the implicit connection between different joints in time sequence;

[0031] Step two, based on the time adjacency matrix, the time learnable topological coding layer output feature is calculated:

[0032]

[0033] wherein, is a learnable weight, denotes element-wise multiplication; all J joint dimensions After merging, the space-time dimension exchange is performed to obtain the time-learnable topological encoding layer output feature .

[0034] As a preferred, the spatial Mamba block is used to process the spatial features of J joints within a single time frame, and the specific operation steps are as follows:

[0035] Step one, obtain the input feature, normalize it and process it through the Mamba token mixer to obtain :

[0036]

[0037] wherein n is the serial number of the Mamba space-time unit in the space-time Mamba block, n = 1, 2, 3,..., N, N is the number of Mamba space-time units in the space-time Mamba block, is the input feature, is the input feature the normalized feature;

[0038] The Mamba token mixer is used to mix the features, including two branches.

[0039] Branch one: feature First, the input feature is projected to a space with a dimension of by the linear projection layer of branch one, then it is processed by the one-dimensional convolution layer and the SiLU activation function, and finally the output is obtained by the selective scanning state space model layer SSML, and the specific process can be represented as:

[0040] ;

[0041] Branch two, first through the linear projection layer , then through the one-dimensional convolution layer and the SiLU activation function to obtain the output , and the specific process can be represented as:

[0042] ;

[0043] Then, the outputs of branch one and branch two are spliced and processed by the final linear projection layer to obtain the output of the token mixer , and the specific formula is:

[0044] ;

[0045] Step two, the​ The normalized input is input into a multi-layer perception (MLP) layer, and the output is added to the input through a residual connection as the output of the final spatial Mamba block The specific formula is:

[0046] ;

[0047] The MLP layer includes two linear transformation layers, and the activation function is GELU, and Dropout is added to prevent network overfitting.

[0048] As a preferred embodiment, the temporal Mamba block is used to process the temporal features of a single joint in T frames in the time dimension, and the specific steps are as follows:

[0049] Step 1: Obtain the input features and perform space-time dimension exchange, i.e. to ;

[0050] Step 2: Normalize the input features , then pass them through the token mixer, and then add them to the input features to obtain an intermediate result. The normalized intermediate result is input into a multi-layer perception (MLP) layer, and the output of the MLP layer is added to the intermediate result through a residual connection, as the extracted temporal features.

[0051] Step 3: Perform space-time dimension exchange on the extracted temporal features to make them consistent with dimension, and obtain the output features of the temporal Mamba block.

[0052] As a preferred embodiment, the spatial Transformer block is used to process the spatial features of J joints in a single time frame, and the specific steps are as follows: normalize the input features, then pass them through the token mixer, and then add them to the input features to obtain an intermediate result. The normalized intermediate result is input into a multi-layer perception (MLP) layer, and the output of the MLP layer is added to the intermediate result through a residual connection, as the output of the spatial Transformer block in the e-th Transformer space-time unit , where e = 1, 2, 3,..., E, and E is the number of Transformer space-time units in the space-time self-attention block.

[0053] The spatial Transformer block uses a spatial multi-head self-attention module as a token mixer to calculate the global spatial correlation of key points in the same frame, i.e. linearly maps its input features to query vectors , key vectors , and value vectors , used to calculate K self-attention heads, the features after the self-attention heads are concatenated are transformed by the linear transformation matrix After projection, a Dropout layer is applied to prevent the network from overfitting, and the output of the token mixer is obtained:

[0054]

[0055] in, is the kth self-attention head, the linear transformation matrix It is implemented by a fully connected layer with both input and output feature dimensions of 128, which is used to fuse the features of multiple self-attention heads.

[0056] Preferably, the temporal Transformer block is used to process the temporal features of a single joint in T frames in the temporal dimension, and the specific steps are as follows:

[0057] Step 1: Input features Exchange time and space dimensions, becomes ;

[0058] Step 2: Input features First normalize, then pass through the token mixer, and then add it to the input feature to get the intermediate result. The intermediate result is normalized and used as the input of the multi-layer perceptron MLP layer. The output of the MLP layer is added to the intermediate result through the residual connection as the extracted time feature;

[0059] Step 3: Exchange the extracted time features with the time and space dimensions. The dimensions are consistent, and the temporal Transformer block output features are obtained.

[0060] As a preference, in step S3, the network model parameters are continuously adjusted by the back propagation algorithm, including: using the gradient descent method to perform back propagation training, and the loss function constrains the difference between the predicted value and the true value of the three-dimensional human posture coordinate; the loss function includes the joint position loss and motion smoothing loss :

[0061]

[0062]

[0063] Among them, the parameters is the length of the sequence, is the number of joints, For the Frame, The predicted coordinate values ​​of the joints, For the frame, the first coordinate true value of the joint, , The final total loss function is:

[0064]

[0065] The hyperparameter is used to balance the pose accuracy and the smoothness of motion.

[0066] Beneficial effects: 1. The application supplements the human skeleton structure priori and human kinematics priori through the space-time position embedding module, and assists the network in efficiently and high-quality modeling human three-dimensional pose features.

[0067] 2. The application makes full use of the advantages of the two models, first extracts global space-time features quickly using Mamba, and then uses Transformer to further supplement global space-time feature information, the hybrid architecture makes the space-time features more effectively fused and complementary. Through the space-time modeling based on the hybrid architecture, the application achieves a good balance between computational efficiency and accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0068] Figure 1 is the flow chart of the embodiment of the application.

[0069] Figure 2 is the network overall structure diagram of the embodiment of the application.

[0070] Figure 3 is the structure diagram of a space Mamba block of the embodiment of the application.

[0071] Figure 4 is the specific structure diagram of the token mixer in a space Mamba block of the embodiment of the application. DETAILED DESCRIPTION

[0072] To make the purpose, technical scheme and advantages of the application clearer, the technical scheme in the application will be described clearly and completely below in combination with the drawings in the application. Obviously, the described embodiments are part of the embodiments of the application, not all the embodiments, and they should not be understood as limiting the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application. In the description of the application, it should be understood that the terms used are only for the purpose of description, and should not be understood as indicating or implying relative importance.

[0073] The application provides a three-dimensional human pose estimation method based on hybrid architecture space-time modeling. Figures 1-4 The application provides a three-dimensional human pose estimation method based on hybrid architecture space-time modeling.

[0074] Example 1: Figure 1 As shown, this embodiment provides a 3D human posture estimation method based on hybrid architecture spatiotemporal modeling, comprising the following steps:

[0075] S1. Acquire historical human posture data and generate a 3D human posture estimation network training set, wherein the human posture data includes a 2D posture joint coordinate sequence and a corresponding 3D posture joint coordinate sequence true value;

[0076] The training set in this example uses the public human posture dataset Human3.6M dataset, which contains 3.6 million video frames and includes 15 daily activities performed by 11 subjects in 4 different perspectives, involving complex actions such as walking, sitting, making phone calls, and taking pictures. ,in is a two-dimensional posture joint coordinate sequence, is the true value of the corresponding three-dimensional posture joint coordinate sequence, is the sequence length, is the number of joints.

[0077] S2. Construct a 3D human pose estimation network model, wherein the 3D human pose estimation network model includes a sequentially arranged spatiotemporal position embedding module, a plurality of alternately stacked spatiotemporal Mamba blocks, a spatiotemporal self-attention block including a Transformer spatiotemporal unit, a fully connected layer, and a regression layer;

[0078] like Figure 2 As shown in Figure 2, the 3D human pose estimation network is sequentially configured with a spatiotemporal position embedding module, two alternately stacked spatiotemporal Mamba blocks and spatiotemporal self-attention blocks, a fully connected layer, and a regression layer.

[0079] The spatiotemporal position embedding module is used to integrate prior knowledge of human skeleton structure and human kinematics. It is composed of a spatial learnable topological encoding layer (spatial position embedding module) and a temporal learnable topological encoding layer (temporal position embedding module) connected in series.

[0080] The spatially learnable topological encoding layer is used to integrate the prior knowledge of the human skeleton structure. The specific steps are: the input two-dimensional posture joint point sequence X is transformed into an output dimension of The linear layer maps to the high-dimensional feature space to obtain features , and then perform element-wise multiplication on The features of T time steps are given learnable weights , and the spatial adjacency matrix Perform matrix multiplication and finally obtain the output features of the spatially learnable topological encoding layer , the specific calculation method is as follows:

[0081] First, express At t ( ) time step features, and then calculate ,symbol Represents element-by-element multiplication, all T time steps Merge ; Spatial adjacency matrix Constructed by the following formula:

[0082]

[0083] Among them, the spatial topological structure matrix Represents the human skeleton model The connection relationship between the key joints, the matrix for The square matrix, the rows and columns correspond to the serial numbers of the corresponding joints. According to the human body structure prior, if the and There is an anatomical connection between the joints. ,otherwise ; Learnable matrix Also for The internal elements of the matrix are updated through network training to express the implicit connections between joints.

[0084] The temporal learnable topological encoding layer is used for human kinematics prior and is constructed by the following steps: First, the input features Exchange time and space dimensions, becomes ,make express The jth ) joint temporal features, and define the inter-frame similarity matrix of the j-th joint as:

[0085]

[0086] in for The transpose of ; define the time adjacency matrix of the j-th joint :

[0087]

[0088] in is the time topology matrix, the matrix The rows and columns correspond to the serial numbers of the corresponding time series; For each row, select the first two largest elements of the row and record the two elements in The position in the time topology matrix The element in the corresponding position is set to 1, and the remaining elements are set to 0; the learnable matrix The internal elements are dynamically updated through training, used to represent the implicit connection between different joints in time sequence; then calculate:

[0089]

[0090] Wherein is a learnable weight, and the symbol represents element-wise multiplication; the of all J joint dimensions is combined and then subjected to space-time dimension exchange to obtain the time learnable topological coding layer output feature .

[0091] As shown in Figure 2 , the space-time Mamba block is used for preliminary extraction of global space-time features, and is composed of 10 Mamba space-time units in series, wherein each Mamba space-time unit is composed of a space Mamba block and a time Mamba block in series;

[0092] The space Mamba block in the n th Mamba space-time unit in the space-time Mamba block processes the spatial features of J joints in a single time frame according to the following flow , wherein n=1,2,3....10: as shown in Figure 3 , first normalize the input feature to obtain , The result after processing by the Mamba token mixer is added to to obtain , After normalization, it is used as the input of the multi-layer perception (MLP) layer, the MLP includes two linear transformation layers, the activation function is GELU, and Dropout is added to prevent network overfitting, the output of the MLP layer is added to as the output of the final space Mamba block , and the specific process can be represented as:

[0093]

[0094]

[0095] Wherein represents the layer normalization method;

[0096] The Mamba token mixer (TokenMixer) is used in the Mamba block to mix the features, as shown in Figure 4 , which includes two branches;

[0097] Branch one, feature First, the linear projection layer of branch one projects the input feature into a space with dimension , then a one-dimensional convolution layer with a kernel size of 4 and an activation function SiLU, and finally the output is obtained by the selective scanning state space model layer SSML.

[0098] ;

[0099] For branch two, first, the linear projection layer , then a one-dimensional convolution layer with a kernel size of 4 and a SiLU activation function to obtain the output , and the specific process of branch two can be represented as:

[0100] ;

[0101] Then the outputs of branch one and branch two are spliced and passed through the final linear projection layer to obtain the output of the token mixer , and the specific process can be represented as:

[0102] .

[0103] The time Mamba block in the nth Mamba space-time unit in the space-time Mamba block processes the time features of a single joint within T frames in the time dimension. First, the input feature is exchanged in space-time dimensions, i.e. is changed to , then the same processing procedure as the nth space Mamba block is performed, and finally the extracted time features are exchanged in space-time dimensions to make them consistent with dimensions, obtaining the output features of the time Mamba block.

[0104] As shown in Figure 2 , the space-time self-attention block further supplements global space-time feature information and is composed of 10 Transformer space-time units, each of which is composed of a space Transformer block and a time Transformer block.

[0105] The space Transformer block in the e-th Transformer space unit in the space-time self-attention block processes the flow of the space features of the J joints in a single time frame, which is basically the same as the space Mamba block, that is, the input features are first normalized, then processed through a token mixer, and then added to the input features to obtain an intermediate result, the intermediate result is normalized and used as the input of a multi-layer perception (MLP) layer, the output of the MLP layer is added to the intermediate result through a residual connection, and the output of the e-th space Transformer block is obtained , wherein e = 1, 2, 3,..., 10

[0106] The space Transformer block uses a space multi-head self-attention module as a token mixer to calculate the global space correlation of the key points in the same frame, that is, the input features thereof are linearly mapped into query vectors , key vectors , and value vectors , which are used to calculate 8 self-attention heads, and the features after splicing of the self-attention heads are projected through a linear transformation matrix , and then a Dropout layer is applied to prevent network overfitting, that is, the output of the token mixer is obtained

[0107]

[0108] , wherein is the k-th self-attention head, and the linear transformation matrix is realized by a fully connected layer with an input feature dimension and an output feature dimension of 128, which is used to fuse the features of multiple self-attention heads

[0109] The time Transformer block in the e-th Transformer space unit in the space-time self-attention block processes the time features of a single joint in T frames in the time dimension, first performs space-time dimension exchange on the input features , that is, changes to , and then performs the same processing flow as the space Transformer block, and finally performs space-time dimension exchange on the extracted time features to make them consistent with the dimension, to obtain the output features of the time Transformer block.

[0110] The fully connected layer is used to expand and deepen the features, and the input feature dimension of the fully connected layer is , the hidden feature dimension of the output is , and the activation function adopts Tanh.

[0111] ​The feature tensor processed by the fully connected layer is input into the regression layer to obtain the predicted value of the three-dimensional coordinate sequence.

[0112] In some embodiments, the number of spatiotemporal Mamba blocks and spatiotemporal self-attention blocks is specifically selected according to actual conditions; the number of Mamba spatiotemporal units and the number of Transformer spatiotemporal units can be specifically selected according to actual conditions.

[0113] S3. Train the network model based on the 3D human pose estimation network training set and loss function, learn the mapping relationship from the 2D pose joint point sequence to the 3D pose joint point sequence, and continuously adjust the network model parameters through the back-propagation algorithm to minimize the loss function;

[0114] The network model parameters are continuously adjusted through the back-propagation algorithm, including: back-propagation training using the gradient descent method, and the loss function constraining the difference between the predicted value and the true value of the 3D human pose coordinates, including the joint position loss. and motion smoothing loss , respectively defined as:

[0115]

[0116]

[0117] Among them, the parameters is the length of the sequence, is the number of joints, For the Frame, The predicted coordinate values ​​of the joints, For the Frame, The true value of the coordinates of the joints, , , and the final total loss function is:

[0118]

[0119] Among them, the hyperparameters Used to balance posture accuracy and motion smoothness.

[0120] This example sets the time series length , number of joints , the training batch size is 4, and the AdamW optimizer is used for 60 cycles with a weight decay coefficient of 0.01. The initial learning rate is , an exponential decay strategy is adopted with a decay factor of 0.99.

[0121] S4, inputting the two-dimensional pose joint sequence data to be estimated into the trained network model, outputting a three-dimensional pose joint sequence to complete the three-dimensional human pose estimation task.

[0122] To verify the three-dimensional human pose estimation effect of the embodiment, the following further describes the effect evaluation of the embodiment in combination with Table 1.

[0123] In the embodiment, the mean joint position error P1 (MPJPE) and the aligned mean joint position error P2 (Procrustes-MPJPE, P-MPJPE) are used as objective evaluation indexes, and the unit is millimeter. GT represents that the real two-dimensional joint sequence is used in the training stage.

[0124] Table 1 performance comparison table

[0125]

[0126] Table 1 lists the comparison results of the embodiment and other advanced three-dimensional human pose estimation methods. It can be seen from the table that the P1 error of the method of the embodiment is the lowest, and the P2 error is flat with the second place, whether the predicted two-dimensional joint sequence or the real two-dimensional joint sequence is used in the training stage. On this basis, the embodiment also ensures the lowest model parameter amount and lower operation complexity than most methods.

[0127] Compared with one of the most advanced human pose estimation methods, MotionAGFormer, the P1 error is reduced by 0.5 millimeter, the P2 error is reduced by 0.6 millimeter, the parameter amount is reduced by 40.5%, and the operation complexity is reduced by 34.6%.

[0128] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A 3D human pose estimation method based on hybrid architecture spatiotemporal modeling, characterized in that: The steps include: S1. Acquire historical human posture data and generate a 3D human posture estimation network training set, wherein the human posture data includes a 2D posture joint coordinate sequence and a corresponding 3D posture joint coordinate sequence true value; S2. Construct a 3D human pose estimation network model, wherein the 3D human pose estimation network model includes a sequentially arranged spatiotemporal position embedding module, a plurality of alternately stacked spatiotemporal Mamba blocks, a spatiotemporal self-attention block including a Transformer spatiotemporal unit, a fully connected layer, and a regression layer; The spatiotemporal position embedding module is used to integrate the prior knowledge of human skeleton structure and human kinematics, and is composed of a spatial learnable topological coding layer and a temporal learnable topological coding layer in series; The spatiotemporal Mamba block is used to preliminarily extract global spatiotemporal features, and is composed of a number of Mamba spatiotemporal units connected in series, wherein each Mamba spatiotemporal unit is composed of a spatial Mamba block and a temporal Mamba block connected in series; The spatiotemporal self-attention block is used to further supplement the global spatiotemporal feature information and is composed of several Transformer spatiotemporal units in series, where each Transformer spatiotemporal unit is composed of a spatial Transformer block and a temporal Transformer block in series; The fully connected layer is used to expand and deepen the features by mapping the input features to a higher dimension to increase the nonlinear capacity of the model. The input feature dimension is , the output latent feature dimension is , the activation function adopts Tanh; The regression layer is used to convert the feature vector output by the fully connected layer into a three-dimensional coordinate sequence prediction value; The spatially learnable topological encoding layer is used to integrate prior knowledge of the human skeleton structure, and the specific steps are as follows: Step 1: Based on the two-dimensional posture joint point sequence X, the output dimension is The linear layer maps it to a high-dimensional space and obtains features ; Step 2: Based on the above features , using the element-by-element multiplication method to obtain the spatial learnable topological coding layer output features , the specific calculation method is as follows: ； in, express The features at the tth time step, , represents element-wise multiplication, Represents the learnable weights, all T time steps Merge , Represents the spatial adjacency matrix, and its construction formula is as follows: ； Among them, the spatial topological structure matrix Represents the human skeleton model The connection relationship between the key joints, the matrix for The square matrix, the rows and columns correspond to the serial numbers of the corresponding joints. According to the human body structure prior, when the and When there is an anatomical connection between the joints, , when the and When there is no anatomical connection between the joints, ; Learnable matrix for The internal elements of the matrix are updated through network training to express the implicit connection between joints; S3. Train the network model based on the 3D human pose estimation network training set and loss function, learn the mapping relationship from the 2D pose joint point sequence to the 3D pose joint point sequence, and continuously adjust the network model parameters through the back-propagation algorithm to minimize the loss function; S4. Input the 2D posture joint point sequence data to be estimated into the trained network model, and output the 3D posture joint point sequence to complete the 3D human posture estimation task.

2. The method for 3D human pose estimation based on hybrid architecture spatiotemporal modeling according to claim 1, characterized in that: The 3D human pose estimation network training set is ,in is a two-dimensional posture joint coordinate sequence, is the true value of the corresponding three-dimensional posture joint coordinate sequence, is the sequence length, is the number of joints.

3. The method for 3D human pose estimation based on hybrid architecture spatiotemporal modeling according to claim 2, characterized in that: The temporal learnable topological encoding layer is used to integrate human kinematics priors and is constructed by the following steps: Step 1: Output features based on spatially learnable topological coding layer , and exchange its space-time dimension into , and then define the inter-frame similarity matrix of the j-th joint and the temporal adjacency matrix of the j-th joint : ； ； in, express The temporal features of the j-th joint in , for The transpose of is the time topology matrix, the matrix The rows and columns correspond to the serial numbers of the corresponding time series; select the matrix The first two largest elements of each row are recorded in The position in the time topology matrix The elements at the corresponding positions are set to 1, and the rest of the elements are set to 0; the matrix can be learned The internal elements of are dynamically updated through training to represent the implicit connections between different joints in the temporal sequence; Step 2: Calculate the output features of the temporal learnable topological coding layer based on the temporal adjacency matrix: ； in, are learnable weights, Represents element-wise multiplication; all J joint dimensions After merging, the time and space dimensions are exchanged to obtain the output features of the time-learnable topological coding layer .

4. The method for 3D human pose estimation based on hybrid architecture spatiotemporal modeling according to claim 3, characterized in that: The spatial Mamba block is used to process the spatial features of J joints in a single time frame. The specific operation steps are as follows: Step 1: Get the input features, normalize them and process them with Mamba TokenMixer to get : ； Where n is the sequence number of the Mamba space-time unit in the space-time Mamba block, n=1,2,3....N, N is the number of Mamba space-time units in the space-time Mamba block, is the input feature, is the input feature Normalized features; The Mamba token mixer is used to mix features and includes two branches; Branch 1: Features First, through the linear projection layer of branch one , the input features Projected to dimension The output is obtained by passing through the selective scanning state space model layer SSML. , the specific process can be expressed as: ； Branch 2, First pass the linear projection layer , and then the output is obtained by the same one-dimensional convolution layer and SiLU activation function , the specific process can be expressed as: ； The outputs of branch one and branch two are then concatenated and passed through the final linear projection layer to obtain the output of the token mixer. , the specific formula is: ； Step 2: After normalization, the input is the multi-layer perceptron MLP layer, and the output result is added through the residual connection As the output of the final spatial Mamba block , the specific formula is: ； The MLP layer includes two linear transformation layers, the activation function is GELU, and Dropout is added to prevent network overfitting.

5. The method for 3D human pose estimation based on hybrid architecture spatiotemporal modeling according to claim 4, characterized in that: The temporal Mamba block is used to process the temporal features of a single joint in T frames in the temporal dimension. The specific steps are as follows: Step 1: Get input features , and exchange its space-time dimension, becomes ; Step 2: Input features First normalize, then pass through the token mixer, and then add it to the input feature to get the intermediate result. The intermediate result is normalized and used as the input of the multi-layer perceptron MLP layer. The output of the MLP layer is added to the intermediate result through the residual connection as the extracted time feature; Step 3: Exchange the extracted time features with the time and space dimensions. The dimensions are consistent, and the temporal Mamba block output features are obtained.

6. The method for 3D human pose estimation based on hybrid architecture spatiotemporal modeling according to claim 5, characterized in that: The spatial Transformer block is used to process the spatial features of J joints in a single time frame. The specific steps are as follows: the input features are first normalized, then processed by the token mixer, and then added to the input features to obtain an intermediate result. The intermediate result is normalized and used as the input of the multi-layer perceptron MLP layer. The output of the MLP layer is added to the intermediate result through a residual connection and used as the output of the spatial Transformer block in the e-th Transformer spatiotemporal unit. , where e=1,2,3....E, E is the number of Transformer spatiotemporal units in the spatiotemporal self-attention block; The spatial Transformer block uses a spatial multi-head self-attention module as a token mixer to calculate the global spatial correlation of key points in the same frame, that is, to linearly map its input features into a query vector , key vector , and the value vector , used to calculate K self-attention heads, the features after the self-attention heads are concatenated are transformed by the linear transformation matrix After projection, a Dropout layer is applied to prevent the network from overfitting, and the output of the token mixer is obtained: ； in, is the kth self-attention head, the linear transformation matrix It is implemented by a fully connected layer with both input and output feature dimensions of 128, which is used to fuse the features of multiple self-attention heads.

7. The method for 3D human pose estimation based on hybrid architecture spatiotemporal modeling according to claim 6, characterized in that: The temporal Transformer block is used to process the temporal features of a single joint within T frames in the temporal dimension. The specific steps are as follows: Step 1: Input features Exchange time and space dimensions, becomes ; Step 2: Input features First normalize, then pass through the token mixer, and then add it to the input feature to get the intermediate result. The intermediate result is normalized and used as the input of the multi-layer perceptron MLP layer. The output of the MLP layer is added to the intermediate result through the residual connection as the extracted time feature; Step 3: Exchange the extracted time features with the time and space dimensions. The dimensions are consistent, and the temporal Transformer block output features are obtained.

8. The method for 3D human pose estimation based on hybrid architecture spatiotemporal modeling according to claim 7, characterized in that: In step S3, the network model parameters are continuously adjusted by the back propagation algorithm, including: using the gradient descent method to perform back propagation training, and the loss function constrains the difference between the predicted value and the true value of the three-dimensional human posture coordinate; the loss function includes the joint position loss and motion smoothing loss : ； ； Among them, the parameters is the length of the sequence, is the number of joints, For the Frame, The predicted coordinate values ​​of the joints, For the Frame, The true value of the coordinates of the joints, , , and the final total loss function is: ； Among them, the hyperparameters Used to balance posture accuracy and motion smoothness.

Images