Three-dimensional human pose estimation method based on multi-scale spatio-temporal encoder network
Patent Information
- Application Number
- CN202410722522.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-05
- Publication Date
- 2026-09-22
- Estimated Expiration
- 2044-06-05
AI Technical Summary
Seq2frame旨在从长序列数据中估计中心帧的关节点信息，此方法虽然降低了时间序列数据的处理复杂度，但会丢失序列中的时间信息；Seq2seq旨在从f个序列帧中预测每个对应帧的姿态，但此方法在将完整的长视频帧分割成多个单帧进行单独处理时，会出现首尾两帧缺乏时间相关性的问题
[0052]本发明基于多尺度时空编码器网络的三维人体姿态估计方法，在使用U-net网络架构提取多层次关节点语义信息的前提下，减少待预测序列的长度变化对整体关节点预测精度的影响，对视频帧中的二维人体图像进行关节点提取，并将二维关节点序列提升至三维关节点序列。
Smart Images
Figure CN118674778B_ABST
Abstract
Description
[0001] This invention belongs to the field of computer vision and artificial intelligence technology, and relates to a three-dimensional human pose estimation method based on a multi-scale spatiotemporal encoder network. Background Technology
[0002] 3D human pose estimation (3D HPE) is a crucial research area in computer vision, aiming to reconstruct the positions of human keypoints in 3D space. Its applications are widespread, including film and animation, intelligent security, medical assistance, and motion analysis. 3D human pose estimation methods are mainly divided into two categories: end-to-end direct estimation and two-stage pose "lifting" methods. The former directly infers the 3D pose from the input image or video; the latter first converts the human image sequence into a 2D keypoint sequence and then predicts the 3D keypoints. Although the accuracy and generalization of 2D human pose estimation models have been significantly improved, inferring 3D human pose from 2D video images still suffers from problems such as self-occlusion and depth blur.
[0003] While U-Net has achieved good performance and application results in deep learning, its convolutional neural network-based approach often suffers from poor generalization ability and poor spatiotemporal correlation modeling when processing long sequence data. Furthermore, the selection of frame sequences along the time axis is a hot research topic in the field of 3D human pose estimation. Seq2frame aims to estimate the keypoint information of the center frame from long sequence data; while this method reduces the processing complexity of time series data, it loses temporal information within the sequence. Seq2seq aims to predict the pose of each corresponding frame from f sequence frames, but this method suffers from a lack of temporal correlation between the first and last frames when dividing a complete long video frame into multiple single frames for individual processing. Summary of the Invention
[0004] The purpose of this invention is to provide a three-dimensional human pose estimation method based on a multi-scale spatiotemporal encoder network, which can extract joint points from two-dimensional human images in video frames.
[0005] The technical solution adopted in this invention is a three-dimensional human pose estimation method based on a multi-scale spatiotemporal encoder network, which is implemented according to the following steps:
[0006] Step 1: Capture two-dimensional human joints in video frames using a two-dimensional pose detector, preprocess the joints, and feed them into the pose estimation pipeline.
[0007] Step 2: Use the sequence filling module to learn the spatiotemporal correlation of the joints in the original sequence and the filled sequence from Step 1;
[0008] Step 3: Use a multi-scale module to enhance the correlation between different scales of joint features, local features and global features based on the topology of the human skeleton;
[0009] Step 4: Use a linear transformation layer to project the high-dimensional output of Step 3 into three-dimensional coordinates;
[0010] Step 5: Construct the loss function and use MPJPE to train the model end-to-end.
[0011] The invention is further characterized by:
[0012] Step 1 is implemented in the following steps:
[0013] Step 1.1: Input the video to be processed frame by frame into the two-dimensional pose detector to obtain the two-dimensional human pose sequence corresponding to the video frame;
[0014] Step 1.2: Stretch the joint coordinate information in the two-dimensional human pose sequence into a one-dimensional tensor vector to obtain the preprocessed joint information.
[0015] Step 2 is implemented in the following steps:
[0016] Step 2.1: Construct a sequence filling module to process the two-dimensional pose sequence after step 1.2. Perform sequence padding; the padded sequence is: Where F is the original sequence length, N is the padding sequence length, J is the number of key points, and the channel size is 2;
[0017] The spatial position of each joint is embedded by linear projection to preserve the spatial position information of the joint in each frame:
[0018]
[0019] In the formula, and represents the linear projection matrix and the spatial position embedding matrix, respectively, and x represents a key point;
[0020] Step 2.2: Construct a spatial encoder module to process the high-dimensional features of the joints after step 2.1. The input is fed into the self-attention layer of the spatial encoder, and multi-head attention is used to model the positional relationships of the high-dimensional features of the key points:
[0021]
[0022]
[0023] In the formula, MSA(·) represents multi-head self-attention, and MLP(·) represents multilayer perceptron;
[0024] Step 2.3: Construct a time encoder module to extract sequence X. sp The global dependencies between spatial feature representations are established, and the temporal encoder utilizes a linear projection matrix. Temporal position embedding is performed on keypoints across different frames to capture their positional information. Then, MSA and MLP are used to model the temporal information of the keypoints to obtain sequence X. tp ;
[0025] Step 2.4: Construct a cross-encoder module for long sequences Downsampling is performed to keep the sequence length the same as the original sequence length. Cross-attention is then used to complement the feature information of the two sequences of different lengths.
[0026] X tp =X tp [N:-N] (5)
[0027] Q = X tp 'W Q K = X tp W K V = X tp W V (6)
[0028] X c =MSA(Q,K,V)+X tp (7)
[0029] In the formula, X tp 'Represents the sliced sequence data, This indicates a linear layer.
[0030] Step 3 is implemented in the following steps:
[0031] Step 3.1: Based on the connection relationship between human joints, three scales {S1, S2, S3} are designed respectively. Scale S1, which includes all 17 joints, extracts point features of each joint in a small receptive field; Scale S2, which includes 11 joint regions, extracts local feature information in a medium receptive field; and Scale S3, which includes 6 joint regions, extracts global feature information in a large receptive field.
[0032] Step 3.2: Construct a feature pooling matrix for the keypoint features processed in Step 2.4. As input to the multi-scale module, the 17 node regions at scale S1 are transformed into 11 node regions at scale S2 using the feature pooling matrix. Then, convolution is used to adaptively fuse the local features of the regions.
[0033]
[0034] In the formula, conv(·) represents two-dimensional convolution. Represents the feature pooling matrix;
[0035] Step 3.3: Construct a feature pooling matrix, which combines the keypoint features processed in Step 3.2. Perform downsampling again, and then use the feature pooling matrix. The 11 node regions at the S2 scale are transformed into 6 node regions at the S3 scale, and then the global features are adaptively fused.
[0036]
[0037] X”=Concat(X'0,X'1,...,X'5) (11)
[0038] Step 3.4: Construct a spatiotemporal encoder to process the multi-scale feature information X obtained from steps 2.4, 3.2, and 3.3. c X' and X" are used as inputs to the spatiotemporal encoder to learn the spatiotemporal features of different key points within the same frame and between adjacent frames.
[0039] Step 3.5: Process the multi-scale output X containing spatiotemporal feature information from step 3.4. 1 X 2 and X 3 Using the spatial solution pooling matrix to solve S 3 Key feature embedding at scale into S 2 middle:
[0040]
[0041]
[0042] In the formula, Represents the pooling matrix The transpose of the matrix, Stack(·), represents feature fusion;
[0043] Step 3.6: Process the two-dimensional pose sequence after step 3.5. Upsample again and utilize the spatial solution pooling matrix. S 2 Key feature embedding at scale into S 1 middle:
[0044]
[0045] Step 4 uses a linear layer to regress the high-dimensional features into three-dimensional pose key points.
[0046] Step 5 is implemented in the following steps:
[0047] Step 5.1: Minimize the error between the predicted pose and the true pose sequence using standard MPJPE;
[0048] Step 5.2: Construct the loss function and train using an end-to-end method.
[0049] L=λ w L w +λ t L t +λ m L m (18)
[0050] In the formula, L w For the weighted average joint position error, L t For time consistency loss, L m λ represents the average joint velocity error. w , λ t and λ m This represents the weight hyperparameter.
[0051] The beneficial effects of this invention are as follows:
[0052] This invention presents a three-dimensional human pose estimation method based on a multi-scale spatiotemporal encoder network. Under the premise of extracting multi-level joint semantic information using the U-net network architecture, it reduces the impact of changes in the length of the sequence to be predicted on the overall joint prediction accuracy. It extracts joints from two-dimensional human images in video frames and elevates the two-dimensional joint sequence to a three-dimensional joint sequence. Attached Figure Description
[0053] Figure 1 This is the overall network framework in the three-dimensional human pose estimation method based on a multi-scale spatiotemporal encoder network of the present invention;
[0054] Figure 2 The two-dimensional human image to be predicted is the input in step 1 of the three-dimensional human pose estimation method based on a multi-scale spatiotemporal encoder network of the present invention.
[0055] Figure 3 This is a diagram of the architecture of the spatiotemporal encoder in step 2 of the three-dimensional human pose estimation method based on a multi-scale spatiotemporal encoder network of the present invention.
[0056] Figure 4 This is a diagram of the cross-attention architecture in step 2 of the three-dimensional human pose estimation method based on a multi-scale spatiotemporal encoder network in this invention.
[0057] Figure 5 This is step 3, the multi-scale partitioning diagram of human joints, in the three-dimensional human pose estimation method based on a multi-scale spatiotemporal encoder network of the present invention.
[0058] Figure 6 This is the 3D prediction result in the 3D human pose estimation method based on a multi-scale spatiotemporal encoder network of the present invention. Detailed Implementation
[0059] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.
[0060] This invention relates to a three-dimensional human pose estimation method based on a multi-scale spatiotemporal encoder network. Specifically, it uses a spatiotemporal Transformer with U-net as the framework to elevate a two-dimensional human pose sequence into a three-dimensional human pose sequence.
[0061] Example 1
[0062] Among them, the 3D human pose estimation method based on multi-scale spatiotemporal encoder network, such as Figure 1 As shown, please follow these steps:
[0063] Step 1: Capture 2D human joints in video frames using a 2D pose detector, preprocess the joints, and feed them into the pose estimation pipeline. The input is a 2D human image. Figure 2 ;
[0064] Step 2: The sequence padding module is used to learn the spatiotemporal correlation of the joints in the original sequence and the padding sequence from Step 1. The spatiotemporal correlation is learned as follows: Figure 3 Cross-attention can be used to perform complementary feature information operations on two sequences of different lengths, such as... Figure 4 ;
[0065] Step 3: Using a multi-scale module, the key feature information from Step 2 is enhanced to strengthen the correlation between different scales of key features, local features, and global features according to the human skeleton topology. The multi-scale human body segmentation is as follows: Figure 5 ;
[0066] Step 4: Use a linear transformation layer to project the high-dimensional output of Step 3 into three-dimensional coordinates;
[0067] Step 5: Construct the loss function and use MPJPE to train the model end-to-end.
[0068] Example 2
[0069] A 3D human pose estimation method based on multi-scale spatiotemporal encoder network
[0070] Step 1, joint preprocessing, is implemented as follows:
[0071] Step 1.1: Input the video to be processed frame by frame into the two-dimensional pose detector to obtain the two-dimensional human pose sequence corresponding to the video frame;
[0072] Step 1.2: Stretch the joint coordinate information in the two-dimensional human pose sequence into a one-dimensional tensor vector to obtain the preprocessed joint information.
[0073] Step 2, the sequence filling module, learns the spatiotemporal correlation of key points, specifically as follows:
[0074] Step 2.1: Construct a sequence filling module to process the two-dimensional pose sequence after step 1.2. Perform sequence padding; the padded sequence is: Where F is the original sequence length, N is the padding sequence length, J is the number of joints, and the channel size is 2; First, the two-dimensional coordinates of each joint are spatially embedded through linear projection to preserve the spatial location information of the joints in each frame:
[0075]
[0076] In the formula, and represents the linear projection matrix and the spatial position embedding matrix, respectively, and x represents a key point;
[0077] Step 2.2: Construct a spatial encoder module to process the high-dimensional features of the joints after step 2.1. The input is fed into the self-attention layer of the spatial encoder, and multi-head attention is used to model the positional relationships of the high-dimensional features of the key points:
[0078]
[0079]
[0080] In the formula, MSA(·) represents multi-head self-attention, and MLP(·) represents multilayer perceptron;
[0081] Step 2.3: Construct a time encoder module to extract sequence X. sp The global dependencies between spatial feature representations are similar to those in step 2.2 of the spatial encoder process. The temporal encoder utilizes a linear projection matrix. Temporal position embedding is performed on keypoints across different frames to capture their positional information. Then, MSA and MLP are used to model the temporal information of the keypoints to obtain sequence X. tp ;
[0082] Step 2.4: Construct a cross-encoder module for long sequences Downsampling is performed to keep the sequence length the same as the original sequence length. Cross-attention is then used to complement the feature information of the two sequences of different lengths.
[0083] X tp =X tp [N:-N] (5)
[0084] Q = X tp 'W Q K = X tp W K V = X tp W V (6)
[0085] X c =MSA(Q,K,V)+X tp (7)
[0086] In the formula, X tp 'Represents the sliced sequence data, This represents a linear layer. Similar to the Transformer in NLP, step 2.4 selects key-value vectors and numerical vectors from the filled sequence and a query vector from the unfilled sequence. Then, an attention mechanism is used to map the feature sequence of length F+2N to a feature sequence of length F. Finally, MSA is used to obtain an output with the same dimension as the query vector. Sequence padding can supplement the lost temporal feature information in the first and last frames at the edges. Specifically, a sliding window method is used to perform step-by-step data sampling on the original sequence, and then a sampled sequence of length N*2 is used to pad the beginning and end of the original sequence.
[0087] Step 3, which enhances the correlation between different scales using the multi-scale module, is implemented as follows:
[0088] Step 3.1: Based on the connection relationship between human joints, three scales {S1, S2, S3} were designed. Scale S1, which includes all 17 joints, extracts point features of each joint in a small receptive field; Scale S2, which includes 11 joint regions, extracts local feature information in a medium receptive field; and Scale S3, which includes 6 joint regions, extracts global feature information in a large receptive field.
[0089] Step 3.2: Construct a feature pooling matrix for the keypoint features processed in Step 2.4. As input to the multi-scale module, the 17 node regions at scale S1 are transformed into 11 node regions at scale S2 using the feature pooling matrix. Then, convolution is used to adaptively fuse the local features of the regions.
[0090]
[0091] In the formula, conv(·) represents two-dimensional convolution. Represents the feature pooling matrix;
[0092] Step 3.3: Construct a feature pooling matrix, which combines the keypoint features processed in Step 3.2. Perform downsampling again, and then use the feature pooling matrix. The 11 node regions at the S2 scale are transformed into 6 node regions at the S3 scale, and then the global features are adaptively fused.
[0093]
[0094] X”=Concat(X'0,X'1,...,X'5) (11)
[0095] Step 3.4: Construct a spatiotemporal encoder to process the multi-scale feature information X obtained from steps 2.4, 3.2, and 3.3. c X' and X" are used as inputs to the spatiotemporal encoder to learn the spatiotemporal features of different key points within the same frame and between adjacent frames.
[0096] Step 3.5: Process the multi-scale output X containing spatiotemporal feature information from step 3.4. 1 X 2 and X 3 Using the spatial solution pooling matrix to solve S 3 Key feature embedding at scale into S 2 middle:
[0097]
[0098] In the formula, Represents the pooling matrix The transpose of the matrix is used to map low-scale features to high-scale features. Stack(·) represents feature fusion and is used to stack two tensors.
[0099] Step 3.6: Process the two-dimensional pose sequence after step 3.5. Upsample again and utilize the spatial solution pooling matrix. S 2 Key feature embedding at scale into S 1 middle:
[0100]
[0101] Step 4 uses a linear layer to regress the high-dimensional features into three-dimensional pose key points.
[0102] Step 5, constructing the loss function, is implemented as follows:
[0103] Step 5.1: Use the standard MPJPE (Mean Joint Position Error) to minimize the error between the predicted pose and the true pose sequence;
[0104] Step 5.2: Construct the loss function and train using an end-to-end method.
[0105] L=λ w L w +λ t L t +λ m L m (18)
[0106] In the formula, L w For the weighted average joint position error (WMPJPE), L t For time consistency loss (TCLoss), L m For the mean joint velocity error (MPJVE), λ w , λ t and λ m This represents the weight hyperparameter.
[0107] Example 3
[0108] Tables 1 and 2 show the experimental results using this method. The results indicate that the network model trained using the proposed method achieves high prediction accuracy on both the Human3.6M and MPI-INF-3DHP datasets. The predicted images are shown below. Figure 6 As shown:
[0109] Table 1. Comparison of results of the method of the present invention with mainstream methods on Human 3.6M, where the underlined result indicates the optimal result.
[0110] GLA-GCN 44.4 Uplifting 44.2 U-STN 43.9 EMHFormer 42.8 HSTFormer 42.7 HDFormer 42.6 Ours 42.0
[0111] Table 2. Comparison of the results of the method of the present invention with mainstream methods on MPI-INF-3DHP, where the underlined result indicates the best result.
[0112] UGCN 68.1 Anatomy3D 78.8 PoseFormer 77.1 CrossFormer 76.3 PATA 69.4 MHFormer 58.0 MixSTE 54.9 Uplifting 46.9 HSTFormer 41.4 Ours 36.4
[0113] This invention presents a three-dimensional human pose estimation method based on a multi-scale spatiotemporal encoder network. Under the premise of extracting multi-level joint semantic information using the U-net network architecture, it reduces the impact of changes in the length of the sequence to be predicted on the overall joint prediction accuracy. It extracts joints from two-dimensional human images in video frames and elevates the two-dimensional joint sequence to a three-dimensional joint sequence.
Claims
1. A three-dimensional human pose estimation method based on a multi-scale spatiotemporal encoder network, characterized in that, The specific steps are as follows: Step 1: Capture two-dimensional human joints in video frames using a two-dimensional pose detector, preprocess the joints, and feed them into the pose estimation pipeline. Step 2: Use the sequence filling module to learn the spatiotemporal correlation of the joints in the original sequence and the filling sequence in Step 1; Step 3: Use a multi-scale module to enhance the correlation between different scales of joint features, local features and global features based on the topology of the human skeleton; Step 4: Use a linear transformation layer to project the high-dimensional output of Step 3 into three-dimensional coordinates; Step 5: Construct the loss function and use MPJPE to train the model end-to-end; Step 3 is implemented in the following steps: Step 3.1: Based on the connection relationships between human joints, design three scales respectively. Scale including all 17 key points Point features of each joint are extracted within a small receptive field; the scale includes 11 joint regions. Local feature information is extracted within a medium-sized receptive field; the scale includes 6 keypoint regions. Extracting global feature information within a large receptive field; Step 3.2: Construct a feature pooling matrix for the keypoint features processed in Step 2. As input to the multi-scale module, and using the feature pooling matrix to... The 17 node regions at the scale are transformed into The system identifies 11 node regions at a given scale, and then uses convolution to adaptively fuse local features within these regions. In the formula, conv(•) represents two-dimensional convolution. Represents the feature pooling matrix; Step 3.3: Construct a feature pooling matrix, which combines the keypoint features processed in Step 3.
2. Perform downsampling again, and then use the feature pooling matrix. Will The 11 node regions at the scale are transformed into Six node regions at the scale are then adaptively fused with global features: Step 3.4: Construct a spatiotemporal encoder to process the multi-scale feature information obtained from steps 2, 3.2, and 3.
3. , and These are used as inputs to the spatiotemporal encoder to learn the spatiotemporal features of different key points within the same frame and between adjacent frames. Step 3.5: Process the multi-scale output containing spatiotemporal feature information from step 3.
4. , and Using the spatial solution pooling matrix to Key feature embedding at scale middle: In the formula, Represents the pooling matrix The transpose of the matrix, Stack(•), represents feature fusion; Step 3.6: Process the two-dimensional pose sequence after step 3.
5. Upsample again and utilize the spatial solution pooling matrix. Will Key feature embedding at scale middle:
2. The three-dimensional human pose estimation method based on a multi-scale spatiotemporal encoder network according to claim 1, characterized in that, Step 1 is implemented in the following steps: Step 1.1: Input the video to be processed frame by frame into the two-dimensional pose detector to obtain the two-dimensional human pose sequence corresponding to the video frame; Step 1.2: Stretch the joint coordinate information in the two-dimensional human pose sequence into a one-dimensional tensor vector to obtain the preprocessed joint information.
3. The three-dimensional human pose estimation method based on a multi-scale spatiotemporal encoder network according to claim 2, characterized in that, Step 2 is implemented as follows: Step 2.1: Construct a sequence filling module to process the two-dimensional pose sequence after step 1.
2. Perform sequence padding; the padded sequence is: Where F is the original sequence length, N is the padding sequence length, J is the number of key points, and the channel size is 2; The spatial position of each joint is embedded by linear projection to preserve the spatial position information of the joint in each frame: In the formula, and represents the linear projection matrix and the spatial position embedding matrix, respectively, and x represents a key point; Step 2.2: Construct a spatial encoder module to process the high-dimensional features of the joints after step 2.
1. The input is fed into the self-attention layer of the spatial encoder, and multi-head attention is used to model the positional relationships of the high-dimensional features of the key points: In the formula, MSA(•) represents multi-head self-attention, and MLP(•) represents multilayer perceptron; Step 2.3: Construct a time encoder module to extract the sequence. The global dependencies between spatial feature representations are established, and the temporal encoder utilizes a linear projection matrix. Temporal position embedding is performed on keypoints across different frames to capture their positional information. Then, MSA and MLP are used to model the temporal information of the keypoints to obtain the sequence. ; Step 2.4: Construct a cross-encoder module for long sequences Downsampling is performed to keep the sequence length the same as the original sequence length. Cross-attention is then used to complement the feature information of the two sequences of different lengths. In the formula, Represents the sliced sequence data. This indicates a linear layer.
4. The three-dimensional human pose estimation method based on a multi-scale spatiotemporal encoder network according to claim 1, characterized in that, In step 4, a linear layer is used to regress the high-dimensional features into three-dimensional pose joints. .
5. The three-dimensional human pose estimation method based on a multi-scale spatiotemporal encoder network according to claim 1, characterized in that, Step 5 is implemented in the following steps: Step 5.1: Minimize the error between the predicted pose and the true pose sequence using standard MPJPE; Step 5.2: Construct the loss function and train using an end-to-end method. In the formula, This is a weighted average of joint position errors. For time consistency loss, The average joint velocity error, , and This represents the weight hyperparameter.