3D human pose estimation method based on multi-level dual-branch cross attention

Abstract

The present invention discloses a 3D human pose estimation method based on multi-level dual-branch cross-attention. The method takes the video to be detected as input and uses a trained 3D human pose estimation model to predict the coordinates of human joints in the 2D image in 3D. Experimental results show that the prediction results of the present invention are closer to the real prediction results than the mainstream methods, enriching the method system of artificial intelligence and computer vision, and providing an option for improving the prediction accuracy in the field of 3D human pose estimation.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and artificial intelligence technology, and relates to a three-dimensional human posture estimation method based on multi-level dual-branch cross attention. Background Art

[0002] 3D human pose estimation (3D HPE) is a popular research area in computer vision. Its goal is to reconstruct the positions of key points of the human body in 3D space. 3D HPE has a wide range of applications, including human-computer interaction, motion analysis, virtual reality, and rehabilitation training. 3D HPE methods can be broadly categorized into two types: direct estimation and 2D-3D lifting. The former directly infers 3D pose from input images or videos, while the latter first converts sequences of human images into sequences of 2D key points and then lifts them to 3D key points. Although the accuracy and generalization of 2D HPE models have significantly improved, inferring 3D pose from 2D video images still faces challenges such as self-occlusion and depth ambiguity.

[0003] To eliminate these problems, CNN-based methods focus on using spatiotemporal graph convolutional neural networks to estimate 3D human poses, but this approach usually relies on dilation techniques, resulting in limited temporal connectivity of the network;

[0004] Because the Transformer architecture embeds an attention mechanism, it can flexibly model long-range dependencies in the input sequence, making it possible to use the Transformer for 3D human pose estimation. PoseFormer is the first to apply the Transformer to 3D human pose estimation. It groups all joints in each frame and then uses spatiotemporal information to estimate the pose of the center frame in the video. However, the human body joints are a topological structure, centered on the spine and extending to the limbs. The dependence of limb joints on their parent nodes often leads to high limb joint estimation errors, and the estimation errors of limb joints gradually accumulate as the complexity of the movement increases. Therefore, it is particularly important to partition the human joints and perform error analysis on each part.

[0005] Therefore, how to reduce the impact of limb joint error accumulation on overall joint prediction while using Transformer to capture long-range dependencies in posture sequences is an important issue that needs to be solved urgently. Summary of the Invention

[0006] The purpose of the present invention is to provide a three-dimensional human posture estimation method based on multi-level dual-branch cross attention, which can extract joint points of two-dimensional human images in video frames and enhance the two-dimensional joint point sequence to a three-dimensional joint point sequence.

[0007] The technical solution adopted by the present invention is a three-dimensional human posture estimation method based on multi-level dual-branch cross-attention, specifically using a spatiotemporal Transformer to upgrade a two-dimensional human posture sequence to a three-dimensional human posture sequence.

[0008] The present invention is also characterized in that:

[0009] The 3D human pose estimation method based on multi-level dual-branch cross attention is implemented in the following steps:

[0010] Step 1: Capture the 2D human joints in the video frame through a 2D posture detector, pre-process the joints, and feed them into pipelines at different levels;

[0011] Step 2: Use the spatial encoder to learn the spatial correlation of human joints at different levels in step 1;

[0012] Step 3: Use the joint partitioning temporal encoder to learn the temporal correlation of the joint feature information at different levels in step 2 according to the skeleton partitioning strategy;

[0013] Step 4: Recombining the feature information of the joint points with spatiotemporal correlation at each level in step 3;

[0014] Step 5: Use dual-branch cross attention to strengthen the correlation between outputs of different layers in step 4;

[0015] Step 6: Use a linear transformation layer to project the high-dimensional output of step 5 into three-dimensional coordinates;

[0016] Step 7: Construct the loss function and use MPJPE to train the model end-to-end.

[0017] The joint point preprocessing in step 1 is specifically implemented as follows:

[0018] Step 1.1: Input the video to be processed frame by frame into a two-dimensional posture detector to obtain a two-dimensional human posture sequence corresponding to the video frame;

[0019] Step 1.2: Stretch the joint point coordinate information in the two-dimensional human posture sequence into a one-dimensional tensor vector to obtain the preprocessed joint point information;

[0020] The learning of spatial correlation in step 2 is specifically implemented as follows:

[0021] Step 2.1, construct the spatial position embedding matrix, and process the two-dimensional posture sequence after step 1.2 Embed the spatial information, where J and T are the number of joint points and frames respectively, and the channel size is 2:

[0022] X′=LN(X)+E Sp (1)

[0023] Where LN(·) represents the LayerNorm layer, Represents the spatial position embedding matrix, X′ represents the output with spatial embedding information;

[0024] Step 2.2: Build a spatial encoder module, use the self-attention mechanism to process the feature information of all joints after the spatial information is embedded in step 2.1, and introduce the results into multiple layers to complete the processing of the spatial features of the frame;

[0025] Step 2.2 is specifically implemented as follows:

[0026] Step 2.2.1: Feed the output X′ obtained in step 2.1 into the spatial encoder module to obtain the output of the first layer:

[0027] X′=LN(X)+E Sp (2)

[0028] Where STE is Spatial Transformer Encoder, Represents the output of the first layer after the spatial encoder;

[0029] In step 2.2.2, the output of the previous layer is used as the input of the next layer to obtain three levels of joint feature information:

[0030]

[0031]

[0032] Where, and are the outputs of the second and third layers respectively;

[0033] The learning of the time correlation in step 3 is specifically implemented as follows:

[0034] Step 3.1: Divide the human body joints into five partitions according to the joint partitioning strategy, namely left arm, right arm, spine, left leg and right leg;

[0035] Step 3.2, construct the time position embedding matrix Embed the time information of each layer of joint points processed in step 2.2.2:

[0036]

[0037] Where, represents the output of the nth layer, where n∈[1, 2, 3], Represents the feature result obtained after time embedding of the nth layer;

[0038] Step 3.3, build the joint partition time encoder module, and embed the joint feature information after the time information in step 3.2 Divided into five parts Where i = 1, 2, 3, 4, 5, represents different groups, D is the embedding dimension, and then Input into the joint partition module to complete the modeling of the time correlation of each partition, and finally fuse the feature outputs of the five partitions into the original shape

[0039]

[0040] In the formula, Concat represents data concatenation, and TTE represents Temporal Transformer Encoder;

[0041] Step 3.4: Design a joint partition multilayer perceptron to merge the features of each partition processed in step 3.3, and apply a nonlinear factor to the merged features according to the number of joint points:

[0042]

[0043] Where BP-MLP stands for joint partition multi-layer perceptron, Represents the result of the nth layer after the joint partitioned time encoder;

[0044] The feature reorganization in step 4 is specifically implemented as follows:

[0045] Step 4.1: The output of the joint partitioned time encoder in step 3.4 is Relationship modeling is performed through multi-head self-attention (MHSA) and then fused into

[0046] Step 4.2: The feature vector Y fused in step 4.1 F Split evenly into original shapes To facilitate the subsequent steps;

[0047] The dual-branch cross attention in step 5 to enhance the correlation between layers is implemented as follows:

[0048] Step 5.1, obtain the vectors of each level after feature reorganization through step 4 The output of each two layers is taken as a group and divided into There are three groups in total;

[0049] Step 5.2: Construct a dual-branch cross attention module, using the two feature vectors of each group as the input of the dual branches to strengthen the correlation between the two feature vectors;

[0050] Step 5.3: Perform a residual connection between the two-branch feature vector obtained in step 5.2 and the first vector of each group to obtain a grouped output:

[0051]

[0052] Where, MSA l and MSA r is the two-branch eigenvector calculated in step 5.2, and Drop represents regularization;

[0053] Step 5.4: Repeat the process of steps 5.2 and 5.3 to strengthen the correlation between the other two sets of feature vectors;

[0054] Step 5.2 is specifically implemented as follows:

[0055] Step 5.2.1, using scaled dot product attention as the basis, the query matrix of each head Key Matrix Sum Matrix The scaled dot product attention calculation formula is as follows:

[0056]

[0057] Where N is the number of tokens and d is the dimension of each token;

[0058] In step 5.2.2, the two feature vectors of each group are input into the h-head multi-head self-attention respectively, and then the hyperparameter is multiplied by the second feature vector of the current group, and finally added to the output of the multi-head self-attention. The specific calculation process of the left branch is as follows:

[0059]

[0060] MSA l =Concat(head1, ..., head h )W o (11)

[0061] Where, head i is the attention calculation result of a single head, W o is the linear projection weight, H is a custom hyperparameter, and MSA is multi-head self-attention;

[0062] Step 5.2.3 is similar to the calculation process of step 5.2.2. Multiply the hyperparameter with the first eigenvector of the current group and then add it to the output of the multi-head self-attention. The specific calculation process of the right branch is as follows:

[0063]

[0064] MSA r =Concat(head1, ..., head h )W o (13);

[0065] The construction of the loss function in step 7 is implemented as follows:

[0066] Step 7.1, use standard MPJPE to minimize the error between the predicted pose and the true pose sequence;

[0067] Step 7.2, construct the loss function and calculate MPJPE:

[0068]

[0069] Where, and are the predicted 3D coordinates and the true 3D coordinates of the j-th joint in the t-th frame, respectively.

[0070] The beneficial effects of the present invention are:

[0071] The three-dimensional human pose estimation method based on multi-level dual-branch cross-attention of the present invention reduces the impact of error accumulation of limb joints on overall joint prediction while using Transformer to capture long-range dependencies in pose sequences. It can extract joint points of two-dimensional human images in video frames and upgrade two-dimensional joint point sequences to three-dimensional joint point sequences, enriching the method system of artificial intelligence and computer vision, and providing an option for improving prediction accuracy in the field of three-dimensional human pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 The two-dimensional human body image to be predicted is input in step 1 of the embodiment of the three-dimensional human body posture estimation method based on multi-level dual-branch cross attention of the present invention;

[0073] Figure 2 It is the three-dimensional prediction result of the three-dimensional human posture estimation method based on multi-level dual-branch cross attention of the present invention;

[0074] Figure 3 It is the overall network framework of the three-dimensional human posture estimation method based on multi-level dual-branch cross attention of the present invention;

[0075] Figure 4 is a temporal correlation graph of various body parts in the three-dimensional human posture estimation method based on multi-level dual-branch cross attention of the present invention;

[0076] Figure 5 3. It is a block diagram of the joint partitioning temporal encoder of step 3 in the embodiment of the method for 3D human pose estimation based on multi-level dual-branch cross attention of the present invention;

[0077] Figure 6 It is the dual-branch cross-attention graph in step 5 of the embodiment of the three-dimensional human posture estimation method based on multi-level dual-branch cross-attention of the present invention. DETAILED DESCRIPTION

[0078] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0079] The present invention provides a three-dimensional human posture estimation method based on multi-level dual-branch cross attention, specifically using a spatiotemporal Transformer to upgrade a two-dimensional human posture sequence to a three-dimensional human posture sequence.

[0080] Example 1

[0081] Among them, the 3D human pose estimation method based on multi-level dual-branch cross attention is Figure 3 As shown, the specific implementation steps are as follows:

[0082] Step 1: Capture the 2D human joints in the video frame through the 2D posture detector, pre-process the joints, and feed them into the pipelines at different levels. The input 2D human image is Figure 1 ;

[0083] Step 2: Use the spatial encoder to learn the spatial correlation of human joints at different levels in step 1;

[0084] Step 3, such as Figure 5 As shown in the figure, the joint partitioning temporal encoder is used to learn the temporal correlation of the joint feature information at different levels in step 2 according to the skeleton partitioning strategy;

[0085] Step 4: Recombining the feature information of the joint points with spatiotemporal correlation at each level in step 3;

[0086] Step 5, such as Figure 6 As shown in Figure 2, a dual-branch cross attention is used to strengthen the correlation between the outputs of different layers in step 4;

[0087] Step 6: Use a linear transformation layer to project the high-dimensional output of step 5 into three-dimensional coordinates;

[0088] Step 7: Construct the loss function and use MPJPE to train the model end-to-end.

[0089] Example 2

[0090] The joint point preprocessing in step 1 of the three-dimensional human posture estimation method based on multi-level dual-branch cross attention of the present invention is specifically implemented according to the following steps:

[0091] Step 1.1: Input the video to be processed frame by frame into a two-dimensional posture detector to obtain a two-dimensional human posture sequence corresponding to the video frame;

[0092] Step 1.2: Stretch the joint point coordinate information in the two-dimensional human posture sequence into a one-dimensional tensor vector to obtain the preprocessed joint point information;

[0093] The learning of spatial correlation in step 2 is specifically implemented as follows:

[0094] Step 2.1, construct the spatial position embedding matrix, and process the two-dimensional posture sequence after step 1.2 Embed the spatial information, where J and T are the number of joint points and frames respectively, and the channel size is 2:

[0095] X′=LN(X)+E Sp (1)

[0096] Where LN(·) represents the LayerNorm layer, Represents the spatial position embedding matrix, X′ represents the output with spatial embedding information;

[0097] Step 2.2: Build a spatial encoder module, use the self-attention mechanism to process the feature information of all joints after the spatial information is embedded in step 2.1, and introduce the results into multiple layers to complete the processing of the spatial features of the frame:

[0098] Step 2.2.1: Feed the output X′ obtained in step 2.1 into the spatial encoder module to obtain the output of the first layer:

[0099] X′=LN(X)+E Sp (2)

[0100] Where STE is Spatial Transformer Encoder, Represents the output of the first layer after the spatial encoder;

[0101] In step 2.2.2, the output of the previous layer is used as the input of the next layer to obtain three levels of joint feature information:

[0102]

[0103]

[0104] Where, and are the outputs of the second and third layers respectively;

[0105] The learning of time correlation in step 3 is specifically implemented as follows: Figure 4 As shown:

[0106] Step 3.1: Divide the human body joints into five partitions according to the joint partitioning strategy, namely left arm, right arm, spine, left leg and right leg;

[0107] Step 3.2, construct the time position embedding matrix Embed the time information of each layer of joint points processed in step 2.2.2:

[0108]

[0109] Where, represents the output of the nth layer, where n∈[1, 2, 3], Represents the feature result obtained after time embedding of the nth layer;

[0110] Step 3.3, build the joint partition time encoder module, and embed the joint feature information after the time information in step 3.2 Divided into five parts Where i = 1, 2, 3, 4, 5, represents different groups, D is the embedding dimension, and then Input into the joint partition module to complete the modeling of the time correlation of each partition, and finally fuse the feature outputs of the five partitions into the original shape

[0111]

[0112] In the formula, Concat represents data concatenation, and TTE represents Temporal Transformer Encoder;

[0113] Step 3.4: Design a joint partition multilayer perceptron to merge the features of each partition processed in step 3.3, and apply a nonlinear factor to the merged features according to the number of joint points:

[0114]

[0115] Where BP-MLP stands for joint partition multi-layer perceptron, Represents the result of the nth layer after the joint partitioned time encoder;

[0116] The feature reorganization in step 4 is specifically implemented as follows:

[0117] Step 4.1: The output of the joint partitioned time encoder in step 3.4 is Relationship modeling is performed through multi-head self-attention (MHSA) and then fused into

[0118] Step 4.2: The feature vector Y fused in step 4.1 F Split evenly into original shapes To facilitate the subsequent steps;

[0119] The dual-branch cross attention in step 5 to enhance the correlation between layers is implemented as follows:

[0120] Step 5.1, obtain the vectors of each level after feature reorganization through step 4 The output of each two layers is taken as a group and divided into There are three groups in total;

[0121] Step 5.2: Construct a dual-branch cross attention module, using the two feature vectors of each group as the input of the dual branches to strengthen the correlation between the two feature vectors:

[0122] Please follow the steps below to implement:

[0123] Step 5.2.1, using scaled dot product attention as the basis, the query matrix of each head Key Matrix Sum Matrix The scaled dot product attention calculation formula is as follows:

[0124]

[0125] Where N is the number of tokens and d is the dimension of each token;

[0126] In step 5.2.2, the two feature vectors of each group are input into the h-head multi-head self-attention respectively, and then the hyperparameter is multiplied by the second feature vector of the current group, and finally added to the output of the multi-head self-attention. The specific calculation process of the left branch is as follows:

[0127]

[0128] MSA l =Concat(head1, ..., head h )W o (11)

[0129] Where, head i is the attention calculation result of a single head, W o is the linear projection weight, H is a custom hyperparameter, and MSA is multi-head self-attention;

[0130] Step 5.2.3 is similar to the calculation process of step 5.2.2. Multiply the hyperparameter with the first eigenvector of the current group and then add it to the output of the multi-head self-attention. The specific calculation process of the right branch is as follows:

[0131]

[0132] MSA r =Concat(head1, ..., head h )W o (13);

[0133] Step 5.3: Perform a residual connection between the two-branch feature vector obtained in step 5.2 and the first vector of each group to obtain a grouped output:

[0134]

[0135] Where, MSA l and MSA r is the two-branch eigenvector calculated in step 5.2, and Drop represents regularization;

[0136] Step 5.4: Repeat the process of steps 5.2 and 5.3 to strengthen the correlation between the other two sets of feature vectors;

[0137] The construction of the loss function in step 7 is implemented as follows:

[0138] In step 7.1, the standard MPJPE (mean position error per joint) is used to minimize the error between the predicted pose and the true pose sequence.

[0139] Step 7.2, construct the loss function and calculate MPJPE:

[0140]

[0141] Where, and are the predicted 3D coordinates and the true 3D coordinates of the j-th joint in the t-th frame, respectively.

[0142] Example 3

[0143] The experimental results are shown in Table 1 and Table 2. The results show that the network model trained by the method proposed in this invention has achieved low error results on both Human3.6M and MPI-INF-3DHP datasets; the predicted result images are as follows Figure 2 As shown:

[0144] Table 1. Comparison of the results of the method of the present invention with those of mainstream methods on Human3.6M, where the underlined results indicate the best results

[0145] Method MPJPE(mm) PoseFormer 44.3 Anatomy3D 44.1 MHFormer 43.0 P-STMO 42.8 MHFormer++ 42.5 Ours <![CDATA[ 42.3 ]]>

[0146] Table 2. Comparison of the results of the method of the present invention with those of mainstream methods on MPI-INF-3DHP, where the underlined results indicate the best results

[0147] Method MPJPE(mm) Anatomy3D 78.8 PoseFormer 77.1 MHFormer 58.0 MHFormer++ 54.0 Ours <![CDATA[ 39.6 ]]>

Claims

1. A 3D human pose estimation method based on multi-level dual-branch cross attention, characterized by: Specifically, a spatiotemporal Transformer is used to upgrade the two-dimensional human posture sequence to a three-dimensional human posture sequence; Please follow the steps below to implement: Step 1: Capture the 2D human joints in the video frame through a 2D posture detector, pre-process the joints, and feed them into pipelines at different levels; Step 2: Use the spatial encoder to learn the spatial correlation of human joints at different levels in step 1; Step 3: Use the joint partitioning temporal encoder to learn the temporal correlation of the joint feature information at different levels in step 2 according to the skeleton partitioning strategy; Step 4: Recombining the feature information of the joint points with spatiotemporal correlation at each level in step 3; Step 5: Use dual-branch cross attention to strengthen the correlation between outputs of different layers in step 4; Step 6: Use a linear transformation layer to project the high-dimensional output of step 5 into three-dimensional coordinates; Step 7: Construct the loss function and use MPJPE to train the model end-to-end. The learning of the time correlation in step 3 is specifically implemented according to the following steps: Step 3.1: Divide the human body joints into five partitions according to the joint partitioning strategy, namely left arm, right arm, spine, left leg and right leg; Step 3.2, construct the time position embedding matrix , embed the time information of each layer of joint points processed in step 2: Where, Representative The output of the layer, , Representative The feature results obtained after the layer is temporally embedded; Step 3.3, build the joint partition time encoder module, and embed the joint feature information after the time information in step 3.2 Divided into five parts ,in , representing different groups, D is the embedding dimension, and then Input into the joint partition module to complete the modeling of the time correlation of each partition, and finally fuse the feature outputs of the five partitions into the original shape : Where, represents data concatenation, and TTE stands for Temporal Transformer Encoder; Step 3.4: Design a joint partition multilayer perceptron to merge the features of each partition processed in step 3.3, and apply a nonlinear factor to the merged features according to the number of joint points: Where BP-MLP stands for joint partition multi-layer perceptron, Representative The result of the layer passing through the joint partitioned temporal encoder; The feature reorganization in step 4 is specifically implemented according to the following steps: Step 4.1: The output of the joint partitioned time encoder in step 3.4 is Relationship modeling is performed through multi-head self-attention and then fused into ; Step 4.2: merge the feature vectors from step 4.1 Split evenly into original shapes , to facilitate the subsequent steps; The dual-branch cross attention in step 5 to enhance the correlation between layers is specifically implemented as follows: Step 5.1, obtain the vectors of each level after feature reorganization through step 4 , taking the output of each two levels as a group, divided into There are three groups in total; Step 5.2: Construct a dual-branch cross attention module, using the two feature vectors of each group as the input of the dual branches to strengthen the correlation between the two feature vectors; Step 5.3: Perform a residual connection between the two-branch feature vector obtained in step 5.2 and the first vector of each group to obtain a grouped output: Where, and is the two-branch eigenvector calculated in step 5.2, represents regularization; In step 5.4, repeat the process of steps 5.2 and 5.3 to strengthen the correlation between the other two groups of feature vectors.

2. The 3D human pose estimation method based on multi-level dual-branch cross attention according to claim 1, characterized in that: The joint point preprocessing in step 1 is specifically implemented according to the following steps: Step 1.1: Input the video to be processed frame by frame into a two-dimensional posture detector to obtain a two-dimensional human posture sequence corresponding to the video frame; Step 1.2: Stretch the joint point coordinate information in the two-dimensional human posture sequence into a one-dimensional tensor vector to obtain the preprocessed joint point information.

3. The 3D human pose estimation method based on multi-level dual-branch cross attention according to claim 2, characterized in that: The learning of spatial correlation in step 2 is specifically implemented according to the following steps: Step 2.1, construct the spatial position embedding matrix, and process the two-dimensional posture sequence after step 1.2 Embed the spatial information, where J and T are the number of joint points and frames respectively, and the channel size is 2: Where LN(·) represents the LayerNorm layer, represents the spatial position embedding matrix, Represents the output with spatial embedding information; In step 2.2, a spatial encoder module is constructed. The self-attention mechanism is used to process the feature information of all joints after the spatial information is embedded in step 2.1, and the results are introduced into multiple layers to complete the processing of the spatial features of the frame.

4. The 3D human pose estimation method based on multi-level dual-branch cross attention according to claim 3, characterized in that: The step 2.2 is specifically implemented as follows: Step 2.2.1, the output obtained in step 2.1 Send it to the spatial encoder module and get the output of the first layer: Where STE is Spatial Transformer Encoder, Represents the output of the first layer after the spatial encoder; In step 2.2.2, the output of the previous layer is used as the input of the next layer to obtain three levels of joint feature information: Where, and are the outputs of the second and third layers respectively.

5. The 3D human pose estimation method based on multi-level dual-branch cross attention according to claim 1, characterized in that: The step 5.2 is specifically implemented as follows: Step 5.2.1, using scaled dot product attention as the basis, the query matrix of each head , key matrix Sum Matrix The scaled dot product attention calculation formula is as follows: Where, is the number of tokens, is the dimension of each token; In step 5.2.2, the two feature vectors of each group are input into the h-head multi-head self-attention respectively, and then the hyperparameter is multiplied by the second feature vector of the current group, and finally added to the output of the multi-head self-attention. The specific calculation process of the left branch is as follows: Where, is the attention calculation result of a single head, is the linear projection weight, is a custom hyperparameter, It is multi-headed self-attention; Step 5.2.3 is similar to the calculation process of step 5.2.

2. Multiply the hyperparameter with the first eigenvector of the current group and then add it to the output of the multi-head self-attention. The specific calculation process of the right branch is as follows:

6. The 3D human pose estimation method based on multi-level dual-branch cross attention according to claim 1, characterized in that: The construction of the loss function in step 7 is specifically implemented as follows: Step 7.1, use standard MPJPE to minimize the error between the predicted pose and the true pose sequence; Step 7.2, construct the loss function and calculate the average position error MPJPE per joint: Where, and are the predicted 3D coordinates and the true 3D coordinates of the j-th joint in the t-th frame, respectively.

Images