A 3D pose estimation method based on a dynamic limb constraint Transformer network model

Abstract

A kind of 3D pose estimation method based on dynamic limb constraint Transformer network model, first, dynamic limb constraint Transformer network model is built, model includes image feature embedding module, Transformer encoder module and MLP module, each module contains the corresponding Layer norm layer of convolution level;Then select training set and test set, and set the training parameter of dynamic limb constraint Transformer network model;According to dynamic limb constraint Transformer network model and its training parameter, the dynamic limb constraint Transformer network model is trained with the minimum loss function as the goal;Finally, the 2D pose sequence to be processed is input into the 3D pose estimation model based on dynamic limb constraint Transformer network, and the corresponding 3D pose is output;The 3D pose obtained by the application has the advantages of high precision, fast calculation, low resource consumption and the like.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and image processing, and particularly relates to a 3D pose estimation method based on a dynamic limb constraint Transformer network model. BACKGROUND

[0002] Estimating three-dimensional human poses from monocular color images or videos is a basic problem in the field of computer vision and image processing, including the positions of key points in space and the relationships between them. Many subsequent research problems are based on this, including human-computer interaction, three-dimensional reconstruction, motion capture, medical rehabilitation, etc. Existing 3D human pose estimation methods include single-stage methods and two-stage methods. Single-stage methods directly regress 3D key points from input images; two-stage methods use a 2D key point detection model to detect 2D key points from input images, and then use the 2D key points as input to predict 3D key points. However, since a two-dimensional human pose is a projection of a three-dimensional human pose in the coronal plane, there are multiple three-dimensional poses corresponding to one two-dimensional pose.

[0003] Existing methods (Zheng C, Zhu S, Mendieta M, et al. 3d human pose estimation with spatial and temporal transformers [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 11656-11665.) based on the Transformer structure solve the three-dimensional pose of the intermediate frame from multiple consecutive input images, and combine the temporal and spatial features in the consecutive frames to improve the calculation accuracy and efficiency. However, since it only considers the spatial positions of the key points and the spatial relationships between them in the human pose estimation process, it ignores the correlation between the limbs during the movement of the human body, so the accuracy of the obtained three-dimensional pose is low. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide a 3D pose estimation method based on a dynamic limb constraint Transformer network model, which obtains a more accurate 3D pose, has the advantages of high precision, fast calculation, and low resource consumption.

[0005] To achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0006] A 3D pose estimation method based on a dynamic limb constraint Transformer network model, comprising the following steps:

[0007] Step one: build a dynamic limb constraint Transformer network model, the dynamic limb constraint Transformer network model comprising an image feature embedding module, a Transformer encoder module and an MLP module;

[0008] Step two: select a training set and a test set, and set the training parameters of the dynamic limb constraint Transformer network model;

[0009] Step three: according to the dynamic limb constraint Transformer network model and its training parameters, minimize the loss function, thereby completing the training of the dynamic limb constraint Transformer network model;

[0010] Step four: input the 2D pose sequence to be processed into the trained dynamic limb constraint Transformer network model to obtain the corresponding 3D pose.

[0011] The image feature embedding module in step one comprises a convolution layer and a linear connection layer, the convolution kernel size of the convolution layer is 17x17, and the number of convolution kernels is 768.

[0012] The Transformer encoder module in step one comprises 4 time feature extractors and 4 space feature extractors, each feature extractor comprises 4 convolution layers with a convolution kernel of 1x1, each convolution layer is provided with a corresponding LayerNorm layer in front, the time feature extractor and the space feature extractor work alternately, and the time feature and the space feature in the image are extracted respectively.

[0013] The MLP module in step one comprises a linear layer and a LayerNorm layer.

[0014] The training set and the test set in step two adopt the training / test split in the Human3.6M dataset, the network is trained by the training set, and the network is tested by the test set.

[0015] The weight initialization of the dynamic limb constraint Transformer network model in step two is performed by a random initialization method.

[0016] The loss function in step three is a combined loss function Loss total =L w +λ t L t +λ s L s +λ a L a , wherein, represents the error of the predicted key point and the real key point; represents the error of the predicted key point in two adjacent frames; represents the error of the predicted limb length in the symmetric position; represents the error of the predicted limb length in two adjacent frames; wherein N represents the number of all key points, T represents the length of the input sequence, P represents the number of all limbs, F represents the limb length between two predicted key points, V represents the number of symmetric limbs, r and l represent the right side and the left side respectively, W, λ t , λ s , λ a are weight coefficients; p i,j is the predicted position of the i-th frame image and the j-th key point, gt i,j is the actual real position of the i-th frame image and the j-th key point, p i,j-1 is the predicted position of the i-th frame image and the j-1-th key point, F i,j(r) is the predicted length of the right j-th limb of the i-th frame image, F i,j(l) is the predicted length of the left j-th limb of the i-th frame image, F i,k is the predicted length of the k-th limb of the i-th frame image, F i-1,k is the predicted length of the k-th limb of the i-1-th frame image.

[0017] The step three of minimizing the loss function adopts the AdamW optimization method.

[0018] Compared with the prior art, the present application has the following beneficial effects:

[0019] (1) The 3D pose estimation method of the present application is based on a Transformer model, and by introducing dynamic limb constraints, the model learning and generalization ability is greatly enhanced, and a stable mapping relationship from 2D pose to 3D pose can be established in different use environments;

[0020] (2) The dynamic limb constraint Transformer network model established by the present application can accurately estimate 3D pose on a data set, and can effectively improve the estimation accuracy and efficiency in actual use scenarios;

[0021] (3) The human pose estimation method based on the dynamic limb constraint Transformer network model of the present application constructs a 2D to 3D human pose estimation network, has faster calculation efficiency, and can fully utilize the temporal and spatial features in the image to obtain human 3D pose information. BRIEF DESCRIPTION OF DRAWINGS

[0022] Figure 1 is a flowchart of the method of the embodiment of the present application.

[0023] Figure 2 A dynamic limb constraint Transformer network model schematic diagram for an embodiment of the present application. DETAILED DESCRIPTION

[0024] The present application will be described in detail below with reference to embodiments and drawings.

[0025] As Figure 1 shown, a 3D pose estimation method based on a dynamic limb constraint Transformer network model greatly enhances the learning ability and generalization ability of the network by means of the feature extraction ability of the time feature extractor and the space feature extractor, accurately learns the mapping relationship from 2D pose to 3D pose, and thus can accurately estimate 3D pose through the learned mapping relationship; specifically comprising the following steps:

[0026] Step one: build a dynamic limb constraint Transformer network model, adopt a self-supervised learning method, in order to accurately estimate 3D pose, the dynamic limb constraint Transformer network model includes an image feature embedding module, a Transformer encoder module and an MLP module;

[0027] The image feature embedding module of the present embodiment includes a convolution layer and a linear connection layer, the convolution kernel size of the convolution layer is 17x17, and the number of convolution kernels is 768; the image feature embedding module uses multiple convolution kernels to segment the original input image, and embeds the key point position features and limb motion features, thereby obtaining multiple image blocks as input for pose estimation;

[0028] Referring to Figure 2 , the Transformer encoder module includes 4 repeated time feature extractors and space feature extractors, each feature extractor includes 4 convolution layers with a convolution kernel of 1x1, and a corresponding LayerNorm layer is added in front of each convolution layer; the time feature extractor and the space feature extractor work alternately, and extract the time feature and the space feature in the image respectively;

[0029] The MLP module includes a linear layer and a LayerNorm layer;

[0030] Step two: select a training set and a test set, and set the training parameters of the dynamic limb constraint Transformer network model;

[0031] In the present embodiment, the training set and the test set contain 2D pose and corresponding 3D pose, the 2D pose has two coordinate channels of x and y, and the 3D pose has three coordinate channels of x, y and z;

[0032] In this embodiment, the training / test split in the Human3.6M dataset widely used in the field of pose estimation is selected as the training set and the test set, the network is trained through the training set, and the network is tested through the test set; before the method of estimating 3D pose based on the dynamic limb constraint Transformer network model is adopted, the dynamic limb constraint Transformer network model needs to be pre-trained, and then the training parameters of the dynamic limb constraint Transformer network model are set; the learning rate is set to 0.001, the decay rate of the learning rate is set to 0.1, and the strategy of decaying the learning rate once every 20 epochs is adopted; the weight parameter initialization of the dynamic limb constraint Transformer network model adopts the random initial method, the batch size is 1024, and the total training times are 200 times;

[0033] Step three: according to the dynamic limb constraint Transformer network model and its training parameters, the loss function is minimized, the loss function is minimized by using the AdamW optimization method, so as to complete the training of the dynamic limb constraint Transformer network model;

[0034] In order to make accurate estimation of 3D pose, the combined loss function is used in this embodiment; in the i-th frame image, the predicted position of the j-th key point is p i,j , and the actual real position is gt i,j , because different key points have different weights when predicting, the weighted prediction error corresponding to the input sequence is represented as The error of the predicted key point on the adjacent two frames is represented as The error of the predicted limb length on the symmetric position is represented as: The error of the predicted limb length on the adjacent two frames is represented as: The final loss function Loss total = L w +λ t L t +λ s L s +λ a L a , wherein N represents the number of all key points, T represents the length of the input sequence, P represents the number of all limbs, F represents the limb length between two predicted key points, V represents the number of symmetric limbs, r and l represent the right side and the left side respectively, W, λ t , λ s , λ a are weight coefficients; p i,j is the predicted position of the i-th frame image and the j-th key point, gt i,j is the actual real position of the i-th frame image and the j-th key point, pi,j-1 is the predicted position of the j-1th key point of the i-th image, F i,j(r) is the predicted length of the right j-limbs of the i-th image, F i,j(l) is the predicted length of the left j-limbs of the i-th image, F i,k is the predicted length of the k-limbs of the i-th image, F i-1,k is the predicted length of the k-limbs of the i-1-th image.

[0035] Step four: input the 2D pose sequence to be processed into the trained dynamic limb constraint Transformer network model to obtain the corresponding 3D pose.

[0036] The 3D pose estimation method based on the dynamic limb constraint Transformer network model of the embodiment greatly enhances the learning ability of the dynamic limb constraint Transformer network model through dynamic limb constraint, establishes a strong mapping relationship from 2D pose to 3D pose, introduces an image feature embedding module to extract and embed the time features and space features contained in the original image, and uses the time features and space features for subsequent 3D pose estimation, introduces a Transformer encoder module to repeatedly calculate the attention between key points, thereby improving the global accuracy of 3D pose estimation, trains the network model based on the dynamic limb constraint Transformer on the Human3.6M dataset to construct a stable mapping from 2D pose to 3D pose, finally obtains a 3D pose close to the real one, and proves that the application has good accuracy and universality.

Claims

1. A 3D pose estimation method based on a dynamic limb constraint Transformer network model, characterized in that, The method comprises the following steps: Step 1: building a dynamic limb constraint Transformer network model, wherein the dynamic limb constraint Transformer network model comprises an image feature embedding module, a Transformer encoder module and an MLP module; Step 2: selecting a training set and a test set, and setting training parameters of the dynamic limb constraint Transformer network model; Step 3: minimizing a loss function according to the dynamic limb constraint Transformer network model and the training parameters, thereby completing training of the dynamic limb constraint Transformer network model; Step 4: inputting a 2D pose sequence to be processed into the trained dynamic limb constraint Transformer network model to obtain a corresponding 3D pose. The loss function in step three is a combined loss function Loss total = L w + λ t L t + λ s L s + λ a L a , wherein represents the error between the predicted key points and the real key points; represents the error between the predicted key points in two adjacent frames; represents the error between the predicted limb length in the symmetric position; represents the error between the predicted limb length in two adjacent frames; wherein N represents the number of all key points, T represents the length of the input sequence, P represents the number of all limbs, F represents the limb length between two predicted key points, V represents the number of symmetric limbs, r and l respectively represent the right side and the left side, W, λ t , λ s , λ a are weight coefficients; p i,j is the predicted position of the i-th frame image and the j-th key point, gt i,j is the actual real position of the i-th frame image and the j-th key point, p i,j-1 is the predicted position of the i-th frame image and the j-1-th key point, F i,j(r) is the predicted length of the right j-th limb of the i-th frame image, F i,j(l) is the predicted length of the left j-th limb of the i-th frame image, F i,k is the predicted length of the k-th limb of the i-th frame image, F i-1,k is the predicted length of the k-th limb of the i-1-th frame image.

2. The 3D pose estimation method of claim 1, wherein: The image feature embedding module in step 1 comprises a convolution layer and a linear connection layer, and the convolution layer has a convolution kernel size of 17*17 and a convolution kernel number of 768.

3. The 3D pose estimation method of claim 1, wherein: The Transformer encoder module in step 1 comprises four time feature extractors and four space feature extractors, each feature extractor comprises four convolution layers with a convolution kernel of 1*1, and each convolution layer is provided with a corresponding LayerNorm layer in front, and the time feature extractors and the space feature extractors work alternately to extract time features and space features in the image respectively.

4. The 3D pose estimation method of claim 1, wherein: The MLP module in step 1 comprises a linear layer and a LayerNorm layer.

5. The 3D pose estimation method of claim 1, wherein: In step 2, the training set and the test set adopt training / test segmentation in a Human3.6M dataset, the network is trained by using the training set, and the network is tested by using the test set.

6. The 3D pose estimation method of claim 1, wherein: In step 2, the weight initialization of the dynamic limb constraint Transformer network model is performed by a random initialization method.

7. The 3D pose estimation method of claim 1, wherein: In step 3, the AdamW optimization method is used to minimize the loss function.

Images