3D attitude estimation method based on dynamic limb constraint Transform network model

Abstract

A 3D attitude estimation method based on a dynamic limb constraint Transform network model comprises the steps that the dynamic limb constraint Transform network model is firstly built, the model comprises an image feature embedding module, a Transform encoder module and an MLP module, and each module comprises a Layer norm layer corresponding to a convolutional hierarchy; then selecting a training set and a test set, and setting training parameters of the dynamic limb constraint Transform network model; according to the dynamic limb constraint Transform network model and training parameters thereof, training the dynamic limb constraint Transform network model by taking minimization of a loss function as a target; and finally, inputting the 2D attitude sequence to be processed into the 3D attitude estimation model based on the dynamic limb constraint Transform network, and outputting a corresponding 3D attitude. The 3D attitude obtained by the method has the advantages of high precision, fast calculation, low resource consumption and the like.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision and image processing, and in particular relates to a 3D posture estimation method based on a dynamic limb constraint Transformer network model. Background Art

[0002] Estimating 3D human pose from monocular color images or videos is a fundamental problem in the field of computer vision and image processing, including the position of each key point in space and the relationship between them. Many subsequent research problems are based on this, including human-computer interaction, 3D reconstruction, motion capture, medical rehabilitation, etc. Existing 3D human pose estimation methods include single-stage methods and two-stage methods. The single-stage method directly regresses 3D key points from the input image; the two-stage method uses a 2D key point detection model to detect 2D key points from the input image, and then uses the 2D key points as input to predict the 3D key points. However, since a 2D human pose is the projection of a 3D human pose on the coronal plane, there is a situation where a 2D pose corresponds to multiple 3D poses.

[0003] Existing methods (Zheng C, Zhu S, Mendieta M, et al. 3D human pose estimation with spatial and temporal transformers [C] / / Proceedings of the IEEE / CVF international conference on computer vision. 2021: 11656-11665.) solve the 3D pose of the intermediate frames from multiple frames of continuous input images based on the Transformer structure, and combine the temporal and spatial features in the continuous frames to improve the calculation accuracy and efficiency; however, since it simply considers the spatial position of each key point in the process of human pose estimation and the spatial relationship between them, it ignores the relationship between the limbs of the human body during movement, so the obtained 3D pose accuracy is low. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide a 3D posture estimation method based on a dynamic limb constraint Transformer network model, which obtains a more accurate 3D posture and has the advantages of high precision, fast calculation, and low resource consumption.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A 3D posture estimation method based on a dynamic limb constraint Transformer network model comprises the following steps:

[0007] Step 1: Build a dynamic limb constraint Transformer network model, which includes an image feature embedding module, a Transformer encoder module, and an MLP module.

[0008] Step 2: Select the training set and test set, and set the training parameters of the dynamic limb constraint Transformer network model;

[0009] Step 3: According to the dynamic limb constraint Transformer network model and its training parameters, the loss function is minimized to complete the training of the dynamic limb constraint Transformer network model;

[0010] Step 4: Input the 2D posture sequence to be processed into the trained dynamic limb constraint Transformer network model to obtain the corresponding 3D posture.

[0011] The image feature embedding module in step 1 includes a convolution layer and a linear connection layer, the convolution kernel size of the convolution layer is 17×17, and the number of convolution kernels is 768.

[0012] The Transformer encoder module in the step 1 includes 4 temporal feature extractors and 4 spatial feature extractors, each feature extractor includes 4 convolutional layers with a convolution kernel of 1×1, and a corresponding LayerNorm layer is added in front of each convolutional layer. The temporal feature extractor and the spatial feature extractor work in turn to extract the temporal features and spatial features in the image respectively.

[0013] The MLP module in step 1 includes a linear layer and a LayerNorm layer.

[0014] In the step 2, the training set and the test set adopt the training / test split in the Human3.6M data set, the network is trained by the training set, and the network is tested by the test set.

[0015] In the step 2, the weight initialization of the dynamic limb constraint Transformer network model is performed by a random initialization method.

[0016] The loss function in step 3 is the combined loss function Loss total =L w +λ t L t +λ s L s +λ a L a ,in, Represents the error between the predicted key point and the true key point; Represents the error of the predicted key point in two adjacent frames; represents the error in predicting limb length in symmetric positions; represents the error of the predicted limb length in two adjacent frames; where N represents the number of all key points, T represents the length of the input sequence, P represents the number of all limbs, F represents the limb length between two predicted key points, V represents the number of symmetrical limbs, r and l represent the right and left sides respectively, W,λ t ,λ s ,λ a is the weight coefficient; p i,j is the predicted position of the jth key point in the i-th frame image, gt i,j is the actual position of the jth key point in the i-th frame image, p i,j-1 is the predicted position of the j-1th key point in the i-th frame image, F i,j(r) is the predicted length of the jth limb on the right side of the i-th frame image, F i,j(l) is the predicted length of the jth limb on the left side of the i-th frame image, F i,k is the predicted length of the kth limb in the i-th frame image, F i-1,k is the predicted length of the kth limb in the i-1th frame image.

[0017] In step 3, the AdamW optimization method is used to minimize the loss function.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] (1) The 3D posture estimation method of the present invention is based on the Transformer model. By introducing dynamic limb constraints, the model learning and generalization capabilities are greatly enhanced, and a strong stable mapping relationship from 2D posture to 3D posture can be established in different usage environments;

[0020] (2) The dynamic limb constraint Transformer network model established by the present invention can perform accurate 3D posture estimation on the data set, and can effectively improve the estimation accuracy and efficiency in actual use scenarios;

[0021] (3) The human body posture estimation method of the present invention is based on the dynamic limb constraint Transformer network model, and a human body posture estimation network from 2D to 3D is constructed, which has faster computing efficiency and can make full use of the temporal and spatial features in the image to obtain the 3D posture information of the human body. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 The figure is a flow chart of a method according to an embodiment of the present invention.

[0023] Figure 2 Schematic diagram of a dynamic limb constraint Transformer network model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The present invention is described in detail below in conjunction with embodiments and drawings.

[0025] like Figure 1 As shown, a 3D posture estimation method based on a dynamic limb constraint Transformer network model greatly enhances the learning and generalization capabilities of the network by means of the feature extraction capabilities of a temporal feature extractor and a spatial feature extractor, and accurately learns the mapping relationship from 2D posture to 3D posture, so that the 3D posture can be accurately estimated through the learned mapping relationship; specifically, the following steps are included:

[0026] Step 1: Build a dynamic limb constraint Transformer network model, using a self-supervised learning method. In order to accurately estimate the 3D posture, the dynamic limb constraint Transformer network model includes an image feature embedding module, a Transformer encoder module, and an MLP module;

[0027] The image feature embedding module of this embodiment includes a convolution layer and a linear connection layer. The convolution kernel size of the convolution layer is 17×17, and the number of convolution kernels is 768. The image feature embedding module uses multiple convolution kernels to segment the original input image and embed key point position features and limb movement features, thereby obtaining multiple image blocks as input for posture estimation.

[0028] Reference Figure 2 ,The Transformer encoder module includes 4 repeated temporal feature extractors and spatial feature extractors. Each feature extractor includes 4 convolutional layers with a convolution kernel of 1×1. Each convolutional layer is preceded by a corresponding LayerNorm layer. The temporal feature extractor and the spatial feature extractor work in turn to extract the temporal features and spatial features in the image respectively.

[0029] The MLP module consists of a linear layer and a LayerNorm layer;

[0030] Step 2: Select the training set and test set, and set the training parameters of the dynamic limb constraint Transformer network model;

[0031] In this embodiment, the training set and the test set contain 2D poses and corresponding 3D poses. The 2D poses have two coordinate channels, x and y, and the 3D poses have three coordinate channels, x, y, and z.

[0032] In this embodiment, the training / test split in the Human3.6M data set widely used in the field of posture estimation is selected as the training set and the test set, the network is trained through the training set, and the network is tested through the test set; before adopting the method based on the dynamic limb constraint Transformer network model to estimate 3D posture, it is necessary to pre-train the dynamic limb constraint Transformer network model, and then set the training parameters of the dynamic limb constraint Transformer network model; the learning rate is set to 0.001, the decay rate of the learning rate is set to 0.1, and the strategy of decaying the learning rate once every 20 epochs of training is adopted; the weight parameter initialization of the dynamic limb constraint Transformer network model adopts the random initialization method, the batch size is 1024, and the total number of training times is 200 times;

[0033] Step 3: According to the dynamic limb constraint Transformer network model and its training parameters, minimize the loss function, and use the AdamW optimization method to minimize the loss function, so as to complete the training of the dynamic limb constraint Transformer network model;

[0034] In order to make an accurate estimate of the 3D pose, this embodiment uses a combined loss function; in the i-th frame image, the predicted position of the j-th key point is p i,j , the actual real position is gt i,j , since different key points have different weights when predicting, the corresponding weighted prediction error on the input sequence is expressed as The error of predicting key points in two adjacent frames is expressed as The error in predicting limb lengths at symmetric locations is expressed as: The error of predicted limb length in two adjacent frames is expressed as: The final loss function Loss total =L w +λ t L t +λ s L s +λ a L a , where N represents the number of all key points, T represents the length of the input sequence, P represents the number of all limbs, F represents the limb length between two predicted key points, V represents the number of symmetrical limbs, r and l represent the right and left sides respectively, W,λ t ,λ s ,λ a is the weight coefficient; p i,j is the predicted position of the jth key point in the i-th frame image, gt i,j is the actual position of the jth key point in the i-th frame image, pi,j-1 is the predicted position of the j-1th key point in the i-th frame image, F i,j(r) is the predicted length of the jth limb on the right side of the i-th frame image, F i,j(l) is the predicted length of the jth limb on the left side of the i-th frame image, F i,k is the predicted length of the kth limb in the i-th frame image, F i-1,k is the predicted length of the kth limb in the i-1th frame image.

[0035] Step 4: Input the 2D posture sequence to be processed into the trained dynamic limb constraint Transformer network model to obtain the corresponding 3D posture.

[0036] This embodiment provides a 3D posture estimation method based on a dynamic limb constraint Transformer network model. Through dynamic limb constraints, the learning ability of the dynamic limb constraint Transformer network model is greatly enhanced, and a strong mapping relationship from 2D posture to 3D posture is established; through the introduction of an image feature embedding module, the temporal features and spatial features contained in the original image can be extracted and embedded, and the temporal features and spatial features can be used for subsequent 3D posture estimation; through the introduction of a Transformer encoder module, the attention between key points can be repeatedly calculated, thereby improving the global accuracy of 3D posture estimation; the network model based on the dynamic limb constraint Transformer is trained on the Human3.6M dataset to construct a stable mapping from 2D posture to 3D posture, and finally a 3D posture close to the real one is obtained, which proves that the present invention has good accuracy and versatility.

Claims

1. A 3D posture estimation method based on a dynamic limb constraint Transformer network model, characterized in that: The following steps are involved: Step 1: Build a dynamic limb constraint Transformer network model, which includes an image feature embedding module, a Transformer encoder module, and an MLP module; Step 2: Select the training set and test set, and set the training parameters of the dynamic limb constraint Transformer network model; Step 3: According to the dynamic limb constraint Transformer network model and its training parameters, the loss function is minimized to complete the training of the dynamic limb constraint Transformer network model; Step 4: Input the 2D posture sequence to be processed into the trained dynamic limb constraint Transformer network model to obtain the corresponding 3D posture.

2. The 3D posture estimation method according to claim 1, characterized in that: The image feature embedding module in step 1 includes a convolution layer and a linear connection layer, the convolution kernel size of the convolution layer is 17×17, and the number of convolution kernels is 768.

3. The 3D posture estimation method according to claim 1, characterized in that: The Transformer encoder module in the step 1 includes 4 temporal feature extractors and 4 spatial feature extractors, each feature extractor includes 4 convolutional layers with a convolution kernel of 1×1, and a corresponding LayerNorm layer is added in front of each convolutional layer. The temporal feature extractor and the spatial feature extractor work in turn to extract the temporal features and spatial features in the image respectively.

4. The 3D posture estimation method according to claim 1, characterized in that: The MLP module in step 1 includes a linear layer and a LayerNorm layer.

5. The 3D posture estimation method according to claim 1, characterized in that: In the step 2, the training set and the test set adopt the training / test split in the Human3.6M data set, the network is trained by the training set, and the network is tested by the test set.

6. The 3D posture estimation method according to claim 1, characterized in that: In the step 2, the weight initialization of the dynamic limb constraint Transformer network model is performed by a random initialization method.

7. The 3D posture estimation method according to claim 1, characterized in that: The loss function in step 3 is the combined loss function Loss total =L w +λ t L t +λ s L s +λ a L a ,in, Represents the error between the predicted key point and the true key point; Represents the error of the predicted key point in two adjacent frames; represents the error in predicting limb length in symmetric positions; represents the error of the predicted limb length in two adjacent frames; where N represents the number of all key points, T represents the length of the input sequence, P represents the number of all limbs, F represents the limb length between two predicted key points, V represents the number of symmetrical limbs, r and l represent the right and left sides respectively, W,λ t ,λ s ,λ a is the weight coefficient; p i,j is the predicted position of the jth key point in the i-th frame image, gt i,j is the actual position of the jth key point in the i-th frame image, p i,j-1 is the predicted position of the j-1th key point in the i-th frame image, F i,j(r) is the predicted length of the jth limb on the right side of the i-th frame image, F i,j(l) is the predicted length of the jth limb on the left side of the i-th frame image, F i,k is the predicted length of the kth limb in the i-th frame image, F i-1,k is the predicted length of the kth limb in the i-1th frame image.

8. The 3D posture estimation method according to claim 1, characterized in that: In step 3, the AdamW optimization method is used to minimize the loss function.

Images