A Transformer-based method for human pose estimation
By combining a dual-branch feature extraction network of CNN and Transformer with a multi-stage regression method, the accuracy and speed issues in human pose estimation are solved, achieving more efficient feature extraction and accurate detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2023-07-14
- Publication Date
- 2026-05-26
Smart Images
Figure CN117315769B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of deep learning and computer vision, specifically to a human pose estimation method based on Transformer. Background Technology
[0002] Human pose estimation is a crucial problem in computer vision, essential for machines to understand human actions and behaviors. It involves identifying and classifying the limb parts and major joints of pedestrians in images or videos. This task is a method of capturing a set of coordinates by defining human joints such as the wrist, shoulder, knee, and ankle, thereby describing the human pose of one or more people in images and videos. Human pose estimation, essentially human skeletal key detection, is fundamental to much computer vision research and has significant real-world applications, such as action recognition, human-computer interaction, intelligent security, augmented reality, and autonomous driving.
[0003] The mainstream methods for human pose estimation are mainly divided into top-down and bottom-up methods. Top-down methods first detect the overall presence of the human body, and then specifically detect the position of each keypoint within the detected human bounding box. The main challenge is that they are highly sensitive to human detection, and the computational load increases proportionally with the number of people being detected. Bottom-up methods first detect keypoints at the joints of the human body, and then use algorithms such as graph structures and conditional random fields to assemble them into the human pose. The main challenge is that different keypoints are grouped into different human figures, and they are greatly affected by occlusion.
[0004] In recent years, numerous human pose estimation methods have been proposed by academia and industry. However, these methods still suffer from low accuracy and slow computation speed in practical applications, primarily due to the following issues:
[0005] (1) There are shortcomings in using only CNN or Transformer. CNN-based work can effectively mine local features, but its ability to acquire global features is relatively low, and very deep networks are required to acquire global features. The built-in attention layer of Transformer can effectively capture long-term relationships, but Transformer mainly focuses on global features and ignores local features. The detection results of Transformer are not accurate enough, mainly because it ignores local features that can be used for fine-grained detection.
[0006] (2) Most human pose estimation methods are based on heatmaps, but these methods may face the following problems: First, post-processing (e.g., "maximum value" operation) is required. The post-processing process may not be differentiable, so the framework cannot be trained end-to-end. Second, the resolution of the heatmap predicted by the neural network is usually lower than the resolution of the input image. The reduced resolution will lead to quantization error and limit the accuracy of key point localization. This quantization error can be solved by moving the output coordinates according to the value of the pixel near the peak, but it makes the framework more complex and introduces more hyperparameters. Third, the ground truth of the heatmap needs to be designed manually and heuristically adjusted, which may lead to a lot of noise and ambiguity in the heatmap ground truth.
[0007] In summary, current human pose estimation suffers from insufficient accuracy and low computational speed, limiting its application scenarios. This invention aims to address these issues by proposing solutions to improve the practicality of human pose estimation and facilitate its wider application. Summary of the Invention
[0008] The technical problems to be solved by this invention in human pose estimation are: (1) the features obtained by using only CNN or Transformer are insufficient; and (2) heatmap-based methods result in large errors. To address these technical problems, this invention provides a Transformer-based human pose estimation method.
[0009] The specific technical solution adopted in this invention is as follows:
[0010] A Transformer-based human pose estimation method includes the following steps:
[0011] S1. Obtain the image dataset for human pose estimation, and perform uniform data preprocessing on all images in the graph dataset. Each image is labeled with the corresponding center point, human key points and heat map.
[0012] S2. Construct a human pose estimation network, which includes a convolutional network, a feature extraction module, and a pose regression module. The input image is processed by the convolutional network to extract initial features, which are then used as the input to the feature extraction module. The feature extraction module includes four stages, each implemented using a dual-branch feature extraction layer. The four stages extract features at different scales, which are then fused using a multi-scale feature fusion model to serve as the final output of the feature extraction module. Each dual-branch feature extraction layer includes a global attention branch and a local attention branch, a feedforward network, and a batch processing normalization layer. The module input features first pass through two attention branches, and then the outputs of the two branches are added to obtain the fused features of the global and local features. The fused features are then residually connected to the module input features to obtain intermediate features. The intermediate features are then passed through the batch processing normalization layer and the feedforward network, and then residually connected to the intermediate features to serve as the output of the current dual-branch feature extraction layer.
[0013] The pose regression module uses the feature map output by the feature extraction module as a shared feature input to three different branches. The first branch predicts the human body center point, the second branch predicts human body key points, and the third branch generates a heatmap for training assistance. The second branch predicts human body key points in two stages. In the first stage, it first predicts the offset of each limb trunk point from the human body center point, and then predicts the offset of each key point from the corresponding limb trunk point. All human body key points are pre-divided into multiple groups through clustering, and all human body key points in each group correspond to one limb trunk point.
[0014] S3. Using the preprocessed image dataset from S1, train the human pose estimation network constructed in S2; the total loss function used in training is the weighted sum of the losses of the center point, human key points and heatmap respectively;
[0015] S4. For the image to be estimated, input it into the human pose estimation network trained in S3 to obtain the corresponding human key points and realize human pose estimation.
[0016] Preferably, in the multi-scale feature fusion model, the features output by the four bi-branch feature extraction layers are upsampled for size alignment, then spliced along the channels to expand the channels, and then convolved to restore the original number of channels, thereby completing the feature fusion at different scales and obtaining the final output of the feature extraction module.
[0017] Preferably, the global attention branch adopts a Transformer architecture. The module input features are processed by PatchEmbedding to form a one-dimensional sequence, and then global information is generated as the branch output through a multi-head attention mechanism.
[0018] Preferably, the local attention branch adopts a CNN architecture. The module input features are processed by PatchEmbedding to form a one-dimensional sequence. After the one-dimensional sequence is converted into an image, it is processed by two intermediate modules in sequence. The output feature map is converted back into a sequence to form local information as the branch output. Each intermediate module consists of a cascaded 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function layer.
[0019] Preferably, in the pose regression module, the inputs to the three branches are the feature maps that are finally output by the feature extraction module after being fused by the multi-scale feature fusion model;
[0020] The first branch is the center perception branch, which is composed of cascaded 3×3 convolutions and 1×1 convolutions, and its output is the center point of the human body.
[0021] The second branch is the key point perception branch, which is formed by two different 1×1 convolutions connected in parallel from the back end of a 3×3 convolution. The output of one 1×1 convolution is the offset of each limb trunk point from the human body center point output by the first branch, and the output of the other 1×1 convolution is the offset of each human body key point from the corresponding limb trunk point.
[0022] The third branch is the heatmap-assisted training branch, which is composed of cascaded 3×3 deconvolutions and 3×3 convolutions, and the output is the heatmap corresponding to the input image.
[0023] As a preferred method, when all human body key points are pre-divided into multiple groups through clustering, mutual information is used as the distance value for measuring similarity during clustering, thereby dividing the human body key points into multiple groups.
[0024] As a preferred method, all human body key points were divided into 5 groups during clustering.
[0025] As a preferred approach, among all human body key points, the key points in the left upper limb are grouped into one group, the key points in the right upper limb are grouped into one group, the key points in the two lower limbs are grouped into one group, the key points in the head are grouped into one group, and the remaining key points are grouped into one group; the second branch needs to predict the limb trunk point corresponding to the key points in each group for each group of key points.
[0026] Preferably, the total loss function during the training of the human pose estimation network is in the following form:
[0027] Loss total =Loss ct +ηLoss kp +λLoss aux
[0028] In the formula: η and λ are both weight hyperparameters greater than 0; Loss ct FocalLoss, where FocalLoss is the sum of the predicted and actual values at the center point.kp The L1 norm loss between the predicted and actual values of human body keypoints, Loss aux This represents the mean squared error loss between the predicted and actual values of the heatmap.
[0029] Preferably, in step S4, after the image to be estimated is input into the human pose estimation network trained in step S3, the feature map extracted by the convolutional network and feature extraction module is first input into the pose regression module. The first branch predicts the coordinates of the human body center point, and the second branch predicts the offsets of each limb trunk point from the human body center point and the offsets of each key point from the corresponding limb trunk point. The two offsets are then superimposed on the human body center point coordinates predicted by the first branch to obtain the coordinates of each key point.
[0030] Compared with the prior art, the beneficial effects of the present invention are as follows:
[0031] (1) The dual-branch feature extraction network based on the combination of CNN and Transformer overcomes the shortcomings of CNN ignoring global features and Transformer ignoring local features. It can take into account both global and local features at the same time, and improves the detection effect of some inconspicuous key points and small human images.
[0032] (2) The multi-stage regression method of “human body center-limb trunk-key point” adds the “limb trunk” midpoint compared with the center offset method, thereby improving the accuracy of key point regression far from the human body center point and reducing the error rate of remote key point detection. Attached Figure Description
[0033] Figure 1 This is a schematic diagram of the overall algorithm framework;
[0034] Figure 2 This is a schematic diagram of the dual-branch feature extraction module.
[0035] Figure 3 A schematic diagram of the specific structure of the dual-branch feature extraction module;
[0036] Figure 4 This is a schematic diagram of the multi-stage regression module of "human body center - limb trunk - key points". Detailed Implementation
[0037] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.
[0038] In a preferred embodiment of the present invention, a human pose estimation method based on Transformer is provided. The method first extracts global and local features using a dual-branch network combining CNN and Transformer. Then, a multi-stage regression method based on "human center-limb trunk-key points" is used to improve detection accuracy. The method includes the following steps:
[0039] S1. Obtain the image dataset for human pose estimation, and perform uniform data preprocessing on all images in the graph dataset. Each image is labeled with the corresponding center point, human key points and heat map.
[0040] S2. Construct a human pose estimation network.
[0041] like Figure 2 The diagram illustrates the model structure of a human pose estimation network. This network comprises a convolutional network, a feature extraction module, and a pose regression module. The input image, after initial feature extraction via the convolutional network, serves as the input to the feature extraction module. The feature extraction module consists of four stages, each implemented using a dual-branch feature extraction layer. Features extracted at different scales from each of the four stages are then fused using a multi-scale feature fusion model, resulting in the final output of the feature extraction module. Figure 2 As shown, each dual-branch feature extraction layer contains a global attention branch and a local attention branch, a feedforward network, and a batch normalization layer. The module input features first pass through the two attention branches respectively, and then the outputs of the two branches are added to obtain the fused features of the global and local features. The fused features are then residually connected with the module input features to obtain intermediate features. The intermediate features are then passed through the batch normalization layer and the feedforward network, and then residually connected with the intermediate features to serve as the output of the current dual-branch feature extraction layer.
[0042] In the embodiments of the present invention, in the above-mentioned multi-scale feature fusion model, the features output by the four bi-branch feature extraction layers are upsampled for size alignment, and then the channels are spliced along the channels to expand the channels and then convolution is used to restore the original number of channels, thereby completing the feature fusion at different scales and obtaining the final output of the feature extraction module.
[0043] In embodiments of the present invention, such as Figure 3 As shown, the global attention branch adopts a Transformer architecture. The module input features are processed by Patch Embedding to form a one-dimensional sequence, and then the global information is generated as the branch output through a multi-head attention mechanism. Patch Embedding (window partitioning) is an existing technology, which transforms the sequence into a 1D sequence through 1*1 convolution and flattening operations. In the multi-head attention mechanism, Patch Embedding divides the sequence into query (Q), key (K), and value (V) vectors. Attention is calculated through Q and K and superimposed on V to obtain the attention output. Due to the large amount of computation in this invention, a linear multi-head attention mechanism can be introduced in the embodiment, using a first-order approximation of Taylor expansion to replace the softmax function in the traditional multi-head attention mechanism. The implementation of the linear multi-head attention mechanism can be found in: Wang L, Fang S, Zhang C, et al. Efficient HybridTransformer: Learning Global-local Context for Urban Context Segmentation.[J].2021. Of course, if there are sufficient computing resources, the conventional multi-head attention mechanism can also be used in this invention.
[0044] In embodiments of the present invention, such as Figure 3 As shown, the aforementioned local attention branch adopts a CNN architecture. The module input features are processed by Patch Embedding to form a one-dimensional sequence. After the one-dimensional sequence is converted into an image, it is processed by two intermediate modules in sequence. The output feature map is then converted back into a sequence to form local information as the branch output. Each intermediate module consists of a cascaded 3×3 convolutional layer, a batch normalization layer, and a ReLU activation function layer.
[0045] The pose regression module uses the feature map output by the feature extraction module as a shared feature input to three different branches: the first branch predicts the human body center point, the second branch predicts the human body key points, and the third branch generates a heatmap for auxiliary training.
[0046] It should be noted that, for example Figure 4As shown, the "human center-limb trunk-key point" multi-stage regression method: When predicting human key points in the second branch, the concept of human limb trunk points is introduced as an intermediate connection point based on the center-offset method, forming a three-layer connection relationship of human center point-limb trunk point-key point. The coordinate prediction of each key point is carried out in two stages. The first stage predicts the offset of each limb trunk point from the human center point, and then predicts the offset of each key point from the corresponding limb trunk point. Here, the limb trunk point is an intermediate connection point between the human center point and the key point. When actually predicting the coordinates of each key point, the human center point is first offset and moved to the limb trunk point. This offset is the object to be predicted in the first step; then the limb trunk point is further moved to the key point. This offset is the object to be predicted in the second step. This staged regression prediction can improve the accuracy of key point regression far from the human center point and reduce the error rate of remote key point detection.
[0047] Furthermore, all human keypoints are pre-divided into multiple groups through clustering, with all human keypoints in each group corresponding to a limb trunk point. In an embodiment of the present invention, when the aforementioned human keypoints are pre-divided into multiple groups through clustering, mutual information is used as the distance value for measuring similarity during clustering, thereby dividing the human keypoints into multiple groups. Further, all human keypoints are preferably divided into 5 groups during clustering. Among all human keypoints, the keypoints in the left upper limb are divided into one group, the keypoints in the right upper limb are divided into one group, the keypoints in the two lower limbs are divided into one group, the keypoints in the head are divided into one group, and the remaining keypoints are divided into one group; the second branch needs to predict the limb trunk point corresponding to the keypoints within each group of keypoints.
[0048] In embodiments of the present invention, such as Figure 4 As shown, in the pose regression module above, the inputs to the three branches are the feature maps that are finally fused by the feature extraction module through a multi-scale feature fusion model. The specific structures and outputs of each branch are different, as follows:
[0049] The first branch is the center perception branch, which is composed of cascaded 3×3 convolutions and 1×1 convolutions, and its output is the center point of the human body.
[0050] The second branch is the key point perception branch, which is formed by two different 1×1 convolutions connected in parallel from the back end of a 3×3 convolution. The output of one 1×1 convolution is the offset of each limb trunk point from the human body center point output by the first branch, and the output of the other 1×1 convolution is the offset of each human body key point from the corresponding limb trunk point.
[0051] The third branch is the heatmap-assisted training branch, which is composed of cascaded 3×3 deconvolutions and 3×3 convolutions, and the output is the heatmap corresponding to the input image.
[0052] S3. Using the preprocessed image dataset from S1, train the human pose estimation network constructed in S2; the total loss function used in training is the weighted sum of the losses of the center point, human key points and heatmap.
[0053] It should be noted that image datasets can be divided into training and testing sets according to conventional model training practices. Training is performed on the training set and testing is performed on the testing set. This is existing technology and will not be elaborated further.
[0054] In an embodiment of the present invention, the total loss function for training the human pose estimation network is as follows:
[0055] Loss total =Loss ct +ηLoss kp +λLoss aux
[0056] In the formula: η and λ are both weight hyperparameters greater than 0; Loss ct FocalLoss, where FocalLoss is the sum of the predicted and actual values at the center point. kp The L1 norm loss between the predicted and actual values of human body keypoints, Loss aux This represents the mean squared error loss between the predicted and actual values of the heatmap.
[0057] The specific three loss functions are as follows:
[0058] (1) Center point regression loss: Center point regression loss is used to measure the error between the predicted center point location and the center point Ground Truth in the center regression branch. In order to overcome the problem of imbalance between positive and negative samples, this invention uses Focal Loss for center point confidence plot regression.
[0059]
[0060] Where N is the number of human instances in the image. and P c This represents the predicted pixel-wise confidence level and the corresponding Ground Truth. α and β are hyperparameters, which are set to 2 and 4 in this invention. In the center point regression loss, only the center pixel with a peak value of 1.0 is a positive sample, and the rest are negative samples.
[0061] (2) Keypoint prediction loss: The keypoint prediction loss is represented by the offset, and the loss is calculated based on the L1 norm, as defined below:
[0062]
[0063] Where i represents a human body instance, j represents a key point, N represents the number of human body instances in the image, and K represents the number of human body key points.
[0064] (3) Heatmap auxiliary loss function: The regression network is trained by calculating the mean squared error loss. The heatmap auxiliary loss function is shown below.
[0065]
[0066] Among them, H and These represent the actual and predicted values of the heatmap, respectively.
[0067] S4. For the image to be estimated, input it into the human pose estimation network trained in S3 to obtain the corresponding human key points and realize human pose estimation.
[0068] In an embodiment of the present invention, in S4 above, after the image to be estimated is input into the human pose estimation network trained in S3, the feature map extracted by the convolutional network and feature extraction module is first input into the pose regression module. The first branch predicts the coordinates of the human body center point, and the second branch predicts the offset of each limb trunk point from the human body center point and the offset of each key point from the corresponding limb trunk point. The two offsets are then superimposed on the human body center point coordinates predicted by the first branch to obtain the coordinates of each key point.
[0069] To verify the effectiveness of the method, this embodiment conducted multiple sets of comparative and ablation experiments on the public dataset COCO, comparing the performance of the method of this invention with some classic or advanced methods and analyzing the impact of various technical details of this invention on the overall method effect. For all datasets, this invention used the OKS metric to evaluate the experimental results.
[0070] Regarding the experimental environment: All experiments in this embodiment are implemented based on the Python 3.8 and PyTorch 1.7.0 framework. In addition, this invention uses 8 Tesla V100 cards for training the algorithm model and 1 Tesla V100 card for model experimental verification. The operating system is Ubuntu 20.04.
[0071] Regarding model training: In this embodiment, the dual-branch feature extraction network was trained in a distributed manner using eight Tesla V100 GPUs on the COCO train2017 dataset. The input size of the dual-branch feature extraction network was set to 256×192. To ensure fairness in the comparative experiments, all models were optimized using AdamW with a base learning rate of 4x10⁻¹⁰. -3 Weight decay is set to 10. -4 Unless otherwise specified, all experiments used a base learning rate of 4x10. -3 The cosine annealing learning rate.
[0072] Regarding model testing: In this embodiment, a dual-branch feature extraction network model trained on the COCO train2017 dataset is used for testing on the COCO val2017 dataset. To verify the lightweight and efficient nature of the model, the OKS metric (specifically including AP, AP1, AP2, AP3, AP4, AP5, AP6, AP7, AP8, AP9, AP1 ... 50 AP 75 AP m AP L In addition to evaluating the model's test accuracy using the PCKh index, the model's parameter quantity index (Params) and computational quantity index (FLOPs) are also calculated.
[0073] Experimental results:
[0074] (1) The effect of using CNN and Transformer together
[0075] The dual-branch feature extraction layer of this invention employs a dual-branch network combining CNN (local branch) and Transformer (global branch). To compare the impact of using CNN and Transformer alone versus in combination on human pose estimation, an ablation experiment was designed and conducted. The detection accuracy results of different experiments are shown in Table 1.
[0076] Table 1. Ablation Experiment Results Using CNN and Transformer in Combination
[0077] Configuration AP <![CDATA[AP 50 ]]> <![CDATA[AP 75 ]]> <![CDATA[AP M ]]> <![CDATA[AP L ]]> CNN only 63.2 86.8 69.6 58.9 70.4 Transformer only 71.1 89.4 79.6 67.1 76.3 This invention model 73.4 91.6 81.1 70.1 79.3
[0078] (2) Results of ablation experiments using a multi-stage regression module
[0079] To verify the role of the multi-stage regression module, the following experiments were designed: ablation experiments were conducted using the traditional center-offset single-stage regression method for keypoints (SPM model) and the method using the proposed "human center-limb trunk-keypoint" multi-stage regression module. The detection accuracy results of the different experiments are shown in Table 2. The results show that the present invention improves the overall detection accuracy.
[0080] Table 2. Results of ablation experiments using the multi-stage regression module.
[0081] Configuration AP <![CDATA[AP 50 ]]> <![CDATA[AP 75 ]]> <![CDATA[AP M ]]> <![CDATA[AP L ]]> SPM 69.9 86.7 75.1 65.5 74.4 This invention model 73.4 91.6 81.1 70.1 79.3
[0082] (3) Overall network performance verification
[0083] The model of this invention was compared with some existing mainstream pose estimation models on the COCO dataset, and the results are shown in Table 3.
[0084] Table 3 shows the comparative experimental results of this model on the COCO dataset.
[0085]
[0086] Experimental analysis shows that the model of this invention achieves the best performance in the one-stage pose estimation model, and also achieves excellent results in the Transformer-based pose estimation scheme. Furthermore, because the model of this invention is based on regression for prediction rather than heatmaps, it can be better integrated with downstream tasks to achieve end-to-end inference learning.
[0087] Compared with traditional detection methods, this invention can simultaneously take into account both global and local features, making up for the shortcomings of CNN ignoring global features and Transformer ignoring local features. At the same time, by adopting a multi-stage regression method, the intermediate points of the "limbs and trunk" are added, which reduces the regression difficulty of key points far from the center of the human body, improves the accuracy of key point regression, and further enhances the accuracy of human pose estimation.
[0088] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.
Claims
1. A human pose estimation method based on Transformer, characterized in that, Includes the following steps: S1. Obtain the image dataset for human pose estimation, and perform uniform data preprocessing on all images in the graph dataset. Each image is labeled with the corresponding center point, human key points and heat map. S2. Construct a human pose estimation network, which includes a convolutional network, a feature extraction module, and a pose regression module. The input image is processed by the convolutional network to extract initial features, which are then used as the input to the feature extraction module. The feature extraction module includes four stages, each implemented using a dual-branch feature extraction layer. The four stages extract features at different scales, which are then fused using a multi-scale feature fusion model to serve as the final output of the feature extraction module. Each dual-branch feature extraction layer includes a global attention branch and a local attention branch, a feedforward network, and a batch processing normalization layer. The module input features first pass through two attention branches, and then the outputs of the two branches are added to obtain the fused features of the global and local features. The fused features are then residually connected to the module input features to obtain intermediate features. The intermediate features are then passed through the batch processing normalization layer and the feedforward network, and then residually connected to the intermediate features to serve as the output of the current dual-branch feature extraction layer. The pose regression module uses the feature map output by the feature extraction module as a shared feature input to three different branches. The first branch predicts the human body center point, the second branch predicts the human body key points, and the third branch generates a heat map for training assistance. When the second branch predicts the human body key points, it performs prediction in two stages. In the first stage, it first predicts the offset of each limb trunk point from the human body center point, and then predicts the offset of each key point from the corresponding limb trunk point. All human key points are pre-divided into multiple groups through clustering, and all human key points in each group correspond to a limb trunk point; S3. Using the preprocessed image dataset from S1, train the human pose estimation network constructed in S2. The total loss function used in training is the weighted sum of the losses of the center point, human key points and heatmap; S4. For the image to be estimated, input it into the human pose estimation network trained in S3 to obtain the corresponding human key points and realize human pose estimation. In the multi-scale feature fusion model, the features output by the four bi-branch feature extraction layers are upsampled for size alignment, then spliced along the channels to expand the channels, and then convolution is used to restore the original number of channels, thereby completing the feature fusion at different scales and obtaining the final output of the feature extraction module.
2. The human pose estimation method based on Transformer according to claim 1, characterized in that, The global attention branch adopts the Transformer architecture. The module input features are processed by Patch Embedding to form a one-dimensional sequence, and then the global information is generated as the branch output through a multi-head attention mechanism.
3. The human pose estimation method based on Transformer according to claim 1, characterized in that, The local attention branch adopts a CNN architecture. The module input features are processed by Patch Embedding to form a one-dimensional sequence. After the one-dimensional sequence is converted into an image, it is processed by two intermediate modules in sequence. The output feature map is converted back into a sequence to form local information as the branch output. Each intermediate module consists of a cascaded 3×3 convolutional layer, a batch normalization layer and a ReLU activation function layer.
4. The human pose estimation method based on Transformer according to claim 1, characterized in that, In the pose regression module, the inputs to the three branches are the feature maps that are finally output by the feature extraction module after being fused by the multi-scale feature fusion model; The first branch is the center perception branch, which is composed of cascaded 3×3 convolutions and 1×1 convolutions, and its output is the center point of the human body. The second branch is the key point perception branch, which is formed by two different 1×1 convolutions connected in parallel from the back end of a 3×3 convolution. The output of one 1×1 convolution is the offset of each limb trunk point from the human body center point output by the first branch, and the output of the other 1×1 convolution is the offset of each human body key point from the corresponding limb trunk point. The third branch is the heatmap-assisted training branch, which is composed of cascaded 3×3 deconvolutions and 3×3 convolutions, and the output is the heatmap corresponding to the input image.
5. The human pose estimation method based on Transformer according to claim 1, characterized in that, When all human body key points are pre-divided into multiple groups through clustering, mutual information is used as the distance value for measuring similarity during clustering, thereby dividing the human body key points into multiple groups.
6. The human pose estimation method based on Transformer according to claim 5, characterized in that, All human body key points were divided into 5 groups during clustering.
7. The human pose estimation method based on Transformer according to claim 6, characterized in that, Of all the key points in the human body, the key points in the left upper limb are grouped into one group, the key points in the right upper limb are grouped into one group, the key points in the two lower limbs are grouped into one group, the key points in the head are grouped into one group, and the remaining key points are grouped into one group; the second branch needs to predict the limb trunk point corresponding to the key points in each group.
8. The human pose estimation method based on Transformer according to claim 1, characterized in that, The total loss function during the training of the human pose estimation network is as follows: In the formula: and All are weight hyperparameters greater than 0; FocalLoss is the sum of the predicted and actual values at the center point. The L1 norm loss between the predicted and actual values of key points on the human body. This represents the mean squared error loss between the predicted and actual values of the heatmap.
9. The human pose estimation method based on Transformer according to claim 1, characterized in that, In step S4, after the image to be estimated is input into the human pose estimation network trained in step S3, the feature map extracted by the convolutional network and feature extraction module is first input into the pose regression module. The first branch predicts the coordinates of the human body center point, and the second branch predicts the offset of each limb trunk point from the human body center point and the offset of each key point from the corresponding limb trunk point. The two offsets are then superimposed on the human body center point coordinates predicted by the first branch to obtain the coordinates of each key point.