Human pose estimation method based on transformer spatio-temporal feature enhancement

Abstract

The application discloses a human posture estimation method based on a Transformer space-time feature enhancement type, a network CSIT designed by the method adopts a CNN+Transformer architecture, considering that visual information is composed of texture features and space features, therefore, the application generates Channel Embeddings and Spatial Embeddings by unfolding features extracted by the CNN from two dimensions of channels and space, meanwhile, considering that the network also needs to learn the dependency between key nodes, then, Keypoints Embeddings are added, three sequences are spliced together as the input of the Transformer, and a one-dimensional vector representation method with more advanced performance and more in line with the characteristics of the Transformer is simultaneously adopted at the end of the network to predict the key points, so that the prediction performance of the network is further improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision and human pose estimation, and particularly relates to a human pose estimation method based on a Transformer spatiotemporal feature enhancement type. BACKGROUND

[0002] Transformers have achieved great success in the field of natural language processing, and soon researchers have introduced them into the field of computer vision and shown great potential.

[0003] In the application of image classification, the document [Dosovitskiy, A., et al. "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale." International Conference on Learning Representations 2021] proposes a ViT model and first applies it to the image classification task, which has a great impact on the field of computer vision. The specific implementation process is to expand the picture according to the patch, encode the position, and then input it into the Transformer network together with the category vector for training. In addition, the document [Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and egou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012.12877, 2020] proposes DeiT, that is, to use a distilled token to distill the Transformer network.

[0004] In the application of object detection, the document [Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213-229. Springer, 2020] proposes a Transformer-based object detection network, which regards object detection as a set prediction problem. The document [Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. iv preprint arXiv:2010.04159, 2020] proposes that the attention module only focuses on a small number of key sampling points around the target frame, thereby obtaining better performance than DETR. The document [Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. Up-detr: Unsupervised pre-training for object detection with transformers. arXiv preprint arXiv:2011.09094, 2020] adopts an unsupervised training method, randomly crops a patch from the original image, obtains the coordinates, length and width of the patch corresponding to the frame, and then inputs the patch as the query into the decoder of the DETR to realize unsupervised training.

[0005] In the application of human pose estimation, the literature [Ce Zheng, Sijie Zhu, Matias Mendieta, Taojiannan Yang, Chen Chen, and Zhengming Ding. 3d human pose estimation with spatial and temporal transformers. arXiv preprint arXiv:2103.10455, 2021] introduced the Transformer network to predict the three-dimensional pose through the two-dimensional human pose sequence in the video frame. The literature [Yang S, Quan Z, Nie M, et al. TransPose: Keypoint Localization via Transformer [J]. 2020] proposed TransPose, which takes CNN as the encoder and predicts the keypoint heat map through the Transformer. The literature [Mao W, Ge Y, Shen C, et al. TFPose: Direct Human Pose Estimation with Transformers [J]. 2021] proposed TFPose, which is different from the previous work, and directly regresses the keypoint. The literature [Li Y, Zhang S, Wang Z, et al. TokenPose: Learning Keypoint Tokens for Human Pose Estimation [J]. 2021] proposed TokenPose, which, in addition to expanding the feature map according to the patch, also added Keypoint Token, which was sent into the Transformer together for learning. The literature [Stoffl L, Vidal M, Mathis A. End-to-End Trainable Multi-Instance Pose Estimation with Transformers [J]. 2021] proposed POET, which takes the multi-instance pose estimation as a direct set prediction problem and proposes an end-to-end trainable multi-instance model. The literature [Zhao S, Liu K, Huang Y, et al. DPIT: Dual-Pipeline Integrated Transformer for Human Pose Estimation [J]. 2022] proposed DPIT, which focused on the fact that in addition to the features of the human body, there are also some features associated between the human body and the scene, and then integrated the bottom-up and top-down methods through the parallel connection of the double pipelines.

[0006] In the above-mentioned method, CNN+Transformer is a better solution, and in this method, how to better send the features extracted by CNN into the Transformer is the key, because in the visual features of the human body, in addition to the texture features, there are also spatial features, and the information contained is very rich, for example, CNN extracts the texture features of the mouth and nose, but actually the place with the nose is probably with the mouth, and secondly, the distance between the nose and the mouth is very regular, and for these two points of spatial information, CNN cannot learn. SUMMARY

[0007] Human key point detection is different from general detection tasks, and the network needs to learn visual information and anatomical constraint information, since CNN has strong feature extraction ability for image texture features, and Transformer can well learn the correlation between key points, so many CNN+Transformer type human pose estimation networks have emerged, but these networks are too simple for the features extracted by CNN, and the spatial features are easily ignored, and for the complex detection task of pose estimation, it is essential to fully learn the features. In view of the above, the present application provides a human pose estimation method based on Transformer spatio-temporal feature enhancement, and the network model CSIT has excellent performance, which can extract more sufficient information from the image.

[0008] A human pose estimation method based on Transformer spatio-temporal feature enhancement, comprising the following steps:

[0009] (1) Obtain a large number of pictures containing people, and label the key point positions of the people in the pictures;

[0010] (2) Construct a CSIT (Channel-Spatial Integrated Transformer) network model framework, which includes a backbone network, a channel feature encoder, a spatial feature encoder, a Transformer module and a one-dimensional vector regression module, wherein:

[0011] The backbone network is used for feature extraction of the input picture;

[0012] The channel feature encoder is used to expand the extracted feature map from the channel dimension and compress the sequence length, and output Channel Embeddings;

[0013] The spatial feature encoder is used to expand the extracted feature map from the spatial dimension and compress the sequence length, and output Spatial Embeddings;

[0014] Channel Embeddings and Spatial Embeddings after position encoding are concatenated along the channel dimension with learnable Keypoints Embeddings, which are randomly initialized and have a channel number of M, and input into the Transformer module.

[0015] The Transformer module is configured to encode the input, thereby outputting a feature vector of M channels.

[0016] The one-dimensional vector regression module is configured to decode the feature vector of the M channels, thereby outputting a predicted result of the position coordinates of each joint.

[0017] (3) The model framework is trained using the pictures and their annotation information obtained in step (1);

[0018] (4) A picture to be estimated containing a person is input into the trained model, and the position coordinates of each joint of the person in the picture can be predicted.

[0019] Further, the backbone network adopts a multi-layer convolutional neural network based on the HRNetW48 structure. After the input picture is subjected to feature extraction by the multi-layer convolutional neural network, a feature map of R x H2 x W2 size is output, where R is the number of channels, H2 = H / 4, W2 = W / 4, and W and H are the width and height of the input picture, respectively.

[0020] Further, the channel feature encoder flattens the feature map of R x H2 x W2 size into R x L c size and sends it to a linear layer for processing and generating Channel Embeddings, L c = H2 x W2; the linear layer is configured to further reduce the length L c of the feature vector, so as to reduce the memory occupation while preserving the fine-grained feature information.

[0021] Further, the spatial feature encoder first splits the feature map of R x H2 x W2 size, i.e., splits the feature map of each channel into C h x P w size patches, where P h = H2 / R h , P w = W2 / R w , C = R h x R w and C = R; then each patch is flattened and concatenated along the channel dimension to obtain C x L sa feature matrix of size L s = P h × P w × R, and finally the feature matrix is sent into a linear layer for processing to generate Spatial Embeddings, which is used to further reduce the length L of the feature vector s to reduce the memory footprint while preserving the fine-grained feature information.

[0022] Further, the Transformer module is composed of a plurality of cascaded encoding structures, each of which is composed of a regularization layer L1, a multi-head self-attention mechanism layer, a regularization layer L2, and a multi-layer perceptron MLP connected in sequence, wherein the input of the regularization layer L2 is the superposition of the output of the multi-head self-attention mechanism layer and the input of the regularization layer L1, and the output of the multi-layer perceptron MLP is superimposed with the input of the regularization layer L2 to serve as the output of the encoding structure.

[0023] Further, the multi-head self-attention mechanism layer is composed of a plurality of Self-Attention, each of which multiplies the input of the multi-head self-attention mechanism layer with linear transformation matrices W Q , W K , and W V to obtain query vector matrix Q, key vector matrix K, and value vector matrix V, respectively, and then calculates the output result of Self-Attention through the following formula, and finally concatenates the output results of the plurality of Self-Attention and passes them through a linear layer to obtain the output of the multi-head self-attention mechanism layer.

[0024]

[0025] wherein: Attention(Q, K, V) is the output result of Self-Attention, s is the column number of the query vector matrix Q, T denotes transposition.

[0026] Further, the one-dimensional vector regression module respectively passes the feature vectors of the M channels through a horizontal axis coordinate classifier and a vertical axis coordinate classifier, the horizontal axis coordinate classifier reduces the length of each feature vector to kW to obtain vectors X1, X2, …, X M , and then extracts the predicted point horizontal coordinate x from the mapping The vertical axis coordinate classifier reduces the length of each feature vector to kH to obtain vectors Y1, Y2, …, Y M , and then extracts the predicted point vertical coordinate y from the mapping Finally, the predicted coordinates of each joint are output W and H are the width and height of the input image, respectively, i is a natural number and 1≤i≤M, and k is a scaling factor greater than 1.

[0027] Furthermore, the predicted coordinates The expression is as follows:

[0028]

[0029]

[0030] Where: σ is the set standard deviation. Let X be a vector i The value of the p-th element in the middle. For vector Y i Let p be a natural number and 1 ≤ p ≤ kW, and q be a natural number and 1 ≤ q ≤ kH.

[0031] Furthermore, the specific implementation of step (3) is as follows:

[0032] 3.1 Initialize model parameters, including the bias vector and weight matrix of each layer, learning rate, and optimizer;

[0033] 3.2 Input the image containing people into the model, and the model outputs the corresponding prediction result, i.e., the coordinates of the key points, through forward propagation. Calculate the loss function between the prediction result and the annotation information.

[0034] 3.3 Based on the loss function, the optimizer iteratively updates the model parameters using gradient descent until the loss function converges, and the training is complete.

[0035] This invention proposes a Transformer-based spatial feature enhancement human pose estimation network, CSIT, which adopts a CNN+Transformer architecture. Considering that visual information consists of texture features and spatial features, this invention expands the feature maps extracted by the CNN from both channel and spatial dimensions to generate Channel Embeddings and Spatial Embeddings. At the same time, considering that the network also needs to learn the dependencies between keypoints, Keypoint Embeddings are added. The three sequences are concatenated together as the input to the Transformer. At the end of the network, this invention also adopts a more advanced one-dimensional vector representation method that is more in line with the characteristics of the Transformer to predict keypoints, thereby further improving the prediction performance of the network. Attached Figure Description

[0036] Figure 1 This is an overall framework diagram of the CSIT network structure of the present invention.

[0037] Figure 2 Generation diagram of Channel Embeddings.

[0038] Figure 3 Generation diagram of Spatial Embeddings.

[0039] Figure 4 Principle diagram of one-dimensional vector representation method SimCC.

[0040] Figure 5 Visualization prediction result diagram of the CSIT network of the application. DETAILED DESCRIPTION

[0041] In order to describe the application more specifically, the technical solutions of the application are described in detail below in combination with the drawings and specific embodiments.

[0042] The human pose estimation method based on the space-time feature enhancement of the Transformer of the application comprises the following steps:

[0043] (1) A large number of pictures containing people are obtained, and the joint positions of the people in the pictures are labeled.

[0044] In the embodiment, the pictures in the COCO dataset are selected for model training. The COCO is a large dataset proposed by Microsoft, which can be used for image classification, target detection, semantic segmentation and pose estimation tasks, and is one of the most influential and most widely used datasets in the field of vision, and the pictures contained therein cover most daily scenes. For each person in the picture, 17 joints are marked, and the dataset contains a total of 200K pictures and 250K human body instances, of which 118K pictures are used for training, 5K pictures are used for verification, and 20K pictures in the test development set are used for online verification. The evaluation indicators used are average accuracy AP and average recall AR, and AP and AR are calculated by OKS (Object Keypoint Similarity). OKS is the similarity between the true value and the predicted value, and the calculation formula is as follows:

[0045]

[0046] Where i represents the i-th joint, d i represents the Euclidean distance between the true value and the predicted value, s is the scale of the target, v i represents whether the target is visible, and δ represents the regularization parameter of the key point.

[0047] The calculation formula of AP and AR is as follows:

[0048]

[0049] where p is the human bounding box number, we denote AP when t is 0.5 and 0.75 50 and AP 75 , when 32 2 < s 2 <96 2 or 96 2 < s 2 , we denote AP m and AP l ; when t takes 0.5, 0.55, …, 0.9, 0.95, the average value we denote AP, and AR can also be divided in the same way.

[0050] (2) The model architecture of CSIT is constructed, which is composed of backbone network, channel feature encoder, spatial feature encoder, Transformer module and one-dimensional vector regression module. Taking two-dimensional pictures as input, the features are extracted by the backbone network first, and then the extracted features are unfolded from the channel dimension and the spatial dimension respectively. Then Channel Embeddings and Spatial Embeddings are output by processing the channel and spatial feature encoder modules. At the same time, the network randomly initializes to generate learnable Keypoints Embeddings. After position encoding of the former two, the three Embeddings are sent into the Transformer module for learning. Finally, the sequence output by the Transformer network is directly predicted by the one-dimensional vector regression method.

[0051] As shown in Figure 1 , CSIT adopts a cascaded network and uses Transformer to fully learn the texture features, spatial features and constraint information between joints of the image. In the end, we use a one-dimensional vector representation method with more advanced performance and more in line with the characteristics of Transformer to predict the key points.

[0052] In this embodiment, we use the powerful HRNetW48 as the backbone network. Assuming that a single-person RGB image with a size of is input, after feature extraction by the multi-layer convolutional neural network, the output feature map is Usually, the size of the output feature map is 1 / 4 of the original image, and we also follow this principle, that is, {H2, W2} = {H1 / 4, W1 / 4}.

[0053] For the feature map extracted by the backbone network, we first unfold it from the channel dimension. The specific unfolding method is as follows Figure 2As shown, X2 is first flattened along the channel dimension. Where L c =H2×W2, then X c After being processed by the channel feature encoder, Channel Embeddings are generated. This encoder consists of linear layers, and its purpose is to reduce the length of the sequence so as to reduce memory usage while preserving fine-grained feature information.

[0054] At the same time, we first sort each feature map in X2 according to {P h ,P w}={H2 / R h W2 / R w} Divide into R s There are several patches, and then each patch is expanded along the spatial dimensions, as shown in the following specific expansion method. Figure 3 As shown, the expanded eigenvector matrix is Where L s =P h ×P w ×C, we pre-set C=R s =R h ×R w Then X s The linear layer in the spatial feature encoder is fed into the linear layer for processing to generate Spatial Embeddings.

[0055] Human pose estimation mainly relies on two types of information: visual information and skeletal constraint information. Channel-Spatial Embeddings primarily address the problem of insufficient visual information extraction in previous models, while Keypoint Embeddings enable the network to better extract constraint information between bones. Therefore, we generate M learnable keypoint vectors (Keypoint Embeddings) through random initialization. In the output stage of the Transformer network, we also extract M feature vectors (Embeddings) as the output of the Transformer network.

[0056] After encoding the Channel-Spatial Embeddings positionally, we feed them together with the Keypoint Embeddings into an N-layer Transformer. Each layer consists of a regularization layer, a multi-head self-attention mechanism, and a multilayer perceptron. Specifically, the input sequence is... R3 = R x +Rs +R k , L = 192, first project X3 by matrix X3 = X3W key value Among them, the calculation is as follows:

[0057] Q = X3W Q , K = XW K , V = X3W V

[0058] Then the self-attention score matrix SA (self-attention) can be expressed as:

[0059]

[0060] Multi-head self-attention mechanism (MSA) is an extension of self-attention mechanism, with h self-attention operations, where h is called head, and the sequence length L h is often set as L / h, and the specific calculation formula is as follows:

[0061] MSA (X3) = [SA1 (X3); SA2 (X3);...; SA h (X3)]W P

[0062] Where: X3 is the input, The attention score obtained determines the attention degree of the current query to the Embedding; after processing by N layers of Transformer, the network outputs M Embeddings, and the size of M depends on the number of nodes.

[0063] In this embodiment, we use one-dimensional vector representation method SimCC to predict key points, and the specific principle is as shown in Figure 4 The vectors output by the Transformer module are respectively sent into the X and Y coordinate classifiers to respectively generate M X and Y vectors, and the calculation formula is as shown below:

[0064] X, Y = MLP x (X4), MLP y (X4)

[0065] Wherein: MLP x and MLP y are composed of linear layers, X ∈ R M×(W·k) , Y ∈ R M×(H·k) , H and W are the height and width of the input picture, and k is the scaling factor, which is taken as k = 2 in this embodiment.

[0066] Subsequently, the generated X, Y vector is subjected to a decoding operation, the formula is as follows:

[0067]

[0068]

[0069] Where: sigma is the standard deviation, (x', y') is the pixel point on the (X i ,Y i ) vector, (x i ,y i ) is the supervisory signal generated by one-dimensional Gaussian distribution, is the predicted joint coordinates.

[0070] (3) Train the CSIT model framework using the pictures and labels in step (1).

[0071] (4) Send the person picture into the trained model, that is, the coordinates of the joint can be predicted.

[0072] Experiment: We use the top-down general method, in COCO dataset we crop the input single person picture to 256x192 size, in MPII we crop to 256x256 size, the initial learning rate is set to 0.001, Adam optimizer is used, Epoch is set to 210, since the Transformer can be stacked with multiple layers, we distinguish our network into CSIT-B and CSIT-L by setting the number of layers N and the number of heads in the multi-head self-attention mechanism, the specific settings are shown in Table 1:

[0073] Table 1

[0074]

[0075] COCO validation dataset: As can be seen from Table 2, the CSIT-B network proposed by us has achieved good results in the main indicators, compared with HRNet-W48, the AP has increased by 1.8, while the GFLOPs has only increased by 1%, compared with HRNet, we have achieved a substantial leap in the main indicators at a cost that can be ignored. For Simple Baseline R-152, the AP has increased by 4.9, while the GFLOPs has decreased by 6%, which also shows the superiority of the CSIT network; and the CSIT-L has achieved better results, the AP has reached 77.5, far exceeding other networks. At the same time, in order to more intuitively illustrate the performance of CSIT, we verify it in the COCO validation set and show the visualization results as Figure 5shown.

[0076] Table 2

[0077]

[0078]

[0079] Note: Table 2 is the comparison results of each version of CSIT and mainstream models on COCO validation test set, where R represents ResNet, H represents HRNet, and the best result in each column is marked in bold.

[0080] COCO test-dev dataset: As shown in Table 3, we also tested on COCO test-dev, and the performance is better than the CNN-based model, and there is also a significant advantage compared with the CNN+Transformer network, where TFPose and TransPose use regression method and heat map method to predict key points respectively, but compared with the one-dimensional vector representation method used by CSIT, there is still a significant disadvantage; compared with TFPose, the AP of CSIT-L network increases by 2.6 under the condition of 26% reduction of GFLOPs, and other indicators also show satisfactory results.

[0081] Table 3

[0082]

[0083] Note: Table 3 is the comparison results of each version of CSIT and CNN-based models and CNN+Transformer models on COCO test-dev, where R50 represents ResNet50, and H32 represents HRNetW32, and the best result in each column is marked in bold.

[0084] MPII validation dataset: We uniformly crop the input picture to 256x256, and the experimental results when PCK@0.5 are shown in Table 4, CSIT-B achieves the best performance on Wri, and the second best performance on other parts, while CSIT-L shows the best performance on most indicators, and is 0.4 percentage points higher than the best HRNet on Mean. In short, CSIT also performs very competitively on the mainstream MPII dataset.

[0085] Table 4

[0086]

[0087]

[0088] Note: Table 4 is the comparison result of each version of CSIT and mainstream model on the MPII validation set, where Mean is set as (PCK@0.5), and the size of the input picture is 256*256.

[0089] Summary: There are many excellent works of Transformer in the field of vision, such as PoseFormer, TransPose, TFPose, TokenPose, POET, DPIT and CSIT proposed in this application, but pose estimation is not a simple detection task, but a more advanced visual task, the information to be learned by the network has complexity and richness, in addition to visual information, anatomical constraint information should also be learned, which has been recognized in TokenPose, therefore it cannot be simply summarized by a single network, although VitPose has become an SOTA model by a very violent method, but combined with the current research status, CNN+Transformer may be a more reasonable solution, making full use of their respective advantages, CNN is used to extract visual information, and Transformer is used to learn the dependence between features and key points. Therefore, the inventor just sees this point and proposes CSIT network, which is another excellent work for CNN+Transformer camp.

[0090] The above description of the embodiments is to facilitate those of ordinary skill in the art to understand and apply the present application, and those skilled in the art can easily make various modifications to the above embodiments, and apply the general principles described herein to other embodiments without inventive labor. Therefore, the present application is not limited to the above embodiments, and the improvements and modifications of the present application made by those skilled in the art according to the disclosure of the present application should be within the scope of protection of the present application.

Claims

1. A human pose estimation method based on Transformer spatiotemporal feature enhancement, comprising the following steps: (1) Obtain a large number of images containing people and mark the key points of the people in the images; (2) Construct the CSIT network model framework, which includes a backbone network, channel feature encoder, spatial feature encoder, Transformer module, and one-dimensional vector regression module, wherein: The backbone network is used to extract features from the input image; The channel feature encoder is used to expand the extracted feature map from the channel dimension and compress the sequence length to output Channel Embeddings; The spatial feature encoder is used to expand the extracted feature map in the spatial dimension and compress the sequence length to output Spatial Embeddings; After being positionally encoded, Channel Embeddings and Spatial Embeddings, together with the learnable KeypointsEmbeddings, are concatenated along the channel dimension and input into the Transformer module. KeypointsEmbeddings are generated through random initialization, and their number of channels is the number of keypoints M. The Transformer module is used to encode the input, thereby outputting a feature vector with M channels; The one-dimensional vector regression module is used to decode the feature vectors of these M channels, thereby outputting the predicted coordinates of each joint point. (3) Use the images and their annotation information obtained in step (1) to train the above model framework; (4) Input the image containing the person to be estimated into the trained model, and the position coordinates of each joint of the person in the image can be predicted.

2. The human pose estimation method according to claim 1, characterized in that: The backbone network adopts a multi-layer convolutional neural network based on the HRNetW48 structure. After feature extraction of the input image by the multi-layer convolutional neural network, a feature map of size R×H2×W2 is output, where R is the number of channels, H2=H / 4, W2=W / 4, and W and H are the width and height of the input image, respectively.

3. The human pose estimation method according to claim 2, characterized in that: The channel feature encoder flattens the feature map of size R×H2×W2 into R×L. c After being sized, the data is fed into a linear layer for processing and to generate Channel Embeddings, L c =H2×W2; The linear layer is used to further reduce the length L of the feature vector. c This aims to reduce memory usage while preserving fine-grained feature information.

4. The human pose estimation method according to claim 2, characterized in that: The spatial feature encoder first segments the feature map of size R×H2×W2, that is, it segments the feature map of each channel into C P-values. h ×P w Patch size, where P h =H2 / R h P w =W2 / R w C=R h ×R w And C = R; Then, each patch is flattened and stitched together along the channel dimension to obtain C×L. s The size of the feature matrix, L s =P h ×P w ×R, and finally the feature matrix is ​​fed into a linear layer for processing to generate Spatial Embeddings. The linear layer is used to further reduce the length L of the feature vector. s This aims to reduce memory usage while preserving fine-grained feature information.

5. The human pose estimation method according to claim 1, characterized in that: The Transformer module consists of a cascaded multi-layer coding structure. Each coding structure is composed of a regularization layer L1, a multi-head self-attention mechanism layer, a regularization layer L2, and a multilayer perceptron (MLP) connected in sequence. The input of the regularization layer L2 is the superposition of the output of the multi-head self-attention mechanism layer and the input of the regularization layer L1. The output of the multilayer perceptron (MLP) is superimposed with the input of the regularization layer L2 and used as the output of the coding structure.

6. The human pose estimation method according to claim 5, characterized in that: The multi-head self-attention mechanism layer consists of multiple Self-Attention mechanisms. Each Self-Attention mechanism modifies the input of the multi-head self-attention mechanism layer with a linear transformation matrix W. Q W K and W V Multiplication yields the query vector matrix Q, the key vector matrix K, and the value vector matrix V. The output of Self-Attention is then calculated using the following formula. Finally, the outputs of multiple Self-Attention layers are concatenated and passed through a linear layer to obtain the output of the multi-head self-attention mechanism layer. Where: Attention(Q,K,V) is the output of Self-Attention, and s is the number of columns in the query vector matrix Q. T This indicates transpose.

7. The human pose estimation method according to claim 1, characterized in that: The one-dimensional vector regression module processes the feature vectors of the M channels through a horizontal axis coordinate classifier and a vertical axis coordinate classifier, respectively. The horizontal axis coordinate classifier reduces the length of each feature vector to kW, resulting in vectors X1, X2, ..., X. M Then, the x-coordinates of the predicted points are extracted from them through mapping. The vertical axis coordinate classifier reduces the length of each feature vector to kH, resulting in vectors Y1, Y2, ..., Y. M Then, the ordinates of the predicted points are extracted from them through mapping. Finally, the predicted coordinates of each joint are output. W and H are the width and height of the input image, respectively, i is a natural number and 1≤i≤M, and k is a scaling factor greater than 1.

8. The human pose estimation method according to claim 7, characterized in that: The predicted coordinates The expression is as follows: Where: σ is the set standard deviation. Let X be a vector i The value of the p-th element in the middle. For vector Y i Let p be a natural number and 1 ≤ p ≤ kW, and q be a natural number and 1 ≤ q ≤ kH.

9. The human pose estimation method according to claim 1, characterized in that: The specific implementation method of step (3) is as follows: 3.1 Initialize model parameters, including the bias vector and weight matrix of each layer, learning rate, and optimizer; 3.2 Input the image containing people into the model, and the model outputs the corresponding prediction result, i.e., the coordinates of the key points, through forward propagation. Calculate the loss function between the prediction result and the annotation information. 3.3 Based on the loss function, the optimizer iteratively updates the model parameters using gradient descent until the loss function converges, and the training is complete.

Images