CNN and Transform-based 2DToken human body posture estimation method
By using the 2DToken processing method combined with CNN and Transformer in human posture estimation, the Transformer module is optimized and the key point Token representation is improved. The problem of insufficient accuracy of existing methods in complex scenarios is solved, and more efficient and accurate human posture estimation is achieved. .
Patent Information
- Application Number
- CN202411469033.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-21
- Publication Date
- 2025-02-18
AI Technical Summary
The existing TokenPose method has problems such as missing detection, missed detection, and insufficient model accuracy when estimating human postures in complex scenarios, and the number of parameters is relatively large.
A 2DToken human pose estimation method based on CNN and Transformer is proposed. By constructing a network including CNN module, 2DToken Processing module, FeatTransformer module, KeypointExtration module and HeatmapRegression module, the Transformer module is optimized to be a FeatTransformer module, so that it can be self-contained in high-width dimensions. Attention operation and change the key point token from 1D to 2D to save the high-width dimension information.
The accuracy improvement and computational volume reduction are achieved on the MPII and Crowdpose datasets, reducing the number of model parameters and improving the accuracy of human pose estimation.
Smart Images
Figure CN119479009A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of computer technology, and in particular to a human body posture estimation technology, and specifically to a human body posture estimation method based on the combination of a convolutional neural network (CNN) and a Transformer. Background Art
[0002] The goal of human posture estimation is to accurately identify the key points of the human body from images or videos, and estimate the relative positions and relationships between these key points, so as to infer the posture of the human body, so as to provide a theoretical basis for downstream action recognition, human-computer interaction and other fields.
[0003] The CNN-based human posture estimation method mainly extracts features by stacking convolutional layers. Although it can learn the local information of the image well, its ability to capture global contextual information is limited. When faced with occlusion problems in complex scenes, the detection error of key points is large.
[0004] The paper "Tokenpose: Learning keypoint tokens for human pose estimation" proposes a human pose estimation method Tokenpose based on CNN and Transformer. The method processes the human pose estimation task by first using a convolutional neural network CNN to extract local features from the initial input image, then converting the extracted feature map into a visual token and inputting it into the Transformer module together with the randomly initialized key point token. Finally, the key point token output by the Transformer module is restored to a high-resolution heat map through the MLP layer. However, this method still has some imperfections:
[0005] 1) The key point token only has one dimension of information and cannot accurately describe the information carried by the key point.
[0006] 2) The input of the Transformer module can only be in the input format of 1DToken. For images, they need to be flattened before they can be input into the Transformer. However, the flattening operation will lose the height and width dimensional information of the image, thus affecting the accuracy of the model.
[0007] Therefore, based on the above literature, the applicant proposed a new 2DToken human posture estimation method that combines CNN and Transformer, optimizing the traditional Transformer module into a FeatTransformer module, so that it can directly perform self-attention operations on the feature map in the high and wide dimensions respectively, and at the same time, changing it from 1DToken to 2DToken in the key point Token generation stage, so that the key point Token can carry richer information. Summary of the invention
[0008] The present invention aims to solve the technical problems of missed key point detection, wrong detection and insufficient model accuracy when the existing TokenPose method is used to estimate human posture in complex scenes due to the loss of spatial information. At the same time, an improved posture estimation method is proposed to address the defect of a large number of parameters in the original method.
[0009] In order to solve the above technical problems, the technical solution adopted by the present invention is:
[0010] A 2DToken human posture estimation method based on CNN and Transformer, which includes the following steps:
[0011] Step 1: Select a dataset that can be used in the field of posture estimation, including the MPII dataset and the Crowdpose dataset;
[0012] Step 2: Build a human posture estimation network based on CNN and Transformer, which includes: CNN module, 2DToken Processing module, FeatTransformer module, KeypointExtration module, HeatmapRegression module;
[0013] Step 3: Set the model's loss function and configure related configuration parameters to train the model;
[0014] Step 4: Use the trained model to estimate the pose of the person in the test image on the test set;
[0015] Step 5: Evaluate the model through evaluation indicators and select the optimal model;
[0016] Step 6: Input the image to be detected into the trained model to obtain the heat map and coordinates of key points of the human body;
[0017] The above steps are used to estimate the human body posture.
[0018] In step 2, the human posture estimation network is constructed as follows:
[0019] The initial image F1 is input to the CNN module, the output of the CNN module is connected to the input of the 2DToken Processing module, the output of the 2DToken Processing module is connected to the input of the FeatTransformer module, the output of the FeatTransformer module is connected to the input of the KeypointExtration module, the output of the KeypointExtration module is connected to the input of the HeatmapRegression module, and the output of the HeatmapRegression module is the required key point heat map;
[0020] For the 2DToken Processing module, its structure is as follows: it includes a SplitPatchs module, the input of the SplitPatchs module is used to input the feature map F2, the output of the SplitPatchs module is added to the position code P to obtain the 2D visual Token feature, and then the Token feature is spliced with the 2D key point Token feature automatically generated in the 2DToken Processing module to obtain the output feature map F4 of the 2DToken Processing module.
[0021] The human pose estimation network uses the following steps when working:
[0022] Step 1) extract local features from the initial image through a CNN module; the CNN module uses a parallel high-resolution HRNet network. Specifically, the initial image F1 of size {B, 3, H, W} is connected to the input of the HRNet network, and then the output size is The feature map F2 is obtained by , where B is the number of batch images, H and W are the height and width of the initial image, 3 represents the RGB channels of the initial image, and C is the dimension of the output feature map;
[0023] Step 2) The 2DToken Processing module processes the feature map output by the CNN module into an input acceptable to the FeatTransformer module; specifically, the SplitPatchs module first splits the feature map F2 into several patches of size {B, C p ,h,w} feature block F3, and then generate the corresponding 2D-sine position code P for each split feature block. The matrix addition of F3 and P gives the same size {B,C p,h,w}2D visual Token features; at the same time, the 2DToken Processing module will automatically generate 2D key point Token features of size {B,num,h,w} through random initialization, and finally concatenate the 2D key point Token features and the 2D visual Token features to obtain the output feature F4 of the 2DToken Processing module; where C p is the dimension of the feature block, h, w are the height and width of the feature block, and num is the number of key points;
[0024] Step 3) The FeatTransformer module performs multi-head self-attention on the input Token to learn the context information of the image; specifically, the input feature F4 of size {B, n, h, w} will perform multi-head self-attention in parallel in the height and width dimensions, and finally obtain the output Token feature F5 of the same size {B, n, h, w}; where n is the number of key points num and the feature block dimension C P The number of FeatTransformer modules K required and the number of multi-head self-attention involved in the module are all related to the specification settings of the model;
[0025] Step 4) Extract the required key point Token information from the output of the FeatTransformer module; specifically, directly extract the first num elements in the input Token F5, and then obtain the 2D key point Token feature F6;
[0026] Step 5) The input 2D key point Token feature F6 is converted into the required key point heat map through the HeatmapRegression module; specifically, the size of the key point Token feature is converted from {B, num, h, w} to the size of the required heat map through linear transformation
[0027] The network structure of the FeatTransformer module in step 3) is as follows:
[0028] The FeatTransformer module mainly includes parallel W and H dimension processing. Figure 3 The upper middle half is the W dimension, and the lower middle half is the H dimension;
[0029] In the W dimension, the input Token F4 is connected to the input of the first LayerNorm module, and the output of the first LayerNorm module is connected to the input of the Attention W module. The output of the Attention W module is added to the input Token F4 as the output of the W dimension. For the Attention W module, it generates the query matrix, key matrix, and value matrix Q according to the Token at the input. w ,K w ,V w , Q w The output connection of the matrix K w The output of the matrix performs the first matrix multiplication. The output of the first matrix multiplication is connected to the input of the first Softmax module. The output of the first Softmax module is connected to V w The output of the matrix undergoes a second matrix multiplication, and the output of the second matrix multiplication is the output of the Attention W module;
[0030] In the H dimension, the input Token feature F4 is connected to the input of the first Reshape module, the output F7 of the first Reshape module is connected to the input of the second LayerNorm module, the output of the second LayerNorm module is connected to the input of the Attention H module, and the output of the Attention H module is added to F7 as the output of the H dimension; for the Attention H module, it will generate the query matrix, key matrix, and value matrix Q according to the Token at the input end. H ,K H ,V H , Q H The output connection of the matrix K H The output of the matrix is multiplied for the third time. The output of the third matrix multiplication is connected to the input of the second Softmax module. The output of the second Softmax module is connected to V H The output of the matrix undergoes a fourth matrix multiplication, and the output of the fourth matrix multiplication is the output of the Attention H module;
[0031] The output of the H dimension is connected to the input of the second Reshape module. The output of the second Reshape module is added to the output of the W dimension to obtain the output TokenF8. F8 is connected to the input of the third LayerNorm module. The output of the third LayerNorm module is connected to the input of Conv. The output of Conv is added to F8 to obtain the output feature F5 of the FeatTransformer module.
[0032] The FeatTransformer module calculates self-attention in parallel on the input Token feature F4 in the height and width dimensions to learn the contextual information of the image;
[0033] For the W dimension, directly perform the first LayerNorm module operation on the input Token feature F4 to obtain T W , and then input it into the Attention W module; according to formula (1), the input Token is output after the first LayerNorm module is operated W ∈R (B,n,h,w) Will pass three transformation matrices respectively Mapped to query matrix, key matrix, value matrix Q in turn w ,K w ,V w ;
[0034]
[0035] Then according to formula (2), Q W Matrix and K W The transposed matrix of is multiplied to obtain the attention score, and it is connected to the input of the first Softmax module. The output of the first Softmax module is then multiplied by the matrix V W Multiply them together to get the output of Attention W;
[0036]
[0037] For the H dimension, the input Token feature F4 is first operated by the first Reshape module to swap the h and w dimensions of F4 to obtain F6, and then the second LayerNorm module is operated on it, and finally it is input into the Attention H module; according to formula (3), the output T after the second LayerNorm module is H ∈R (B,n,w,h) Through three transformation matrices respectively Mapped into query matrix, key matrix, value matrix Q H ,K H ,V H ;
[0038]
[0039] Then according to formula (4), Q H Matrix and K H The transposed matrix of is multiplied to obtain the attention score, and it is connected to the input of the second Softmax module. The output of the second Softmax module is then multiplied by the matrix V HMultiply them together to get the output of Attention H;
[0040]
[0041] The output of Attention W is added to feature F4 to use as the output of W dimension. The output of Attention H is first added to feature F7, and then the second Reshape module operation is performed to swap the h and w dimensions to use as the output of H dimension. The output of W dimension is then added to the output of H dimension to obtain the output Token feature F8. Feature F8 goes through the third LayerNorm module and Conv module in turn, and the output of the convolution Conv module is matrix-added with the initial feature F8, and finally the output Token feature F5 that has learned the global context information is obtained.
[0042] The network structure of the HeatmapRegression module in step 5) is as follows:
[0043] The input end of the HeatmapRegression module receives the output feature F6 from the KeypointExtration module, the feature F6 is connected to the input end of the third Reshape module, the output of the third Reshape module is connected to the input of the fourth LayerNorm module, the output of the fourth LayerNorm module is connected to the input of the first Liner module, the output of the first Liner module is connected to the input of the fifth LayerNorm module, the output of the fifth LayerNorm module is connected to the input of the second Liner module, the output of the second Liner module is connected to the input of the fourth Reshape module, and the output of the fourth Reshape module is the output of the HeatmapRegression module;
[0044] The HeatmapRegression module mainly includes the Reshape module, the LayerNorm module and the Linear module; the main function of the Reshape module is to reshape the shape of the Token feature, and the purpose of reshaping is to perform subsequent linear operations; the third Reshape module reshapes the shape of the input Token from {B, num, h, w} to {B, num, h*w}; the fourth and fifth LayerNorm modules both normalize the data to accelerate the convergence speed; the Linear module performs a linear transformation on the Token feature. The first Linear module converts the shape of the Token feature from {B, num, h*w} to {B, num, hidden}, and the second Linear module converts the shape of the Token from {B, num, hidden} to Finally, the fourth Reshape module converts the output of the second Linear module Reshape Thus, the num key point heat maps required by the network are obtained; hidden is the number of hidden layers, and its value is set manually.
[0045] Compared with the prior art, the present invention has the following technical effects:
[0046] 1) The present invention adopts a network structure combining CNN and Transformer, which respectively utilizes the advantages of CNN in local feature extraction and Transformer in global context information learning. The network framework itself has strong advantages;
[0047] 2) The present invention proposes the concept of 2D key point Token, each Token represents a specific key point of the human body (such as head, shoulder, neck, etc.). Compared with the existing 1D key point Token, the 2D key point Token avoids the loss of key point Token in height and width dimensional information at the source;
[0048] 3) The present invention optimizes the traditional Transformer Encoder module, and calculates the self-attention of the input 2DToken in parallel in the height and width dimensions, thereby reducing the number of model parameters and preserving the height and width dimensional information of the input Token;
[0049] 4) Compared with the TokenPose network, the proposed method achieves improved accuracy and reduced computational complexity on both the MPII dataset and the Crowdpose dataset. BRIEF DESCRIPTION OF THE DRAWINGS
[0050] The present invention will be further described below in conjunction with the accompanying drawings and embodiments:
[0051] Figure 1 is a flow chart of the present invention;
[0052] Figure 2 This is a schematic diagram of the structure of the FeatToken model for human posture estimation proposed in the present invention;
[0053] Figure 3 for Figure 2 Schematic diagram of the model structure of the FeatTransformer module;
[0054] Figure 4 for Figure 2 Schematic diagram of the model structure of the HeatmapRegression module in . DETAILED DESCRIPTION
[0055] like Figure 1 As shown in the figure, a 2DToken human posture estimation method based on CNN and Transformer includes the following steps:
[0056] Step 1: Select a dataset that can be used in the field of posture estimation, including the MPII dataset and the Crowdpose dataset;
[0057] Step 2: Build a human posture estimation network based on CNN and Transformer, which includes: CNN module, 2DToken Processing module, FeatTransformer module, KeypointExtration module, HeatmapRegression module;
[0058] Step 3: Set the model's loss function and configure related configuration parameters to train the model;
[0059] Step 4: Use the trained model to estimate the pose of the person in the test image on the test set;
[0060] Step 5: Evaluate the model through evaluation indicators and select the optimal model;
[0061] Step 6: Input the image to be detected into the trained model to obtain the heat map and coordinates of key points of the human body;
[0062] The above steps are used to estimate the human body posture.
[0063] In step 2, the human posture estimation network is constructed as follows:
[0064] The initial image F1 is input to the CNN module, the output of the CNN module is connected to the input of the 2DToken Processing module, the output of the 2DToken Processing module is connected to the input of the FeatTransformer module, the output of the FeatTransformer module is connected to the input of the KeypointExtration module, the output of the KeypointExtration module is connected to the input of the HeatmapRegression module, and the output of the HeatmapRegression module is the required key point heat map;
[0065] For the 2DToken Processing module, its structure is as follows: it includes a SplitPatchs module, the input of the SplitPatchs module is used to input the feature map F2, the output of the SplitPatchs module is added to the position code P to obtain the 2D visual Token feature, and then the Token feature is spliced with the 2D key point Token feature automatically generated in the 2DToken Processing module to obtain the output feature map F4 of the 2DToken Processing module.
[0066] When the human body posture estimation network is working, the following steps are taken:
[0067] Step 1) extract local features from the initial image through a CNN module; the CNN module uses a parallel high-resolution HRNet network. Specifically, the initial image F1 of size {B, 3, H, W} is connected to the input of the HRNet network, and then the output size is The feature map F2 is obtained by , where B is the number of batch images, H and W are the height and width of the initial image, 3 represents the RGB channels of the initial image, and C is the dimension of the output feature map;
[0068] Step 2) The 2DToken Processing module processes the feature map output by the CNN module into an input acceptable to the FeatTransformer module; specifically, the SplitPatchs module first splits the feature map F2 into several patches of size {B, C p ,h,w} feature block F3, and then generate the corresponding 2D-sine position code P for each split feature block. The matrix addition of F3 and P gives the same size {B,C p ,h,w}2D visual Token features; at the same time, the 2DToken Processing module will automatically generate 2D key point Token features of size {B,num,h,w} through random initialization, and finally concatenate the 2D key point Token features and the 2D visual Token features to obtain the output feature F4 of the 2DToken Processing module; where C p is the dimension of the feature block, h, w are the height and width of the feature block, and num is the number of key points;
[0069] Step 3) The FeatTransformer module performs multi-head self-attention on the input Token to learn the context information of the image; specifically, the input feature F4 of size {B, n, h, w} will perform multi-head self-attention in parallel in the height and width dimensions, and finally obtain the output Token feature F5 of the same size {B, n, h, w}; where n is the number of key points num and the feature block dimension C P The number of FeatTransformer modules K required and the number of multi-head self-attention involved in the module are all related to the specification settings of the model;
[0070] Step 4) Extract the required key point Token information from the output of the FeatTransformer module; specifically, directly extract the first num elements in the input Token F5, and then obtain the 2D key point Token feature F6;
[0071] Step 5) The input 2D key point Token feature F6 is converted into the required key point heat map through the HeatmapRegression module; specifically, the size of the key point Token feature is converted from {B, num, h, w} to the size of the required heat map through linear transformation
[0072] The network structure of the FeatTransformer module in step 3) is as follows:
[0073] The FeatTransformer module mainly includes parallel W and H dimension processing. Figure 3 The upper middle half is the W dimension, and the lower middle half is the H dimension;
[0074] In the W dimension, the input Token F4 is connected to the input of the first LayerNorm module 1, and the output of the first LayerNorm module 1 is connected to the input of the Attention W module. The output of the Attention W module is added to the input Token F4 as the output of the W dimension. For the Attention W module, it generates the query matrix, key matrix, and value matrix Q according to the Token at the input. w ,K w ,V w , Q w The output connection of the matrix K w The output of the matrix performs the first matrix multiplication, and the output of the first matrix multiplication is connected to the input of the first Softmax module 2. The output of the first Softmax module 2 is connected to V w The output of the matrix undergoes a second matrix multiplication, and the output of the second matrix multiplication is the output of the Attention W module;
[0075] In the H dimension, the input Token feature F4 is connected to the input of the first Reshape module 3, the output F7 of the first Reshape module 3 is connected to the input of the second LayerNorm module 4, the output of the second LayerNorm module 4 is connected to the input of the AttentionH module, and the output of the AttentionH module is added to F7 as the output of the H dimension; for the AttentionH module, it will generate the query matrix, key matrix, and value matrix Q according to the Token at the input end. H ,K H ,V H , Q H The output connection of the matrix K H The output of the matrix is multiplied for the third time, and the output of the third matrix multiplication is connected to the input of the second Softmax module 5. The output of the second Softmax module 5 is connected to V H The output of the matrix undergoes a fourth matrix multiplication, and the output of the fourth matrix multiplication is the output of the Attention H module;
[0076] The output of the H dimension is connected to the input of the second Reshape module 6. The output of the second Reshape module 6 is added to the output of the W dimension to obtain the output TokenF8. F8 is connected to the input of the third LayerNorm module 7. The output of the third LayerNorm module 7 is connected to the input of Conv. The output of Conv is added to F8 to obtain the output feature F5 of the FeatTransformer module.
[0077] The FeatTransformer module calculates self-attention in parallel on the input Token feature F4 in the height and width dimensions to learn the contextual information of the image;
[0078] For the W dimension, directly perform the first LayerNorm module 1 operation on the input Token feature F4 to obtain T W , and then input it into the Attention W module; according to formula (1), the input Token is output after the first LayerNorm module 1 operation T W ∈R (B,n,h,w) Will pass three transformation matrices respectively Mapped to query matrix, key matrix, value matrix Q in turn w ,K w ,V w ;
[0079]
[0080] Then according to formula (2), Q W Matrix and K W The transposed matrix of is multiplied to obtain the attention score, and it is connected to the input of the first Softmax module 2. The output of the first Softmax module 2 is then multiplied by the matrix V W Multiply them together to get the output of Attention W;
[0081]
[0082] For the H dimension, the input Token feature F4 is first operated by the first Reshape module 3 to swap the h and w dimensions of F4 to obtain F6, and then the second LayerNorm module 4 is operated on it, and finally it is input into the Attention H module; according to formula (3), the output T after the second LayerNorm module 4 is H ∈R (B,n,w,h) Through three transformation matrices respectively Mapped into query matrix, key matrix, value matrix Q H ,K H ,V H ;
[0083]
[0084] Then according to formula (4), Q H Matrix and K H The transposed matrix of is multiplied to obtain the attention score, and it is connected to the input of the second Softmax module 5. The output of the second Softmax module 5 is then multiplied by the matrix V H Multiply them together to get the output of Attention H;
[0085]
[0086] The output of Attention W is added to the feature F4 to use as the output of the W dimension. The output of Attention H is first added to the feature F7, and then the second Reshape module 6 is used to swap the h and w dimensions to use as the output of the H dimension. The output of the W dimension is then added to the output of the H dimension to obtain the output Token feature F8. Feature F8 goes through the third LayerNorm module 7 and the Conv module in turn, and the output of the convolution Conv module is matrix-added with the initial feature F8, and finally the output Token feature F5 that has learned the global context information is obtained.
[0087] The network structure of the HeatmapRegression module in step 5) is as follows:
[0088] The input end of the HeatmapRegression module receives the output feature F6 from the KeypointExtration module, and the feature F6 is connected to the input end of the third Reshape module 8. The output of the third Reshape module 8 is connected to the input of the fourth LayerNorm module 9. The output of the fourth LayerNorm module 9 is connected to the input of the first Liner module 10. The output of the first Liner module 10 is connected to the input of the fifth LayerNorm module 11. The output of the fifth LayerNorm module 11 is connected to the input of the second Liner module 12. The output of the second Liner module 12 is connected to the input of the fourth Reshape module 13. The output of the fourth Reshape module 13 is the output of the HeatmapRegression module;
[0089] The HeatmapRegression module mainly includes the Reshape module, the LayerNorm module and the Linear module; the main function of the Reshape module is to reshape the shape of the Token feature, and the purpose of reshaping is to perform subsequent linear operations; the third Reshape module 8 reshapes the shape of the input Token from {B, num, h, w} to {B, num, h*w}; the fourth LayerNorm module 9 and the fifth LayerNorm module 11 both normalize the data to accelerate the convergence speed; the Linear module performs a linear transformation on the Token feature, the first Linear module 10 converts the shape of the Token feature from {B, num, h*w} to {B, num, hidden}, and the second Linear module 12 converts the shape of the Token from {B, num, hidden} to Finally, the fourth Reshape module 13 converts the output of the second Linear module 12 Reshape Thus, the num key point heat maps required by the network are obtained; hidden is the number of hidden layers, and its value is set manually.
[0090] The CNN module uses the HRNet network to perform preliminary local feature extraction on the initial image. Specifically, the initial image F1 with a size of {B, 3, H, W} is connected to the input of the HRNet network. The output of the HRNet network is a The feature map of , where B is the number of batch images, H, W are the height and width of the initial image, 3 represents the RGB channels of the initial image, and C is the dimension of the output feature map.
[0091] The 2DToken Processing module processes the output of the CNN module into an input that the FeatTransformer module can accept. Specifically, the feature map F2 output by the CNN module is first split into several blocks of size {B, C p ,h,w} feature block F3, and then add the position encoding information to each feature block to get the 2D visual Token; at the same time, the 2D key point Token of size {B,num,h,w} is randomly initialized, and finally the 2D key point Token and the 2D visual Token are concatenated to get the output Token F4 of size {B,n,h,w}. p is the dimension of the feature block, h and w are the height and width of the feature block, num is the number of key points, and n is the sum of num and C P The sum.
[0092] The function of the FeatTransformer module is to perform self-attention on the input Token to learn the context information of the image. Specifically, the output of the 2DToken Processing module is connected to the input of the first FeatTransformer module, and the output of the first FeatTransformer module is connected to the input of the second FeatTransformer module, and so on. A total of K FeatTransformer modules are required, and the value of K is related to the specifications of the model.
[0093] The function of the KeypointExtration module is to extract the required key point Token information from the output of the FeatTransformer module. In simple terms, it directly extracts the first num elements in the input Token F5, and then obtains the 2D key point Token F6.
[0094] The function of the HeatmapRegression module is to obtain the heat map of the required key points through a series of processing of the output of the KeypointExtration module. Specifically, the size of the key point Token is converted from {B, num, h, w} to the size of the required heat map through linear transformation.
[0095] In order to facilitate those skilled in the art to better understand the present invention, the following embodiments are provided;
[0096] Example:
[0097] The code of the present invention is implemented based on the Pytorch framework. The NVIDIA GeForce RTX3080 graphics card is used to train the model under Ubuntu. The Adam optimizer is used to optimize the network. The initial learning rate is set to 0.0001. The MultiStepLR learning rate adjustment strategy is used to reduce the learning rate to one tenth of the original at 100, 150, and 200 rounds, respectively. The maximum number of learning times is set to 210. The MPII dataset and the Crowdpose dataset are selected for testing.
[0098] Table 1 Specific configuration details of the FeatToken network
[0099]
[0100] The FeatToken network proposed in this paper has three scales: S, B, and L. The specific configuration details are shown in Table 1. Among them, CNNbackbone represents the network adopted by the CNN module. Since the complete HRNet will cause too many parameters and seriously affect the training effect, the present invention adopts a small HRNet with only 3 stages (the complete HRNet has four stages). Stem-net is a CNN module composed of two convolutional layers with 3*3 convolution kernels and a RELU layer. The value of Layers represents the number of FeatTransformer modules, and Heads represents the number of heads in the multi-head attention mechanism in the FeatTransformer module. PatchSize is the parameter required for the SplitPatchs module to split feature blocks. The value of PatchSize of the present invention is 4*4 when it is on the MPII dataset, and the value of PatchSize is 4*3 when it is on the Crowdpose dataset.
[0101] Table 2 Experimental results on the MPII validation set
[0102]
[0103]
[0104] Table 2 shows the experimental results of the FeatToken network on the MPII dataset. It can be seen from Table 2 that on the MPII dataset, the FeatToken-L network proposed in the present invention has a good improvement compared with the original network TokenPose-L. Among them, the FeatToken-L-D6 network has a 0.4% improvement in accuracy Mean compared with the original TokenPose-L-D6 network, and the number of parameters and computational complexity are reduced by 8.9% and 1.5% respectively; the FeatToken-L-D24 network has a 0.2% improvement in accuracy Mean compared with the original TokenPose-L-D24 network, and the number of parameters and computational complexity are reduced by 14.2% and 4.1% respectively.
[0105] Table 3 Experimental results on the Crowdpose test set
[0106]
[0107] Table 3 shows the experimental results of the FeatToken network on the Crowdpose dataset. Compared with the original network TokenPose, the FeatToken network proposed in the present invention has been improved in the three scales of S, B, and L, with the accuracy increased by 0.2%, 0.2%, and 0.3% respectively, and the number of parameters has been reduced by 26.9%, 13.2%, and 13.1% respectively. In addition, by comparing the experimental results of FeatToken-S and FeatToken-B, it can be seen that the accuracy difference between the two is large, which is mainly because the two use different CNN modules. This also shows that the choice of CNN module has a greater impact on the accuracy of the network.
[0108] Experimental data on the MPII dataset and Crowdpose dataset show that the FeatToken network not only improves accuracy while saving computational costs, but also performs well on the Crowdpose dataset specifically designed for complex scenarios. In particular, the accuracy of FeatToken-B and FeatToken-L on the Crowdpose dataset reached an upper-middle level, which also shows that the FeatToken network has good commercial value.
Claims
1. A 2DToken human posture estimation method based on CNN and Transformer, characterized in that: It contains the following steps: Step 1: Select a dataset that can be used in the field of posture estimation, including the MPII dataset and the Crowdpose dataset; Step 2: Build a human posture estimation network based on CNN and Transformer, which includes: CNN module, 2DToken Processing module, FeatTransformer module, KeypointExtration module, HeatmapRegression module; Step 3: Set the model's loss function and configure related configuration parameters to train the model; Step 4: Use the trained model to estimate the pose of the person in the test image on the test set; Step 5: Evaluate the model through evaluation indicators and select the optimal model; Step 6: Input the image to be detected into the trained model to obtain the heat map and coordinates of key points of the human body; The above steps are used to estimate the human body posture.
2. The method according to claim 1, characterized in that: In step 2, the human posture estimation network is constructed as follows: The initial image F1 is input to the CNN module, the output of the CNN module is connected to the input of the 2DToken Processing module, the output of the 2DToken Processing module is connected to the input of the FeatTransformer module, the output of the FeatTransformer module is connected to the input of the KeypointExtration module, the output of the KeypointExtration module is connected to the input of the HeatmapRegression module, and the output of the HeatmapRegression module is the required key point heat map; For the 2DToken Processing module, its structure is as follows: it includes a SplitPatchs module, the input of the SplitPatchs module is used to input the feature map F2, the output of the SplitPatchs module is added to the position code P to obtain the 2D visual Token feature, and then the Token feature is spliced with the 2D key point Token feature automatically generated in the 2DToken Processing module to obtain the output feature map F4 of the 2DToken Processing module.
3. The method according to claim 2, characterized in that When the human body posture estimation network is working, the following steps are adopted: Step 1) extract local features from the initial image through a CNN module; the CNN module uses a parallel high-resolution HRNet network. Specifically, the initial image F1 of size {B, 3, H, W} is connected to the input of the HRNet network, and then the output size is The feature map F2 is obtained by , where B is the number of batch images, H and W are the height and width of the initial image, 3 represents the RGB channels of the initial image, and C is the dimension of the output feature map; Step 2) The 2DToken Processing module processes the feature map output by the CNN module into an input acceptable to the FeatTransformer module; specifically, the SplitPatchs module first splits the feature map F2 into several patches of size {B, C p ,h,w} feature block F3, and then generate the corresponding 2D-sine position code P for each split feature block. The matrix addition of F3 and P gives the same size {B,C p ,h,w}2D visual Token features; at the same time, the 2DToken Processing module will automatically generate 2D key point Token features of size {B,num,h,w} through random initialization, and finally concatenate the 2D key point Token features and the 2D visual Token features to obtain the output feature F4 of the 2DToken Processing module; where C p is the dimension of the feature block, h, w are the height and width of the feature block, and num is the number of key points; Step 3) The FeatTransformer module performs multi-head self-attention on the input Token to learn the context information of the image; specifically, the input feature F4 of size {B, n, h, w} will perform multi-head self-attention in parallel in the height and width dimensions, and finally obtain the output Token feature F5 of the same size {B, n, h, w}; Where n is the number of key points num and the dimension of the feature block C P of and; Step 4) Extract the required key point Token information from the output of the FeatTransformer module; specifically, directly extract the first num elements in the input Token F5, and then obtain the 2D key point Token feature F6; Step 5) The input 2D key point Token feature F6 is converted into the required key point heat map through the HeatmapRegression module; specifically, the size of the key point Token feature is converted from {B, num, h, w} to the size of the required heat map through linear transformation 4. The method according to claim 3, characterized in that: The network structure of the FeatTransformer module in step 3) is as follows: The FeatTransformer module mainly includes parallel W and H dimension processing; In the W dimension, the input Token F4 is connected to the input of the first LayerNorm module (1), and the output of the first LayerNorm module (1) is connected to the input of the Attention W module. The output of the Attention W module is added to the input Token F4 as the output of the W dimension. For the Attention W module, it generates a query matrix, a key matrix, and a value matrix Q according to the Token at the input. w ,K w ,V w , Q w The output connection of the matrix K w The output of the matrix is subjected to the first matrix multiplication, and the output of the first matrix multiplication is connected to the input of the first Softmax module (2). The output of the first Softmax module (2) is connected to V w The output of the matrix undergoes a second matrix multiplication, and the output of the second matrix multiplication is the output of the Attention W module; In the H dimension, the input Token feature F4 is connected to the input of the first Reshape module (3), the output F7 of the first Reshape module (3) is connected to the input of the second LayerNorm module (4), the output of the second LayerNorm module (4) is connected to the input of the Attention H module, and the output of the Attention H module is added to F7 as the output of the H dimension; for the Attention H module, it will generate a query matrix, a key matrix, and a value matrix Q according to the Token at the input end. H ,K H ,V H , Q H The output connection of the matrix K H The output of the matrix is subjected to a third matrix multiplication, and the output of the third matrix multiplication is connected to the input of the second Softmax module (5), and the output of the second Softmax module (5) is connected to V H The output of the matrix undergoes a fourth matrix multiplication, and the output of the fourth matrix multiplication is the output of the Attention H module; The output of the H dimension is connected to the input of the second Reshape module (6), the output of the second Reshape module (6) is added to the output of the W dimension to obtain the output TokenF8, F8 is connected to the input of the third LayerNorm module (7), the output of the third LayerNorm module (7) is connected to the input of Conv, and the output of Conv is added to F8 to obtain the output feature F5 of the FeatTransformer module.
5. The method according to claim 4, characterized in that The FeatTransformer module calculates self-attention in parallel on the input Token feature F4 in the height and width dimensions to learn the contextual information of the image; For the W dimension, directly perform the first LayerNorm module (1) operation on the input Token feature F4 to obtain T W , and then input it into the Attention W module; according to formula (1), the input Token is output after the first LayerNorm module (1) is operated W ∈R (B,n,h,w) Will pass three transformation matrices respectively Mapped to query matrix, key matrix, value matrix Q in turn w ,K w ,V w ; Then according to formula (2), Q W Matrix and K W The transposed matrix of is multiplied to obtain the attention score, which is then connected to the input of the first Softmax module (2). The output of the first Softmax module (2) is then multiplied by the matrix V W Multiply them together to get the output of Attention W; For the H dimension, the input Token feature F4 is first subjected to the first Reshape module (3) operation to swap the h and w dimensions of F4 to obtain F6, which is then subjected to the second LayerNorm module (4) operation and finally input into the Attention H module. According to formula (3), the output T after the second LayerNorm module (4) is H ∈R (B,n,w,h) Through three transformation matrices respectively Mapped into query matrix, key matrix, value matrix Q H ,K H ,V H ; Then according to formula (4), Q H Matrix and K H The transposed matrix of is multiplied to obtain the attention score, which is then connected to the input of the second Softmax module (5). The output of the second Softmax module (5) is then multiplied by the matrix V H Multiply them together to get the output of AttentionH; The output of Attention W is added to the feature F4 to obtain the output of the W dimension. The output of Attention H is first added to the feature F7, and then the second Reshape module (6) is used to swap the h and w dimensions to obtain the output of the H dimension. The output of the W dimension is then added to the output of the H dimension to obtain the output Token feature F8. The feature F8 is sequentially passed through the third LayerNorm module (7) and the Conv module. The output of the convolution Conv module is matrix-added to the original feature F8, and finally the output Token feature F5 that has learned the global context information is obtained.
6. The method according to claim 2, characterized in that The network structure of the HeatmapRegression module in step 5) is as follows: The input end of the HeatmapRegression module receives the output feature F6 from the KeypointExtration module, the feature F6 is connected to the input end of the third Reshape module (8), the output of the third Reshape module (8) is connected to the input of the fourth LayerNorm module (9), the output of the fourth LayerNorm module (9) is connected to the input of the first Liner module (10), the output of the first Liner module (10) is connected to the input of the fifth LayerNorm module (11), the output of the fifth LayerNorm module (11) is connected to the input of the second Liner module (12), the output of the second Liner module (12) is connected to the input of the fourth Reshape module (13), and the output of the fourth Reshape module (13) is the output of the HeatmapRegression module; The HeatmapRegression module mainly includes the Reshape module, the LayerNorm module and the Linear module; the main function of the Reshape module is to reshape the shape of the Token feature, and the purpose of reshaping is to perform subsequent linear operations; the third Reshape module (8) reshapes the shape of the input Token from {B, num, h, w} to {B, num, h*w}; the fourth LayerNorm module (9) and the fifth LayerNorm module (11) both normalize the data to accelerate the convergence speed; the Linear module performs linear transformation on the Token feature, the first Linear module (10) converts the shape of the Token feature from {B, num, h*w} to {B, num, hidden}, and the second Linear module (12) converts the shape of the Token from {B, num, hidden} to {B, num, hidden}. Finally, the fourth Reshape module (13) converts the output of the second Linear module (12) Reshape Thus, the num key point heat maps required by the network are obtained; hidden is the number of hidden layers, and its value is set manually.